Publication Bias

Funnel Plots

Examples

References

Publication Bias and Funnel Plots

Julia Braun
University of Zurich

Zurich, March 31, 2009

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Outline

Publication Bias

Funnel Plots

Examples

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Denition
Publication Bias (Dickersin, 1990):
Publication bias occurs when the publication of research results depends on their nature and direction. Other kinds of bias:

language bias availability bias cost bias familiarity bias outcome bias ...
, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Sources of publication bias

Explorative studies: Experimental: submission of similar manuscripts with positive and negative ndings, comparison of recommendations to publish Cohort: pursuit of all studies accepted by a local ethics committee Main factors: Failure to publish due to negative or null ndings (mostly on the side of the researchers, not editors/journals) Selective reporting of outcomes (e.g. due to changes in research plan) Selective citation of positive results
, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Impact of publication bias

Comparison of the results that are reported and the best guess (informed by the results of publication bias analyses). Three levels: minimal: essentially similar estimates of the eect size modest: eect size changes substantially, but key nding (e.g. treatment is or is not eective) remains the same severe: basic conclusion of the analysis changes = danger of false conclusions and patient harm
, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Solutions?
Registration and consequent tracking of clinical trials Mandatory? One general registry? Encouraging the publication of all studies, e.g. Journal of Negative Results in Biomedicine Acceptance? Practicability?

Required:
Every meta-analysis should include an analysis of publication bias, and this analysis should be reported as a standard part of the results. = Techniques for the assessment of publication bias
, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Funnel Plot
0.0 14
q q q q q q

0.2

q q q q q

Standard error

q q q

0.4

q q q q q q

q q

Inverse standard error

q qq

10

q qq

12

q q q q q q q q q

q q q q qq q q q q q q q qq

q q q q qq q q qq qq q q q q

q q q q q qq q q q q q q q qq q q q

0.6

1.5

1.0

0.5

0.0 log OR

0.5

1.0

1.5

1.0

0.5

0.0 log OR

0.5

1.0

Fundamental assumption: In the absence of bias, a plot from studies which estimate the same intervention eect will be symmetrical and bear some resemblance to a funnel (German translation: funnel = Kamin, Schacht, Trichter).
, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Denition
Funnel Plot:
A funnel plot is a scatter plot of the treatment eects estimated from individual studies against a measure of study size. Properties: The smaller the study size, the wider the spread of the treatment eects and vice versa If there is no bias: form of a funnel, symmetrical If there is bias: asymmetrical Explorative tool, no quantitative information on the amount or the source of the bias No empirical investigations on funnel plots for continuous outcomes so far
, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Composition of a funnel plot

Decisions: Which direction (horizontal vs. vertical, ascending vs. descending order)? Which measure for the treatment eect (log OR, log RR, RD)? Which measure for the study size? Inclusion of summary estimate? Which condence intervals (pointwise vs. lines around the summary estimate)? Mirror option? Choice of these elements can have a substantial inuence on the visual impression!
, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Measures for the treatment eect

Risk dierence:
0.00
q q q q q q q q q qq q q q q q q q q q q q qq q q q q q q q q q q qq q q q qq q q q q q q q qq q q q q q q qq q q q

0.05

Standard error

Easy interpretation Possibly increased heterogeneity can cause additional asymmetry

0.10

q q

0.15

q q q

0.20

0.4

0.2

0.0 Risk difference

0.2

0.4

not recommended
, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Measures for the treatment eect

Log risk ratio:
0.0

qq q q

qq q

qq

q q q q q q

q q qq q q q

Easy interpretation
Standard error

0.5

1.0

Possible heterogeneity if event rate is high Conclusions may depend on outcome denition

q q q q q q q q q q q q q q q q q q q q q q q

q q

q q

q qq q q q q q

1.5

q q q q

0 Log risk ratio

not recommended if event rate is high

, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Measures for the treatment eect

Log odds ratio:
0.0

qq q q qq q q q q q q q q q q q q q q q qq qq q q q q q q q q q q

0.5

q q

Interpretation more complicated Shape of the plot not dependent on outcome denition

Standard error

q q q q

q q

q q q q q

q q

1.0

q q q q q q q q q q

1.5

q q q q q

Log odds ratio

recommended in most cases

, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Measures for the treatment eect

0.00

0.0

0.0

q q q q q q qq q q q q q q qq q q q q q

q q q q q q qq q q q q q q q q q qq q q q q q q q q qq q q q q q q q q q q q q qq q qq q q q q q q q q

0.5

Standard error

Standard error

Standard error

q q

q qq qq q q q qq q q q q q q q q q q q q q q

0.5

q q q q qq q qq qq q q q q q qq q q q q q qq q q q q q q q q q q q qq q q qq q q q q q q q q q q q q q qq q q qq q q q q q q

q q

qq q q q q q

q q

0.10

0.05

1.0

1.0

q q q

q q q

q q q q q

1.5

1.5

qq q q

qq q q

0.20

0.15

0.4

0.2

0.0

0.2

0.4

Log odds ratio

Log risk ratio

Risk difference

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Measures for the study size

Study size/logarithmic study size:
q

10

q q

15000

10000

Log study size

Study size

q q q q q q q q q q q q qq q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q

q q

5000

q q q q q q q q q q q q q q q qq q q q qq qqq qq q q qq q q qq qq q qq q q q

q q q

qq q qq

q q

q q q

qq q q

1 Log odds ratio

1 Log odds ratio

Expected shape of the plot in the case of no bias can not be derived not recommended!
, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Measures for the study size

Standard error:
0.0
qq q q qq q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q

Characteristic funnel shape Straight condence lines Emphasis on smaller studies (bias more likely) Axis often inverted
Standard error

q q

0.5

q q

q q q q

qq q q

q q

q q

1.0

1.5

q q q q q

Log odds ratio

recommended in most cases

, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Measures for the study size

Inverse standard error:
20
q

No funnel shape Curved condence lines Emphasis on larger studies (small studies compressed at the bottom) Probably useful for the comparison of meta-analyses of small and large studies recommended for special tasks
Inverse of standard error 15

10

q q q q q q q

qq q q q

qq

q q q q q q qq q qqq q q q q qq q qq q q q q q q q qq qq q qq q q q q q q

q q

q q q q

Log odds ratio

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Measures for the study size

Variance:
0.0
qq q q q qq q q q q qqq q q q q q q q q q

q q q

q q

q q q q qq q q q q q q q q q q

0.5

q q

No funnel shape
1.0 Variance

q q q q q

Curved condence lines Emphasis on smaller studies (bias more likely) Axis often inverted

q q q

1.5

q q q q q

2.0

q q q

2.5

Log odds ratio

not recommended
, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Measures for the study size

Inverse variance:
400
q

No funnel shape Curved condence lines Emphasis on larger studies (small studies compressed at the bottom) Probably useful for the comparison of meta-analyses of small and large studies
Inverse of variance 300
q

100

200

qq q

q q q qq q q q q q q qq qqq q q q q q qq q q q qq q q qqqq q q q q qqq qq q q q q q q q

qq q q

Log odds ratio

sometimes useful, but inverse standard error should be preferred

, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Measures for the study size

0.0 0.0
q q q q q q q q q q q q q q qq q q q q q q q q q q q q q qq q q q q q qq qq q q q q q q q q q q q q q q q q q q q q q

0.5

q q q

0.5

Standard error

1.5

q q

q q q q q

q q q q

1.0

q q q q q q q q q q

5000

q q

2.0

Study size

Variance

q qq qq q q q q q qq q q q q q

q q

15000

q q q q q q q q q q qq q q qq q q q q q

1.0

10000

q q

q q q

1.5

qq

2.5

q q

qq q

q qq qq q qqq q q q q q qq qq q q q q q q qq q qq q qq q q q qq q q q q q q qq q qq qq q

qq q q

1 Log odds ratio

Log odds ratio

Log odds ratio

400

20

10

q q

Inverse of standard error

300

15

Inverse of variance

Log study size

q q qq q q q q q q qq q q q q q q qq q q q q q q q q q q

10

200

q q q q q q q q q q q q q q q

q q

100

q q q

q q q q q q q

q q q q q q q q q q q q qq q q q q q q q qq qq qq q q q q q q q qq q qq q q q qq q q q qq q q qq q q q q qqq q q qq

q q q q q q q q

q q q q q q q q qq q q q q q qq q q qq q q qqq q q q qq q q q qq q q q q q q qq q q q q qqq q q qq

1 Log odds ratio

Log odds ratio

Log odds ratio

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Condence intervals
Alternative to condence lines: Pointwise condence intervals

q q

q q

Inverse standard error

Inverse standard error

q q

q q

q q

qq

2.0

1.5

1.0

0.5

Log odds ratio

Log odds ratio

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Mirror option

q q

q q

q q

q q

qq

qq

Inverse standard error

Inverse standard error

q q q

q q

q q q

qq q

q q

qq q

qq

qq

qq

qq

0 4

Log odds ratio

Log odds ratio

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Use of funnel plots

Problems:
No clear recommendations for non-binary outcomes Primary visual tool, no quantication of bias Asymmetry not necessarily due to bias (e.g. heterogeneity in the underlying eects can also lead to asymmetry) No information on the type of bias Interpretation dicult if only few studies in meta-analysis Funnel plots and tools for meta-analysis in R: packages meta and rmeta
, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Example 1: Environmental tobacco smoke

Meta-analysis by Hackshaw et al. (1997): Eect of environmental tobacco smoke on risk of lung cancer Comparison of spouses of smokers and non-smokers 37 studies Controversy: Aected by publication bias?

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Funnel plot of Hackshaw et al. (1997)

0.0
q q q q q q q q q qq q q q q q q

0.2

Standard error

q q q q

q q

0.4

q q q q q

q q

0.6

q q

q q

1.0

0.5

0.0

0.5

1.0

1.5

Log odds ratio

Smaller studies tend to show greater adverse eects asymmetry Not clear if real bias or chance Visual inspection not enough
, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Example 2: Antibacterial catheter coating

Meta-analysis by Veenstra et al. (1998): Eectiveness of a special coating on venuous catheters Two endpoints: Bacterial colonisation of the catheter and bloodstream infection 16 controlled trials

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Funnel plot of Veenstra et al. (1998)

Bacterial colonisation:
0.0

q q q q q q q

0.5

Standard error

q q q

1.5

1.0

q q

1 Log odds ratio

hints for publication bias

, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Funnel plot of Veenstra et al. (1998)

Bloodstream infection:
0.0

q q

0.5

Standard error

q q q q

1.0

q q

1.5

Log odds ratio

hints for publication bias

, , ,

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

Summary

Visual tool for investigation of (publication) bias in meta-analysis Scatter plots of treatment eect versus study size For binary outcomes: odds ratio and standard errors best choice In the case of bias: asymmetry Other causes of asymmetry possible Interpretation dicult if number of studies small

Julia Braun Publication Bias and Funnel Plots

University of Zurich

Publication Bias

Funnel Plots

Examples

References

References
Dickersin, K. (1990). The existence of publication bias and risk factors for its occurrence, Journal of the American Medical Association 263(10): 13851395. Hackshaw, A., Law, M. and Wald, N. (1997). The accumulated evidence on lung cancer and environmental tobacco smoke, British Medical Journal 315: 980988. Rothstein, H., Sutton, A. and Borenstein, M. (eds) (2005). Publication Bias in Meta-Analysis, Wiley. Veenstra, D., Saint, S., Saha, S., Lumley, T. and Sullivan, S. (1998). Ecacy of antiseptic impregnated central venous catheters in preventing nosocomial infections: a meta analysis, Journal of the American Medical Association 281: 261267.

Julia Braun Publication Bias and Funnel Plots

University of Zurich