Gavin Stewart

Centre for Evidence-Based Conservation

University of Birmingham, UK

Meta-analysis

Talk overview
A word

of warning
Meta-analysis
Data extraction for meta-analysis

Warning

Do you want or need a

meta-analysis?
heterogeneity
Sample sizes
standardisation

Meta-analysis
Steps,

usefulness and limitations of metaanalysis

Heterogeneity and methods for its
investigation
Weighting and bias

Systematic review in relation to metaanalysis

Systematic
review

Metaanalysis

quantitative
pooling of
results of
individual
studies

What does Meta-analysis do?

Meta-analysis

aims to quantitatively
combine results of different studies
Pooled estimate should be a weighted
average of all studies included in a metaanalysis
increase statistical power
improves generalisability

Weight studies based on amount of

information
The

larger studies are given more weight

than smaller studies
Sample size
Given the same sample size, more weight
will be given to studies with smaller variance
(Inverse of variance)
Scale and Pseudoreplication

Weight based on the level of validity

Quality scales
Quality components e.g., randomisation method,
baseline, sampling methods
SCALE and PSEUDOREPLICATION (again)
Different weightings for different elements, so very
controversial and often for the purpose of sensitivity
analysis
limited by sample size so maybe necessary to sum

Heterogeneity in meta-analysis
Variation

in results across studies

Distinguish between statistically significant
heterogeneity and ecologically important
heterogeneity

Causes/sources of heterogeneity
Chance
Variations

in populations
Variations in interventions
Different methodological quality
Different outcome measures

Why investigate heterogeneity?

to decide whether the results of individual studies could

be combined (FE models)
to identify effect modifiers (Study-level variables that
are associated with the results of studies) e.g. time,
method, ex situ/in situ

Methods for investigating heterogeneity

Graphical

methods
Statistical testing
Excluding outliers
Subgroup analysis
Meta-regression

Statistical testing for heterogeneity

Are the differences in result across studies greater

than could be expected by chance?
Q statistic

Excluding outliers
Outliers

are excluded one by one until the

statistical test of heterogeneity is no longer
significant

Should be used very cautiously or not at all

Subgroup analysis in meta-analysis

To separate

studies according to certain

study-level variables
Then, to conduct quantitative pooling
separately for each subgroup of studies

Meta-regression
The

estimate of study results is the

dependent variable and one or more studylevel variables are the independent variables
(predictors)

Biases and errors in meta-analysis

Meta-analysis is basically retrospective

Results of meta-analysis may be misleading
Biases may be introduced if the
identification, inclusion and assessment of
primary studies are not systematic
because of publication related biases

Publication bias
studies

with significant, positive, results are

easier to find than those with non-significant
or 'negative' results. The subsequent overrepresentation of positive studies in
systematic reviews may mean that our
reviews are biased toward a positive result.
Also have time lag bias, language bias and
citation bias
Funnel plot (se against es) asymmetry

Bad or inappropriate meta-analysis

Not systematic in study identification and assessment

Inappropriate pooling of heterogeneous results, (FE models)
No
- investigation of heterogeneity
Lack of details of included studies
Inappropriate weighting individual studies
Failed to consider publication and related biases
- sensitivity analysis
Lack of
Inappropriate interpretation of the results of subgroup analysis
Inappropriate interpretation of the pooled average

Data extraction

Extract data with synthesis in mind e.g. Mean, n, sd for treatment

and control
Extract data on effect modifiers
Use standardised piloted method and check repeatability
Consider scale and pseudoreplication
Contact authors for missing data where possible
Do not get side tracked into extracting more than you need

References

Cooper, H. and Hedges, L.V. (1994) (eds.) The Handbook of Research Synthesis. Russell Sage Foundation,
New York.
Deeks, J.J., Altman, D.G. and Bradburn, M.J. (2001) Statistical methods for examining heterogeneity and
combining results from several studies in meta-analysis. Systematic Reviews in Health Care. Meta-analysis in
Context. (eds M. Egger, G.D. Smith and D.G. Altman), pp 285-312. British Medical Journal Publishing Group,
London.
DerSimonian, R. and Laird, N. (1986) Meta-analysis in clinical trials. Controlled. Clinical. Trials, 7, 177-188.
Egger, M., Davey-Smith, G., Schneider, M. and Minder, C. (1997) Bias in meta-analysis detected by a simple
graphical test British Medical Journal, 315, 629-34.
Gates, S. (2002). Review of methodology of quantitative reviews using meta-analysis in ecology. Journal of
Animal Ecology, 71, 547557.
Gurevitch, J. and Hedges, L.V. (1999) Statistical issues in ecological meta-analyses. Ecology, 80, 11421149.
Gurevitch, J. and Hedges, L.V. (2001) Meta-analysis. Combining results of independent experiments. Design
and Analysis of Ecological Experiments (eds S.M. Scheiner and J. Gurevitch), pp. 347369. Oxford University
Press, Oxford.
Hedges, L.V., and Olkin, I. (1985). Statistical Methods for Meta-analysis. San Diego: Academic Press, San
Diego.
Hurlbert, S.H, (1984) Pseudoreplication and the design of ecological field experiments. Ecological. Monographs,
54, 187-211.
Osenberg, C.W., Sarnelle, O., Cooper, S.D. and Holt, R.D. (1999) Resolving ecological questions through metaanalysis: goals, metrics and models. Ecology, 80, 11051117.
Thompson, S.G. and Sharp, S.J. (1999) Explaining heterogeneity in meta-analysis: a comparison of methods.
Statistics in Medicine, 18, 2693-708.