{Jobs}0688jw/makeup/688ch38.

3d

38
The Use of Meta-analysis in
Pharmacoepidemiology
JESSE A. BERLIN
University of Pennsylvania School of Medicine, Center for Clinical Epidemiology and Biostatistics, University of
Pennsylvania Medical Center, Philadelphia, PA, USA

INTRODUCTION review has been increasingly recognized as being

Meta-analysis has been defined as ``the statistical
analysis of a collection of analytic results for the
purpose of integrating the findings.''1 Other
definitions have included qualitative, as well as
quantitative, analyses.2 Meta-analysis is used to
identify sources of variation among study findings
and, when appropriate, to provide an overall
measure of effect as a summary of those findings.3
While epidemiologists have been cautious about
adopting meta-analysis,4 ± 6 the need to make the
most efficient and intelligent use of existing data
prior to (or instead of) embarking on a large,
primary data collection effort, has dictated a
progressively more accepting approach.6± 12
Meta-analysis may be regarded as a ``state-of-
the-art'' literature review, employing statistical
methods in conjunction with a thorough and
systematic qualitative review.13 The distinguishing
feature of meta-analysis, as opposed to the usual
qualitative literature review, is its systematic,
structured, and presumably objective presentation
and analysis of available data. The traditional
subjective.13± 16 With the support of leading scien-
tists17 and journal editors,18 there has been
growing acceptance of the concept that the
literature review can be approached as a more
rigorous scientific endeavor, specifically, an ob-
servational study with the same requirements for
planning, prespecification of definitions, use of
eligibility definitions, etc., as any other observa-
tional study. In recent years, the terms research
synthesis and systematic review have been used to
describe the structured review process, in general,
while meta-analysis has been reserved for the
quantitative aspects of the process. For the
purposes of this chapter, we shall use meta-analysis
in the more general sense. Meta-analysis provides
the conceptual and quantitative framework for
such rigorous literature reviews; similar measures
from comparable studies are tabulated system-
atically and the effect measures are combined
when appropriate.
Several activities may be included under the
above definition of meta-analysis. Perhaps the
most popular conception of meta-analysis, for

Pharmacoepidemiology, Third Edition. Edited by B. L. Strom.

# 2000 John Wiley & Sons, Ltd.
{Jobs}0688jw/makeup/688ch38.3d
634 PHARMACOEPIDEMIOLOGY
most clinically oriented researchers, is the sum-
mary of a group of randomized clinical trials
dealing with a particular therapy for a particular
disease. An example of this approach would be a
study that examined the effects of aspirin following
myocardial infarction. Typically, this type of meta-
analysis would present an overall measure of the
efficacy of treatment, e.g., a summary odds ratio.
Summary measures may be presented for different
subsets of trials involving specific types of patients,
e.g., studies restricted to men versus studies that
include both men and women. More sophisticated
meta-analyses also examine the variability of
results among trials and, when results have been
conflicting, attempt to uncover the sources of the
disagreements.19
More recently, meta-analyses of nonexperimen-
tal epidemiologic studies have been performed20± 28
and articles have been written describing the
methodologic considerations specific to those
meta-analyses.3, 6, 11± 13, 29 ±38 In general, both the
meta-analyses of nonexperimental studies and the
associated methodologic articles tend to focus
more on the exploration of reasons for disagree-
ment among the results of prior studies, including
the possibility of bias. Given the greater diversity
of designs of nonexperimental studies, it is logical
to find more disagreement among nonexperimen-
tal studies than among randomized trials.
This chapter summarizes many of the major
conceptual and methodologic issues surrounding
meta-analysis and offers the views of one meta-
analyst about possible avenues for future research
in this field.
CLINICAL PROBLEMS TO BE
ADDRESSED BY
PHARMACOEPIDEMIOLOGY
RESEARCH
There are a number of reasons why a pharmacoe-
pidemiologist might be interested in conducting a
meta-analysis. These include the study of uncom-
mon negative outcomes of therapies (adverse drug
reactions) free of the confounding and bias of
nonexperimental studies, the exploration of rea-
sons for inconsistencies of results across previous
studies, the exploration of subgroups of patients in
whom therapy may be more or less effective, the
combination of studies involved in the approval
process for new therapies, and the study of positive
effects of therapies, as in the investigation of new
indications for existing therapies, particularly
when the outcomes being studied are uncommon
or the past studies have been small.
The investigation of adverse effects has been a
recurring theme throughout this book, as it is a
major focus of pharmacoepidemiology. It is most
often, but not always, pursued through nonexperi-
mental studies. The difficulties in studying these
events have also been detailed throughout this
book. One major challenge involves obtaining
information on adverse reactions that is uncon-
founded by indication (see Chapter 43). These
adverse events often occur only rarely, making
their evaluation still more difficult. The results of
nonexperimental studies of whether such events
are associated with a particular drug may be
conflicting, leaving a confusing picture for the
practicing clinician and the policy makers to
interpret. Meta-analysis, by combining results
from many randomized studies, can address the
problem of rare events and rectify the associated
lack of adequate statistical power in a setting free
of the confounding and bias of nonexperimental
studies. For example, Chalmers and colleagues
used meta-analysis of randomized clinical trials to
explore possible gastrointestinal side-effects of
nonsteroidal anti-inflammatory drugs (NSAIDs).39
These studies individually had almost no power to
detect any association between NSAIDs and
adverse gastrointestinal outcomes, but collectively
the number of subjects was adequate both to show
some important associations and to show the
rarity of most complications. The details of this
NSAID meta-analysis will be presented later in
this chapter.
When reports of several investigations of a
specific adverse drug reaction disagree, whether
randomized or nonexperimental in design, meta-
analysis can also be used to help resolve these
disagreements. These disagreements among studies
may arise from differences in the choice of
endpoints, the exact definition of exposure, the
eligibility criteria for study subjects, the methods
{Jobs}0688jw/makeup/688ch38.3d
THE USE OF META-ANALYSIS IN PHARMACOEPIDEMIOLOGY
of obtaining information, other differences in
protocols, or a host of other reasons possibly
related to the quality of the studies. While it is not
possible to produce a definitive answer to every
research question, the exploration of the reasons
for heterogeneity among studies' results may at
least provide valuable guidance concerning the
design of future studies.
The exploration of subgroups of patients in
whom therapy may be more or less effective is a
controversial question in individual randomized
trials. Most trials are not designed with sample
sizes adequate to address efficacy in subgroups.
The finding of statistically significant differences
between the effects of therapy in different sub-
groups, particularly when those groups were not
defined prospectively, raises the question of
whether those are spurious findings. Conversely,
the lack of statistical significance for clinically
important differences between prospectively de-
fined subgroups can often be attributed to a lack
of statistical power. Such clinically meaningful but
statistically nonsignificant findings are difficult to
interpret. Meta-analysis can be used to explore
these questions with improved statistical power.
The use of meta-analysis in the approval process
for new drugs represents another potential appli-
cation, although experience in this area is as yet
rather limited. However, many of the methodolo-
gic issues arising in the context of new drug
approval also arise in the investigation of new
indications for pharmaceutical products that have
previously been approved for other purposes. For
some therapies, such as streptokinase in the
treatment of myocardial infarction, meta-analysis
could have been used to summarize evidence prior
to embarking on a very large scale, multicenter,
randomized trial.40
METHODOLOGIC PROBLEMS TO BE
ADDRESSED BY
PHARMACOEPIDEMIOLOGY
RESEARCH
As the skeptical reader might imagine, many
methodologic issues can arise in the context of
performing a meta-analysis. Many, but not all, of
635
these problems relate to the process of combining
studies that are often diverse with respect to
specific aspects of design or protocol, some of
which may be of questionable quality.
QUALITY OF THE ORIGINAL STUDIES
Meta-analysis seems particularly prone to the
``garbage in, garbage out'' phenomenon. Combin-
ing a group of poorly done studies can produce a
precise summary result built on a very weak
foundation. This apparent precision may lend
undue credibility to a result that truly should not
be used as a basis for formulating clinical or policy
strategies.5 However, if the quality judgment is
subtly influenced by the direction or magnitude of
the findings of the study, excluding studies based
on such a subjective judgment about their quality
could open the meta-analytic process to a serious
form of bias.
COMBINABILITY OF STUDIES
Clearly, no one would suggest combining studies
that are so diverse that a summary would be
nonsensical. For example, one would not combine
studies of aspirin in the primary prevention of
coronary heart disease with studies of aspirin given
after myocardial infarction. Beyond obvious ex-
amples like this, however, the choices may not be
so clear. Should studies with different patient
populations be combined? How different can those
populations be before it becomes unacceptable to
combine the studies? Should nonrandomized
studies be combined with randomized studies?
Should nonrandomized studies ever be used in a
meta-analysis? These are questions that cannot be
answered without generating some controversy.
PUBLICATION BIAS
Unpublished material cannot be retrieved by
literature searches and is likely to be difficult to
find referenced in published articles. Publication
bias occurs when study results are not published,
or their publication is delayed, because of the
results.41± 50 The usual pattern is that statistically
significant results are published more easily than
{Jobs}0688jw/makeup/688ch38.3d
636 PHARMACOEPIDEMIOLOGY
nonsignificant results, although this bias may not
be as severe for randomized studies as it is for
nonrandomized studies.42, 51 While one could
simply decide not to include unpublished studies
in a meta-analysis, since those data have often not
been peer reviewed,52 unpublished data can repre-
sent a large proportion of all available data.53 If
the results of unpublished studies are system-
atically different from those of published studies,
particularly with respect to the magnitude and=or
direction of the findings, their omission from a
meta-analysis would yield a biased summary
estimate (assuming that the quality of the unpub-
lished studies is at least equal to the quality of the
published studies).
Publication bias is a potentially serious limita-
tion to any meta-analysis. The retrospective
identification of completed unpublished trials is
clearly possible53 in some instances, but generally
is not practical. One study54 used a survey of
investigators to attempt to identify unpublished
studies. The authors surveyed 42 000 obstetricians
and pediatricians, asking whether they had parti-
cipated in any unpublished trials completed more
than two years previously, i.e., during the period
prior to the end of 1984. They identified only 18
such studies, despite an overall response rate of
94% to their survey.
Other forms of bias, related to publication bias,
have also been identified.47 These include reference
bias, i.e., preferential citation of significant find-
ings,55 language bias, i.e., exclusion of studies in
languages other than English,56, 57 and bias related
to source of funding.42, 43, 58± 60
BIAS IN THE ABSTRACTION OF DATA
Meta-analysis, by virtue of being conducted after
the data are available, is a form of retrospective
research and is thus subject to the potential biases
inherent in such research.61 In the meta-analysis of
gastrointestinal side-effects of NSAIDs mentioned
above39 and described more fully below, Chalmers
and colleagues examined over 500 randomized
studies. They measured the agreement of different
individuals when reading Methods sections of
papers that had been masked as to their source
and the results. There were disagreements on
10± 20% of items, which had to be resolved in
conference with a third person. These disagree-
ments arose from errors on the part of the reader
and from lack of clarity of the presentation of
material in the original articles. Whatever its
source, when such variability exists, the opportu-
nity for observer bias may exist as well.61
In a number of instances, more than one meta-
analysis has been performed in the same general
area of disease and treatment. A review of 20 of
these instances52 showed that, for almost all
disease=treatment areas, there were differences
between two meta-analyses of the same topic in
the acceptance and rejection of papers to be
included. While there was only one case (out of
the 20) of extreme disagreement regarding efficacy,
there were several cases in which one or more
analyses showed a statistically significant result
while the other(s) showed only a trend. These
disagreements were not easily explainable. For
example, differences between meta-analyses of the
same topic in the acceptance and rejection of
papers did not always lead to differences in
conclusions.
More generally, the acceptance or rejection of
different sets of studies can drastically change
conclusions. This is illustrated by several meta-
analyses of whether or not corticosteroid drugs
cause peptic ulcer. The first published paper
argued that corticosteroids did not cause peptic
ulcer, because the p-value for the meta-analysis
was only 0.07.62 Five years later, a second analysis,
by a second set of authors, included a larger
number of studies and found evidence for an
association with a p-value of less than 0.001.63 Re-
analysis of the data from the second meta-analysis
by the authors of the first meta-analysis, with the
addition of several more studies, gave a p-value of
0.40.64 Another meta-analysis done by the second
set of authors gave a revised p-value of 0.01.65
Despite efforts to make meta-analysis an objective,
reproducible activity, there is evidently some
judgment involved.
In a separate commentary,66 DerSimonian
reanalyzed data from one meta-analysis and one
clinical review of parenteral nutrition with
branched-chain amino acids in hepatic encephalo-
pathy. She pointed to differences in the data
{Jobs}0688jw/makeup/688ch38.3d
THE USE OF META-ANALYSIS IN PHARMACOEPIDEMIOLOGY
extracted by the two sets of authors67, 68 for the
same endpoints from the same original papers.
When combined statistically, the data extracted by
the two sets of authors led to substantively
different conclusions about the efficacy of therapy.
CURRENTLY AVAILABLE SOLUTIONS
This section will first present the general principles
of meta-analysis and a general framework for the
methods typically employed in a meta-analysis.
Much of this material has been presented in review
articles in major clinical journals,7, 9, 10 so only the
most important points will be highlighted here. In
the second part of this section, specific solutions to
the methodologic issues raised in the previous
section are presented. Finally, case studies of
applications that should be of interest to pharma-
coepidemiologists will be presented, illustrating
approaches to some of the clinical and methodo-
logic problems raised earlier.
STEPS INVOLVED IN PERFORMING A
META-ANALYSIS (SEE TABLE 38.1)
Define the Purpose
While this is an obvious component of any
research, it is particularly important to define
precisely the primary and secondary objectives of a
meta-analysis. The important primary question
might be ``Are NSAIDs associated with an
increased risk of gastrointestinal side effects?''.
Another might be ``Are corticosteroids effective in
the treatment of alcoholic hepatitis?''. Secondary
objectives might include the identification of
subgroups in which a treatment appears to be
Table 38.1. General steps involved in conducting a
meta-analysis
(1) Define the purpose
(2) Perform the literature search
(3) Establish inclusion=exclusion criteria
(4) Collect the data
(5) Perform statistical analyses
(6) Formulate conclusions and recommendation
637
uniquely more or less effective. For NSAIDs,
estimating the absolute risk difference (and, thus,
the public health implications) as well as the
relative risk (and, thus, the etiologic implications)
might be a secondary objective.
Perform the Literature Search
While computerized searches of the literature can
facilitate the retrieval of all relevant published
studies, these searches are not always reliable.
Several studies have examined problems with the
use of electronic searches.69± 71 Use of search terms
that are too nonspecific can result in large numbers
of mostly irrelevant citations that need to be
reviewed to determine relevance. Use of too many
restrictions can result in missing a substantial
number of relevant publications. For example, in
preparing for meta-analyses of neonatal hyperbi-
lirubinemia, MEDLINE was searched for relevant
clinical trials.69 A search by a trained librarian
identified only 29% of known trials in the Oxford
Database of Perinatal Trials.72 It is generally
suggested that a professional librarian with train-
ing and experience in searches of clinical topics be
consulted, although, as just cited, even a trained
librarian may not perform perfectly. Other meth-
ods of searching, such as review of the reference
sections of retrieved publications found to be
relevant, and hand searches of relevant journals,
are also recommended.
Establish Inclusion=Exclusion Criteria
A set of rules for including and excluding studies
from the meta-analysis should be defined during
the planning stage of the meta-analysis and should
be based on the specific hypotheses being tested in
the analysis. One might, for example, wish to limit
consideration to randomized studies with more
than some minimum number of patients. In a
meta-analysis of epidemiologic studies, one might
wish to include studies of incident cases only,
excluding studies of prevalent cases, assuming that
the relationship between exposure and outcome
could be different in the two types of study.
Practical considerations may, of course, force
changes in the inclusion criteria. For example,
{Jobs}0688jw/makeup/688ch38.3d
638 PHARMACOEPIDEMIOLOGY
one might find no randomized studies of a
particular new indication for an existing therapeu-
tic agent, thus forcing consideration of nonrando-
mized studies.
In establishing inclusion=exclusion criteria, one
is also necessarily defining the question being
addressed by the meta-analysis. If broad inclusion
criteria are established, then a broad, and perhaps
more generalizable, hypothesis may be tested. The
use of broad entry criteria also permits the
examination of the effects of research design on
outcome (e.g., do randomized and nonrandomized
studies tend to show different effects of therapy?)
or the exploration of subgroup effects. In the
example of aspirin administered following myo-
cardial infarction, restriction of the meta-analysis
to studies using more than a certain dose of aspirin
would not permit an exploratory, cross-study
comparison of dose ±response effects, which might
prove illuminating.
A key point is that exclusion criteria should be
based on a priori considerations of design of the
original studies and completeness of the reports
and specifically not on the results of the studies. To
exclude studies solely on the basis of results that
contradict the majority of the other studies will
clearly introduce bias into the process.11 While that
may seem obvious, the temptation to try to justify
such exclusions on a post hoc basis may be strong,
particularly when a clinically plausible basis for
the exclusion can be found. Such exclusions made
after having seen the data, and the effect of
individual studies on the pooled result, may form
the basis for legitimate sensitivity analyses (com-
paring pooled results with and without that
particular study included), but should not be
viewed as primary exclusion criteria.
Another important note is that studies may
often generate more than one published paper. For
example, later reports might update analyses
previously published, or might report on outcomes
not addressed in earlier papers. It is essential, for
two reasons, that only one report on the same
patients be accepted into the meta-analysis. First,
the validity of the statistical methods depends on
the assumption that the different studies represent
different groups of individuals. Second, the inclu-
sion of a study more than once would assign undue
weight to that study in the summary measure. A
caution is that it is not always obvious that the
same patients have been described in two different
publications. Contacting the authors may be of
some help in determining whether there is duplica-
tion, although some authors may perceive the
inquiry as questioning their academic integrity.
The issue of multiple publications based on the
same study has been addressed in more detail by
Huston and Moher.73
Collect the Data
When the relevant studies have been identified and
retrieved, the important information regarding
study design and outcome needs to be extracted.
Typically, data abstraction forms are developed,
pilot tested on a few articles, and revised as
needed. As in any research, it is necessary to strike
a balance between the completeness of the
information abstracted and the amount of time
needed to extract that information. Careful speci-
fication in the protocol for the meta-analysis of the
design features and patient characteristics that will
be of clinical or academic interest may help avoid
over- or undercollecting information. It is gener-
ally advisable, when possible, to collect raw data
on outcome measures, e.g., numbers treated and
number of events in each group, rather than
derived measures such as odds ratios, which may
not be the outcome measures of interest in the
meta-analysis or may have been calculated incor-
rectly by the original authors.
Many articles on ``how to do a meta-analysis''
(e.g.,9, 10) recommend that the meta-analyst assess
the quality of the studies being considered in a
meta-analysis. One might wish to use a measure of
study quality as part of the weight assigned to each
study in the analysis, as an exclusion criterion (e.g.,
excluding studies with quality scores below some
arbitrary threshold), or as a stratification factor
allowing the separate estimation of effects for good
quality and poor quality studies.74, 75 Chalmers and
colleagues have developed a quality assessment
scoring system for randomized trials.76 Several
groups have opted for other, far shorter and
simpler, systems (e.g.,77± 80) Issues related to
quality scoring have been discussed more generally
{Jobs}0688jw/makeup/688ch38.3d
THE USE OF META-ANALYSIS IN PHARMACOEPIDEMIOLOGY
by Moher and colleagues,81 and an annotated
checklist of quality scoring systems is available.82
Most of these systems were proposed as very
general systems that could be applied to clinical
trials covering a wide range of therapies and
endpoints. Scoring systems designed for epidemio-
logic studies have been developed as well, in the
context of evaluating studies of specific exposure ±
disease relationships (e.g.,21).
The argument has been made, however, that
general scoring systems are arbitrary in their
assignment of weights to particular aspects of
study design, and that such systems risk losing
information, and can even be misleading.38, 83, 84
Juni and colleagues,84 for example, examined
studies comparing low molecular weight heparin
with standard heparin with respect to prevention
of postoperative thrombosis. They used 25 differ-
ent quality assessment scales to identify high
quality trials. For six scales, the studies identified
as being of high quality showed little to no benefit
of low molecular weight heparin, while for seven
scales, the ``high quality'' studies showed a
significant advantage of low molecular weight
heparin. This apparent contradiction raises ques-
tions about the validity of such scales as methods
for stratifying studies.
Thus, in a given meta-analysis, one might wish
to examine specific aspects of study design that are
unique to that clinical or statistical situation.38, 83 ±
85
For example, Schulz and colleagues85 found that
trials in which the concealment of randomized
allocation was inadequate on average produced
larger estimates of treatment effects compared
with trials in which allocation was adequately
concealed. This specific finding was not detected
when these same authors looked for an overall
association between quality score and treatment
effect. In the analysis of low molecular weight
heparin, Juni and colleagues84 found that studies
with unmasked outcome assessment showed lar-
ger, and presumably biased, benefits of low
molecular weight heparin than studies using
masked assessment of outcome.
Two procedural recommendations have been
made regarding the actual techniques for data
extraction. One is that studies should be read
independently by two readers. The justification for
639
this comes from meta-analyses in which modest
but important inter-reader variability has been
demonstrated.39, 61 A second recommendation is
that readers be masked to certain information in
studies, such as the identity of the authors and the
institutions at which a study was conducted, and
masked to the specific treatment assignments.52
While masking has a high degree of intuitive
appeal, the effectiveness of masking in avoiding
bias has not been demonstrated. Only one
randomized trial examines the issue of the effect
of masking on the results of meta-analyses.86 This
study compared the results of the same meta-
analyses performed independently by separate
teams of meta-analysts, with one team masked
and the other unmasked. The masked and
unmasked teams produced nearly identical results
on a series of five meta-analyses, lending little
support to the need for masking.
Perform Statistical Analyses
In most situations, the statistical methods for the
actual combination of results across studies are
fairly straightforward. If one is interested in
combining odds ratios or other estimates of
relative risk across studies, for example, some
form of weighted average of within-study results
is appropriate, and several of these exist.87 A
popular example of this is the Mantel ± Haenszel
procedure, in which odds ratios are combined
across studies with weights proportional to the
inverse of the variance of the within-study odds
ratio.29, 31, 87 Other approaches include inverse
variance weighted averages of study-specific
estimates of multivariate adjusted relative
risks29, 87 and exact stratified odds ratios.88 One
popular method, sometimes called the ``one-step''
method, which is similar to the Mantel ± Haenszel
method,89 has been shown to be biased under
some circumstances. Since this method offers no
clear advantage over the Mantel ± Haenszel meth-
od, which is more robust, the Mantel ± Haenszel
method may be preferable in many circum-
stances.31 Choices are never simple, however. In
a simulation study, Deeks and colleagues90
showed, at an international meeting, that in
situations where there are rare events, and
{Jobs}0688jw/makeup/688ch38.3d
640 PHARMACOEPIDEMIOLOGY
consequently frequent zero cells in contingency
tables, the one-step method tended to perform
better than other alternatives, including Mantel ±
Haenszel and exact methods.
One basic principle in most analytic approaches
is that the comparisons between treated (exposed)
and untreated (unexposed) patients are typically
made within a study prior to combination across
studies. In the combination of randomized trial
results, this amounts to preserving the randomiza-
tion within each study prior to combination. In all
of the procedures developed for stratified data,
``study'' plays the role of the stratifying variable.
In general, more weight is assigned to large studies
than to small studies because of the increased
precision of larger studies. A second basic principle
to note is that these methods generally assume that
the studies are all estimating a single, common
effect, e.g., a common odds ratio. In other words,
the underlying treatment effect (whether beneficial
or harmful) that all studies are estimating is
assumed to be the same for all studies. Any
variability among study results is assumed to be
random and is ignored in producing a summary
estimate of the effect.91, 92
In any meta-analysis, however, the possible
existence of heterogeneity among study designs
and results should be examined, and may warrant
a set of exploratory analyses designed to investi-
gate the sources of that heterogeneity. Methods for
detecting and describing heterogeneity are de-
scribed in detail by Hardy and Thompson.93 One
might stratify the studies according to patient
characteristics or study design features and in-
vestigate heterogeneity within and across strata.
To the extent that the stratification explains the
heterogeneity, the combined results would differ
between strata and the heterogeneity within the
strata would be reduced compared to the overall
result. In addition to stratification, regression
methods such as weighted least squares linear
regression could be used to explore sources of
heterogeneity.3, 29, 94± 96 These might be important
when various components of study design are
correlated with each other, acting as potential
confounders. Graphical methods for meta-analysis
have also been proposed, that focus on issues
related to heterogeneity.97
There are also methods for combining studies
that do not make the assumption of a common
treatment effect across all studies. These are the so-
called ``random effects'' models, which allow for
the possibility that the underlying true treatment
effect, which each study is estimating, may not be
the same for all studies, even when examining only
studies with similar designs, protocols, and patient
populations. Hidden or unmeasured sources of
among-study variability of results are taken into
account by these random effect models through
the incorporation of such variability into the
weighting scheme when computing a weighted
average summary estimate. The practical conse-
quence of the random effect models is to produce
wider confidence intervals than would otherwise
be produced by the traditional methods.91, 92 This
approach is particularly useful when there is
heterogeneity among study results, and explora-
tory analyses have failed to uncover any known
sources of observed heterogeneity. However, ran-
dom effect models should not be viewed as a
panacea for unexplained heterogeneity. One dan-
ger is that a summary measure of heterogeneous
studies may not really apply to any particular
study population or study design, i.e., they lose
information by averaging over potentially impor-
tant study and population characteristics.38
A second practical effect of random effect
models, which is only apparent from examining
the mathematics involved, is that they tend to
assign relatively higher weights to small studies
than the traditional methods would assign.91 This
equalization of weights may have unwanted
consequences in some circumstances (see section
on ``Publication bias,'' below), and can lead to
counterintuitive results, with very small studies
making contributions to the summary equal to
those of very large studies. A thorough discussion
of the interpretation and application of fixed
versus random effect models is presented by
Hedges and Vevea.98
Bayesian statistical methods are also being
proposed with increasing frequency in the statis-
tical literature.99 ±104 These include the so-called
``confidence profile'' method developed by Eddy
and colleagues.105, 106 These methods can incorpo-
rate into the analysis the investigator's prior beliefs
{Jobs}0688jw/makeup/688ch38.3d
THE USE OF META-ANALYSIS IN PHARMACOEPIDEMIOLOGY
about the size of an effect or about the factors
biasing the observed effects. When the investigator
has no prior beliefs about the effect, the results of
the observed studies are sometimes used to
estimate the components of the ``prior'' distribu-
tion. Thus, the final answers reflect the observed
data very closely. In practice, when the investiga-
tor does not specify prior beliefs, the summary
results are similar to those from standard methods,
especially the random effect models described
above.
Another approach107 is based on summing the
statistical evidence provided by each study for each
value of the effect measure (e.g., the odds ratio).
The value of the odds ratio with the maximum
evidence (the maximum likelihood estimate) has
usually proved the same as the pooled estimate
produced by other methods, in the situations
where this method has been used.107 By providing
a mathematical and graphical picture of what is
occurring at other values of the effect estimate, this
method also provides information on the con-
tribution of each study to the total.
An important word of caution is that statistical
tests of heterogeneity, i.e., formal statistical tests
of the variability among the studies, suffer from a
notorious lack of statistical power.108 Thus, a
finding of significant heterogeneity may safely be
interpreted as meaning the studies are not all
estimating the same parameter. A lack of
statistical significance, however, may not mean
that heterogeneity is not important in a data set
or that sources of variability should not be
explored.
Formulate Conclusions and
Recommendations
As with all research, the conclusions of a meta-
analysis should be clearly summarized, with
appropriate interpretation of the strengths and
weaknesses of the meta-analysis. Authors should
clearly state how generalizable the result is, how
definitive it is, and should outline the areas that
need future research. Any hypotheses generated by
the meta-analysis should be stated as such, and not
as conclusions.
641
APPROACHES TO SELECTED
METHODOLOGIC PROBLEMS IN
META-ANALYSIS
Combinability of Results from Diverse
Studies: Heterogeneity is your Friend
The underlying question in any meta-analysis is
whether it is clinically and statistically reasonable
to estimate an average effect of therapy, either
positive or negative. If one errs on the side of being
too inclusive, and the studies differ too greatly,
there is the possibility that the average effect may
not apply to any particular subgroup of pa-
tients.38, 109 Conversely, diversity of designs and
results may provide an opportunity to understand
the factors that modify the effectiveness (or
toxicity) of a drug. In fact, it has been argued that
because of the potential for bias in observational
epidemiologic studies, exploring heterogeneity
should be the main point of meta-analyses of such
studies, rather than producing a single summary
measure.6, 33, 38, 110
In addition to considering the clinical and
methodologic differences among studies, prior to
the actual combination of results, it is essential to
evaluate the extent to which study results are
combinable from a strictly statistical viewpoint.
This would usually involve some kind of statistical
test of the variability (heterogeneity) of results
among studies. If the variability of odds ratios (on
the natural logarithm scale), for example, is greater
than that which could be attributed to sampling
variability alone, one should question the wisdom
and meaning of a combined result. Efforts should
subsequently focus on exploring the reasons for
the variability of results, rather than proceeding to
combine them.
The generality of the question posed will clearly
influence the generalizability of the result, but also
may affect whether the primary studies involved
are viewed as combinable or not. Because the set of
available studies is likely to be heterogeneous with
respect to design features, the choice of a more
general question may be preferred to a very
specific one. For example, Dickersin and Berlin13
suggest that a more general question might be ``Is
taking aspirin associated with decreased mortality
{Jobs}0688jw/makeup/688ch38.3d
642 PHARMACOEPIDEMIOLOGY
in patients who have already had a myocardial
infarction?'' rather than ``Is administration of 325
milligrams of aspirin per day, started within seven
days of a first documented myocardial infarction
and taken for at least six months, in the absence of
other preventive treatments, in patients followed
for at least one year, associated with a decrease in
subsequent cardiovascular mortality?''. If addres-
sing the second question, the meta-analyst may
quickly find him or herself with only one available
study. Diversity of study designs, on the other
hand, can provide a more generalizable result than
restriction to a very narrowly defined group of
studies. Issues of study design, such as dose or
duration of therapy, and how study design relates
to study results, could be addressed through a
series of exploratory analyses.13
As an example of the type of analysis that could
be used to investigate study design issues, Berlin
and colleagues,3 in a methodologic paper, examine
data originally presented by Romieu and collea-
gues20 on the relationship between duration of oral
contraceptive use and risk of breast cancer. They
show, using regression methods and stratified
analyses, that case ± control studies involving
mostly premenopausal women show a marked
and quite homogeneous increase in risk with
increasing duration of use of oral contraceptives.
Studies including postmenopausal cases, conver-
sely, show no such increase in risk, on average.
Furthermore, the average effect is difficult to
interpret, since the results of the postmenopausal
studies are quite heterogeneous, with some studies
showing an increase in risk with increasing
duration of oral contraceptive use and others
showing a decrease. One technical difficulty with
meta-analysis of group-level data is also illustrated
by this example. Although some studies present
age-specific results, or are restricted to certain age
groups, many studies do not present subgroup
analyses that are of interest in a meta-analysis.
Thus, for example, some studies that include
postmenopausal women also include premenopau-
sal women, and the data within such studies may
not be presented by menopausal status.
In the situation just described, there were strong
biological reasons for performing separate ana-
lyses by menopausal status. It is sometimes
instructive to perform more exploratory analyses
of meta-analytic data as well. These may provide
valuable insights into the biology of the problem
and=or may generate hypotheses for future con-
firmation. Morgenstern et al.28 found that the
association between neuroleptic medication and
tardive dyskinesia was stronger in studies con-
ducted in the United States than in studies
conducted elsewhere. They used regression meth-
ods to show that this association was not simply
the product of confounding by other study design
features. The authors suggest that the US study
samples may have had a higher baseline frequency
of unmeasured factors (e.g., affective disorders
such as schizophrenia) than the exposed groups in
other countries. As with any exploratory analysis,
due caution must be exercised in the interpretation
of such a posteriori hypotheses, even though they
may be based on very sound biological reasoning.
Publication Bias
As discussed above, when the primary source of
data for a meta-analysis is published data, there is
usually a danger that the published studies
represent a biased subset of all the studies that
have been done. In general, it is more likely that
studies with statistically significant findings will be
published than studies with no significant findings.
A practical technique for determining the potential
for publication bias is the ``funnel plot,'' first
proposed by Light and Pillemer.111 The method
involves plotting the effect size (e.g., the risk
difference) against a measure of study size, such as
the sample size, or the inverse of the variance of
the individual effect sizes. If there is no publication
bias, the points should produce a kind of funnel
shape, with a scatter of points centered around the
true value of the effect size, and with the degree of
scatter narrowing as the variances decrease. If
publication bias is a problem, the funnel would
look as though a bite had been taken out, with
very few (if any) points around the point indicating
no effect (e.g., odds ratio of 1.0) for studies with
large variances. This method requires a sufficient
number of studies to permit the visualization of a
funnel shape to the data. If the funnel plot does
indicate the existence of publication bias, then one
{Jobs}0688jw/makeup/688ch38.3d

THE USE OF META-ANALYSIS IN PHARMACOEPIDEMIOLOGY 643

or more of the correction methods described below
should be considered. In the presence of publica-
tion bias, the responsible meta-analyst should also
evaluate the ethics of presenting a summary result
that is likely to represent an overestimate of the
effect in question.
Two examples of funnel plots are given in
Figures 38.1 and 38.2. These plots represent
studies of psychoeducational programs for surgical
patients.111, 112 In the first plot, only the published
studies are represented. The funnel appears to have
a ``bite'' taken out of it where the small studies
showing no effect of these programs should be. In
the second plot, the unpublished studies, including
doctoral dissertations, are included, and the
former ``bite'' is now filled with these unpublished
studies.
Several mathematical approaches to the pro-
blem of publication bias have also been proposed.
An early method, first described by Rosenthal,113
is the calculation of a ``fail-safe N'' when the result
of the meta-analysis is a statistically significant
rejection of the null hypothesis. This method, in a
kind of sensitivity analysis, uses the Z-statistics
from the individual studies included in a meta-
analysis to calculate the number of unpublished
studies with a Z-statistic of exactly 0 that would be
required to exist, in order for the combined Z-
score (published ‡ unpublished studies) to become
nonsignificant. Because this method focuses only
on Z-statistics, and ignores the estimation of
effects (e.g., odds ratios), it is of limited utility.
That is, the fail-safe N approach focuses only on
the statistical significance of the combined result
and does not help provide an overall estimate of
the effect that is ``adjusted'' for publication bias.
A number of related methods to deal with
potential unpublished studies have been developed
in recent years. These include other methods for
estimating the number of unpublished stu-
dies,114, 115 formal methods to test for the presence
of publication bias,116 ±118 and methods to adjust
summary estimates to account for unpublished
studies,114, 119 ± 122 but several of those methods
make some fairly strong assumptions about the
specific mechanism producing the publication bias.
The available data are used to estimate both the
average effect size for the comparison of groups
and the probabilities of publication. (Note that
only published data are used to estimate the
probability of publication, strengthening the de-
pendence of such methods on the underlying

Figure 38.1. Funnel plot for published studies only: analysis of data from the review by Devine and Cooks'112 of
psychoeducational programs for surgical patients. Reprinted by permission of the publishers from Summing Up: the
Science of Reviewing Research by Richard J. Light and David B. Pillemer, Cambridge, MA: Harvard University Press,
copyright 1984 by the President and Fellows of Harvard College.
{Jobs}0688jw/makeup/688ch38.3d

644 PHARMACOEPIDEMIOLOGY

Figure 38.2. Funnel plot for published (open boxes) and unpublished (closed triangles) studies combined: analysis of
data from the review by Devine and Cooks'112 of psychoeducational programs for surgical patients. Reprinted by
permission of the publishers from Summing Up: the Science of Reviewing Research by Richard J. Light and David B.
Pillemer, Cambridge, MA: Harvard University Press, copyright 1984 by the President and Fellows of Harvard College.

assumptions regarding the nature of the relation-
ship between study findings and publication
probabilities.)
An additional methodologic caution relates to
the use of random effect models for combining
results. When the results of the studies being
analyzed are heterogeneous and a random effect
model is being used to combine those results, one
of the properties of the model, described above, is
to assign relatively higher weights to small studies
than would otherwise be assigned by more
traditional methods of combining data. If publica-
tion bias is a problem in a particular data set, one
consequence implied by the funnel plot is that
small studies would tend to show larger effects
than large studies. Thus, if publication bias is
present, one of the reasons for heterogeneity of
study results is that the small studies show
systematically larger effects than the large studies.
The assignment of higher relative weights to the
small studies could, when publication bias is
present, lead to a biased summary result.38 In fact,
this appears to be exactly the situation presented
by Poole and Greenland in an examination of
studies of water chlorination and cancer.35 Ran-
dom effect summary estimates of the relative risk
for various cancers were larger than corresponding
fixed effect summaries. This was apparently due to
the assignment of higher relative weights to small
studies which, in this case, showed relatively larger
effects, that may not be representative of the
findings of all small studies.
A proposed solution to the problem of publica-
tion bias is the use of prospective registration of
studies at their inception, prior to the availability
of results.47, 123 ± 129 Going one step further, several
prospectively planned meta-analyses are either
being planned or have been conducted.130 ±133
CASE STUDIES OF APPLICATIONS OF
META-ANALYSIS
Investigation of Adverse Effects
As mentioned earlier, the investigation of adverse
or unwanted effects of existing therapies is an
important application of meta-analysis. As dis-
cussed in Chapter 3, adverse events associated with
pharmaceutical products are often so uncommon
as to be difficult to study. In particular, the usual
premarketing randomized studies frequently have
too few patients to provide any useful information
{Jobs}0688jw/makeup/688ch38.3d
THE USE OF META-ANALYSIS IN PHARMACOEPIDEMIOLOGY
on the incidence of uncommon adverse events. By
the same token, individual studies may have low
statistical power to address particular questions.
Meta-analysis provides the benefit of vastly
increased statistical power to investigate adverse
events. In fact, since 1982, the safety evaluation of
drugs in the US has included pooled analyses.134
The assessment of the excess risk of gastrointest-
inal side-effects associated with nonsteroidal anti-
inflammatory drugs (NSAIDs) provides an excel-
lent example of a situation in which meta-analysis
has been helpful. Four different meta-analytic
approaches to this problem will be reviewed here.
Chalmers and colleagues examined data from
randomized trials of NSAIDs.39 They argued that
the typical epidemiologic approaches to investigat-
ing NSAIDs as risk factors for gastrointestinal
side-effects, i.e., cohort or case ±control studies,
are subject to too many potential biases. Rando-
mized trials, on the other hand, would provide
internally valid comparisons of NSAID users to
nonusers. Presumably, although not stated expli-
citly, the combination of results from numerous
studies, with varied entry and exclusion criteria,
would alleviate the problem of the potential lack of
generalizability from patients enrolled in a parti-
cular trial. The pooling of results from numerous
studies would permit the assessment of rare events.
The authors performed a meta-analysis of
randomized trials, excluding trials involving topi-
cal usage of NSAIDs, those that examined
pharmacological endpoints only, studies of new-
borns, trials involving fewer than four days of
treatment, trials in which patients were taking
NSAIDs within the three days before randomiza-
tion, and trials of drugs for dysmenorrhea (because
of the short duration of the drug regimen and the
confounding gastrointestinal symptoms from the
dysmenorrhea). The meta-analysis was limited to
those trials in which the anti-inflammatory drug
was compared with a placebo, no drug, or a drug
with no anti-inflammatory property. Photocopies
of the ``Methods'' sections of 525 potentially
relevant studies (blinded as to author, journal,
and time and place of study, as well as all allusions
to results) were read by two independent observers
who determined inclusion suitability according to
the above criteria.
645
Data were extracted for the following endpoints:
nausea, indigestion or dyspepsia, gross gastroin-
testinal bleeding, suspected ulcer only, proven
ulcer, gastric side-effects, and unspecified gastric
side-effects. Factors that might be related to the
incidence of these endpoints were also extracted
from the studies: disease under study, drug
ingested, dose and duration of drug, age of
patients, sex of patients, and date of publication.
The data were analyzed by crude pooling (i.e.,
ignoring the stratification by study and simply
collapsing over studies), by unweighted averaging
of the within-study risk differences, and by a
weighted average of the risk differences. Addi-
tional analyses were performed to determine
whether any factors seemed to be associated with
a study showing a harmful effect of NSAIDs.
As a methodologic aside, the authors examined
inter-reader disagreements. Overall, a disagree-
ment rate of 19% was observed for the final
decision on inclusion or exclusion of studies. These
disagreements were resolved in conference.
There were 100 randomized trials of nonaspirin
NSAIDs included in the final analysis, containing
123 comparisons with a no-treatment control
group, which usually received a placebo. A total
of 12 853 patients were included in these trials,
with a mean duration of treatment of about 67
days (median 21 days) and a mean age of 46 years.
For the sake of brevity, the aspirin trials will not
be discussed here. The data revealed a generally
low risk of gastrointestinal side-effects. For
example, only two patients were reported with
proven ulcers out of 6460 treated patients, with
none in the controls. In the ten studies explicitly
mentioning gross upper-gastrointestinal hemor-
rhage, the risk was 8=1103 (0.73%) in the control
patients and 24=1157 (2.1%) in treated patients,
giving a crude relative risk of 2.8. The length of
followup for these ten studies was not specifically
mentioned by the authors of the meta-analysis.
However, the analysis of duration of therapy
showed that duration was longer for studies
showing a harmful effect of NSAIDs (geometric
mean = 81 d) than for studies showing no effect of
NSAIDs (geometric mean = 25 d) for the gross
hemorrhage endpoint, consistent with a duration ±
response effect.
{Jobs}0688jw/makeup/688ch38.3d
646 PHARMACOEPIDEMIOLOGY
This meta-analysis was faced with some inter-
esting statistical and other methodologic ques-
tions. There were numerous studies that did not
explicitly mention side-effects in general or did not
mention particular side-effects, even though others
were mentioned. The authors chose to do a kind of
sensitivity analysis by analyzing all studies, assum-
ing that the risk of an unreported side-effect was
zero, and separately analyzing results from only
those studies explicitly mentioning a particular
side-effect.
Another issue was the extensive number of
studies with no occurrences of a particular end-
point in either the treated or the control group.
The usual pooling procedures, e.g., the Mantel ±
Haenszel procedure,87 essentially ignore such
studies, since they contribute no information,
under one interpretation, concerning the common
odds ratio. On the other hand, if over 90 of 100
separate trials report no proven ulcers in either the
treated or the control groups, then another
interpretation of those results is that the risk of
an ulcer is fairly low. Chalmers and colleagues
chose to work with risk differences to address this
issue, allowing studies with no events in either
group to enter the calculations. This is the type of
situation considered by Deeks and colleagues,90
whose results suggest that the one-step method
would have been the most appropriate for these
studies with frequent occurrence of zero cells.
Another meta-analytic approach to the problem
of side-effects of NSAIDs was used by Gabriel and
colleagues,135 who examined the results of 16 non-
experimental studies (nine case ± control and seven
cohort) of serious gastrointestinal complications
related to use of NSAIDs. The studies had to have
a comparison group and provide an estimate of
risk for serious gastrointestinal complications
(defined as bleeding, perforation, or other adverse
gastrointestinal events resulting in hospitalization
or death) in NSAID users compared with nonu-
sers, regardless of underlying disease. They ex-
cluded studies if the primary goal was to assess
effectiveness.
The odds ratio found by these authors for
gastrointestinal bleeding, based on nine studies
reporting this endpoint, was 2.39 (CI 2.11, 2.70).
The authors performed separate analyses for case ±
control and cohort studies. Although these sepa-
rate summaries are only reported graphically and
the exact values are difficult to read, the summary
odds ratio from the cohort studies is clearly closer
to unity than the result from the case ± control
studies. These authors also found that the size of
the odds ratio was related to the duration of
NSAID use. Interestingly, though, the highest
odds ratios were obtained from studies in which
the duration of NSAID consumption was less than
1 month. (Note: Gabriel et al. only presented this
finding without adjustment for study design, case ±
control versus cohort. Although they performed a
multiple regression to examine interstudy hetero-
geneity, the findings of that model with respect to
the individual potential sources of heterogeneity
were not presented. It is possible that the studies
with under one month of NSAID use were also
predominantly case ±control studies, but that
cannot be determined from their paper, leaving
the underlying source of heterogeneity somewhat
ambiguous.)
The consistency of results for gastrointestinal
bleeding between the two meta-analyses is of
interest and lends some support to a causal
association. Several points are important in con-
sidering the above results. In a cohort study of
gastrointestinal bleeding and NSAIDs, Carson
and colleagues136 found a quadratic duration±
response relation. They argue that this is compa-
tible with an increasing risk with increasing
duration of NSAID use, as suggested by Chalmers
et al.,39 until many of those patients who would
develop gastrointestinal bleeding from NSAIDs
were removed from the cohort and then the risk
declined. This reasoning may explain the appar-
ently anomalous finding by Gabriel et al.135 of
highest odds ratios for studies with less than one
month of NSAID use.
Bollini and colleagues137 also examined epide-
miologic studies that investigated the association
between NSAIDs and severe upper gastrointest-
inal tract disease, including hematemesis, melena,
peptic ulcer, ulcer perforation, and death attribu-
table to these outcomes. The studies had to
compare groups exposed and unexposed to
NSAIDs. Of the 34 studies they examined, seven
were cohort, eight case ± control with community
{Jobs}0688jw/makeup/688ch38.3d
THE USE OF META-ANALYSIS IN PHARMACOEPIDEMIOLOGY
controls, and 19 case ± control with hospital con-
trols. The type of study design was associated with
varying estimates of the relative risk. Case ± control
studies with hospital controls had the highest
average relative risk (4.4 [3.3 ± 6.0]) and the cohort
studies had the lowest (2.0 [1.2 ±3.2]). They found
that studies with satisfactory methods yielded on
average a lower relative risk (2.6 [1.8 ±3.9]) as
compared with studies whose methods were
unsatisfactory (4.2 [3.1 ±5.6]).
In perhaps the most comprehensive and clini-
cally useful of the systematic reviews in the area of
NSAIDs side effects, Henry and colleagues138
addressed the issue of comparative relative risks
of serious gastrointestinal complications with
individual NSAIDs. Their stated motivation for
this approach was that one strategy for reducing
NSAID toxicity in populations would be to
choose, as first line therapy, a drug and dose with
a comparatively low risk of gastrointestinal side-
effects.
The authors used meta-analytic methods to
examine the range of relative risks for particular
NSAIDs and explore the extent to which differ-
ences in toxicity could be related to different doses,
or to different susceptibility among patients
receiving the various drugs. To do this, they
identified case ± control or cohort studies of rela-
tionships between use of specific NSAIDs in the
community and development of serious peptic
ulcer complications requiring hospital admission.
In estimating pooled relative risks, analyses were
restricted to studies that compared another drug
with ibuprofen as the reference. They used
unadjusted relative risks based on 2  2 tables in
the pooling.
The authors found 12 studies examining 14
NSAIDs, including two unpublished reports.
Eleven of the studies were case ±control studies.
The estimated relative risks for specific drugs
versus ibuprofen ranged from 1.6 (95% CI 1.0, 2.5)
for fenoprofen, to 9.2 (95% CI 4.0, 21) for
azapropazone. All of the relative risks were
significantly greater than 1.0. Using a weighted
ranking system, which incorporated study size into
the weights, the authors found that ibuprofen had
the lowest rank (least toxicity), followed by
diclofenac. Aspirin and naproxen had intermediate
647
risks, while azapropazone, tolmetin, and ketopro-
fen had the highest risks. High dose ibuprofen (i.e.,
greater than 1600 mg daily) was also associated
with an elevated relative risk.
It is important to keep in mind that the
conclusions reached by Henry et al. were based
on indirect comparisons of the various drugs with
ibuprofen. They claimed to find little evidence that
the relative rankings were due to confounding by
patient susceptibility. Despite any shortcomings of
their approach, as the authors point out, clinical
and regulatory decisions have to be made on some
type of scientific basis, and these are the only data
available. Risks need to be weighed against
benefit, and the authors highlight the known
variability across patients in clinical response to
particular drugs. Thus, it seems that this systema-
tic review provided useful information for clinical
decision making.
In qualitative reviews of the literature on the
gastrointestinal side effects of NSAIDs, Taragin et
al.139 and Carson and Strom140 point out differ-
ences in study designs that could lead to differ-
ences in results. For example, bleeding could be
defined as all bleeding, fatal bleeding, bleeding
requiring hospitalization, bleeding requiring trans-
fusions, or bleeding requiring surgery. Several
procedures exist for the detection of gastrointest-
inal bleeding. The clinical relevance of the different
methods is sometimes unclear. Case ±control
studies may show higher odds ratios because of
the likelihood of recall bias; patients with bleeding
requiring hospitalization might be more likely to
recall NSAID use than controls, particularly if
probing by interviewers, or by health care provi-
ders prior to interview, is more extensive for cases
than for controls. This possibility is supported by
the data from the meta-analysis of Gabriel et al.
Cohort studies based on claims data, such as that
conducted by Carson and colleagues136 described
above, sometimes use unvalidated outcomes. To
the extent that false events may be documented for
both the exposed and unexposed cohorts, the
relative risk observed in such studies would show
less of an effect of exposure. Of course, these
cohort studies may exaggerate the apparent effect
of exposure if spurious diagnoses of gastrointest-
inal events are more likely to occur when a patient
{Jobs}0688jw/makeup/688ch38.3d
648 PHARMACOEPIDEMIOLOGY
has a history of NSAID use. Further variability
may be generated among study results by the
inclusion of many different kinds of NSAID, some
of which may have more potential to cause
gastrointestinal side-effects than others.
Thus, another benefit of meta-analysis is the
ability to examine findings according to study
characteristics and study design, leading to hy-
potheses about subgroups or particular therapies
of special interest and suggestions for the design of
subsequent studies. Meta-analysis can quantify
differences related to study design that the tradi-
tional review can only observe in qualitative terms.
There are numerous other examples of the
application of meta-analysis to the evaluation of
adverse effects of pharmaceutical therapies. These
include the following.
1. Two meta-analyses have been published on the
effects of prophylactic lidocaine in acute
myocardial infarction.141 ± 143 These studies
showed that, although lidocaine effectively
prevented ventricular fibrillation,143 there
seemed to be an excess in mortality among
those patients randomly allocated to lidocaine
compared with those allocated to placebo. In a
related paper, using meta-analytic regression
methods, Antman and Berlin144 calculate that,
given the low baseline incidence of ventricular
fibrillation in the current coronary care unit
environment, 400 patients would require treat-
ment with lidocaine to prevent a single episode
of ventricular fibrillation. Considering this
estimate in addition to the possibly increased
risk of mortality, the authors suggest that
prophylactic lidocaine should not be given
routinely.
2. In a meta-analysis of randomized trials regard-
ing the efficacy and safety of quinidine therapy
for the maintenance of sinus rhythm after
cardioversion,145 the authors show that quini-
dine is, indeed, effective at maintaining sinus
rhythm, but that mortality seems to be elevated
in quinidine patients (mortality odds ra-
tio = 2.98, 95% CI 1.1, 8.3, based on 12 deaths
among patients randomized to quinidine versus
only three among patients randomized to
placebo). In a subsequent paper, the authors
examine the relationship between study design
and outcome.146 The nonrandomized studies
tended to show less benefit of quinidine with
respect to maintenance of sinus rhythm com-
pared with the randomized studies. The odds
ratios for mortality also varied according to
study design, although there were few deaths
overall: OR = 3.5 (1.0, 12.4) for randomized
studies; OR = 9.9 (0.8, 123.2) for nonrando-
mized studies. The overall mortality risk was
2.0% (34=1709) for quinidine-treated patients
and 0.6% (4=681) for all control patients.
3. A third example is the meta-analysis of
nonexperimental studies of oral contraceptives
and the risk of breast cancer discussed above.20
An association between increasing odds ratio
and increasing duration of oral contraceptive
use was found in case ±control studies in which
the cases were mostly premenopausal (defined
as age limit of 45 years old or less). In a
subsequent methodologic paper, Berlin and
colleagues3 use these same data to show that
the magnitude of the odds ratio depends not
only on the menopausal status of the cases but
on the calendar years during which cases were
accrued, presumably because of the changing
formulations of oral contraceptives.
New Indications for Existing Therapies
Meta-analysis has also been used to assess the
effectiveness of existing therapies for new indica-
tions. As an example, the efficacy of antilympho-
cyte antibodies in the perioperative period of
cadaveric kidney transplantation (induction ther-
apy) had not, until recently, been conclusively
demonstrated. Individual studies, both rando-
mized and nonexperimental, had failed individu-
ally to show a significant benefit of induction
therapy with respect to allograft survival. Szczech
and colleagues performed a meta-analysis of the
published data from the randomized trials of
induction therapy147 in adults receiving cadaveric
renal transplants. That analysis, using survival
analytic methods on the group level (published)
data, showed a statistically significant 31% lower
{Jobs}0688jw/makeup/688ch38.3d
THE USE OF META-ANALYSIS IN PHARMACOEPIDEMIOLOGY
rate of allograft failure at two years in patients
receiving induction therapy.
In a subsequent analysis of the individual
patient data from five of the seven randomized
trials of induction therapy, Szczech and colleagues
examined the effect of induction therapy beyond
two years and in subgroups of patients with risk
factors for early allograft failure.148 The subgroup
analyses are examined in the next section. The five
studies included in the individual patient analyses
gave results for the two year analysis virtually
identical to those obtained from the full set of
seven studies using the published data, i.e., a
relative rate of 0.69 favoring induction therapy.
When extended to five years, the rate of allograft
survival was 69.0% in patients receiving induction
therapy and 64.4% in those not receiving induc-
tion therapy (p = 0.13). Thus, the overall benefit
demonstrated at two years was smaller and it was
no longer significant at five years.
Differential Effects Among Subgroups of
Patients
In the analysis of individual patient data by
Szczech and colleagues, the authors were able to
examine the specific effects of induction therapy in
subgroups of patients at high risk for allograft
failure. Before proceeding to analyses within
particular subgroups, the authors tested statistical
interactions between each of the relevant patient
characteristics and induction therapy. One of the
patient characteristics of interest was the panel
reactive antibody level (PRA), an indicator of
immune system presensitization. Patients with
PRA levels less than 20% were considered
unsensitized, while those with PRA of 20% or
higher were considered presensitized. At two years,
the effect of induction therapy differed in pre-
sensitized and unsensitized patients (p = 0.03 for
interaction). The rate ratio at two years was 0.12
(CI 0.03 ± 0.44, p = 0.002) in presensitized patients
(85 patients with 15 failures) and 0.74 (CI 0.50 ±
1.09, p = 0.13) in unsensitized patients (511
patients with 100 failures). This interaction was
still significant at five years (p = 0.009 for inter-
649
with 33 failures) and 0.97 (CI 0.71 ± 1.32, p > 0.2) in
unsensitized patients (510 patients with 163 fail-
ures). The authors found no other significant
interactions between induction therapy and any
other variable.148
Several advantages of meta-analysis, and parti-
cularly individual patient analyses, are demon-
strated by this example. The improved precision
provided by large numbers of patients is an
important benefit. Having individual level data
allowed an analysis that could go beyond the
simple, unadjusted analyses to which most meta-
analyses of published data are limited. The
availability of patient characteristics permitted
not only adjustment for those characteristics, but
also examination of subgroup effects in larger
numbers of patients than would typically be
included in a single trial. Although one might
wish to confirm these subgroup results in an
independent data set, the patient level analyses
strongly suggest that induction therapy is effective
in the 14% of patients who are presensitized. If
confirmed, these results could mean that induction
therapy could be targeted to the group in which it
is highly effective, while avoiding needless treat-
ment and potential toxicity in other patients.
In another example of a meta-analysis addres-
sing subgroup issues, Midgette and colleagues149
examined the use of intravenous streptokinase in
patients with suspected acute myocardial infarc-
tion. The authors found a summary relative risk of
0.72 (0.65, 0.79), favoring streptokinase, in pa-
tients with suspected acute anterior myocardial
infarction, and a summary risk ratio of 0.87 (0.76,
1.01) in those with a suspected inferior infarction.
The authors conclude that there is a protective
effect of intravenous streptokinase in anterior
infarction, but that the protective effect in inferior
infarction is smaller and less certain.
Selection From Among Several Alternative
Therapies
In a meta-analysis of therapies for the prevention
of supraventricular arrhythmias after coronary
bypass surgery, Andrews et al.150 looked separately
action), with a rate ratio of 0.20 (CI 0.09 ± 0.47, at verapamil, digoxin, and -adrenoceptor block-
p < 0.001) in presensitized patients (85 patients ers as prophylactic agents. Only randomized trials
{Jobs}0688jw/makeup/688ch38.3d

650 PHARMACOEPIDEMIOLOGY

were included. Neither digoxin nor verapamil
reduced the risk of supraventricular arrhythmias
after coronary artery bypass surgery (digoxin:
OR = 0.97, CI = 0.62, 1.49; verapamil:
OR = 0.91, CI = 0.57, 1.46). The risk of a
supraventricular arrhythmia in patients treated
benefit from intravenous administration.152 The
authors of the meta-analysis speculate that the
particular study failed to achieve therapeutic levels
of heparin in the subcutaneous group.
A somewhat different approach to subgroup
analyses has emerged in recent years. This strategy
with -blockers was dramatically reduced views the risk of events in the control group of a
(OR = 0.28, CI = 0.21, 0.36), although significant trial (baseline risk) as a general indicator of
heterogeneity among the study results was present. severity of disease in the treated population. The
The authors explored this heterogeneity by exam- relationship between treatment benefit and base-
ining separately studies of different -blockers,
and by summarizing separately preoperative and
postoperative treatment. While these separate
summaries suggested varying degrees of hetero-
geneity within subgroups of studies, all of the
summaries showed statistically significant benefits
line risk can then be estimated, i.e., examining
whether there is an interaction between treatment
and baseline risk.101, 102, 153 A number of statistical
issues arise in such analyses, including the inherent
association induced by regressing treatment bene-
fit (e.g., the log relative risk, which is calculated
of -blockers. The authors drew no firm conclu-
sions from the subgroup analyses other than to
suggest directions for future research.
As another example, for several decades, hepar-
in has been used as the primary antithrombotic
drug for the initial treatment of venous throm-
boembolism. There has been some controversy
over the optimal mode of administration of
heparin: intermittent intravenous, continuous in-
travenous, or subcutaneous injection. While con-
tinuous infusion has been shown to be safer than
intermittent injection, and equally effective, con-
tinuous infusion has disadvantages, such as the
need for hospitalization for most patients, possibly
prolonged immobilization, enhanced risk for sepsis
related to the infusion cannula, and possible
increased cost.151 Hommes and colleagues per-
formed a meta-analysis of randomized trials of
intravenous versus subcutaneous heparin admin-
istration in the initial treatment of deep vein
thrombosis.151 They found eight studies meeting
their inclusion criteria. The overall summary
relative risk for efficacy was 0.62 (0.39, 0.98),
indicating a benefit from the use of subcutaneous
compared with intravenous administration. The
analysis of safety (i.e., risk of major hemorrhage)
also showed a modest benefit of subcutaneous
injection (relative risk = 0.79, CI 0.42, 1.46). In the
analysis of efficacy, as in others described above,
there was highly significant among study hetero-
geneity. The source of this heterogeneity was
apparently a single study showing a significant
using the baseline risk) on baseline risk, and the
fact that the baseline risk, except in very large
studies, is affected by sampling variability. These
approaches all use group level (published) data,
examining whether the risk in the population is
associated with the magnitude of treatment bene-
fit. This may not necessarily yield the same result
as looking at the interaction on an individual
patient basis between treatment and individual level
estimates of risk. It is also clearly different
clinically from examining specific patient charac-
teristics, as opposed to calculating estimates of risk
that depend on several characteristics. Information
may be lost by using multivariable risk estimates,
as opposed to potentially biologically specific
patient characteristics.
Saving Time and Money If You Believe a
Meta-Analysis
One of the potential benefits of meta-analysis is the
potential to shorten the time between a medical
research finding and the clinical implementation of
a new therapy. This is a concern not only for the
development of new drugs, but for the exploration
of new indications for existing therapies. A group
at Harvard University has advocated the routine
use of what they have termed ``cumulative meta-
analysis,'' i.e., performing a new meta-analysis
each time the results of a new clinical trial are
published.40, 154 Antman et al.40 applied this
technique in combination with a classification
{Jobs}0688jw/makeup/688ch38.3d

THE USE OF META-ANALYSIS IN PHARMACOEPIDEMIOLOGY 651

scheme of the treatment recommendations for
myocardial infarction found in review articles and
textbook chapters. They found many discrepancies
between the evidence contained in the published
randomized trials and the timeliness of the
recommendations.
As an example, Antman and colleagues ana-
summary odds ratio arbitrarily estimated after 17
trials had been completed. On the right hand side
of Figure 38.3, the same data are presented as a
cumulative meta-analysis, with an updated sum-
mary estimate calculated after the completion of
each new trial. The cumulative meta-analysis
clearly shows that the updated pooled estimate
lyzed data from 17 trials of -blockers for the
prevention of death in the years following a
myocardial infarction.40 On the left hand side of
Figure 38.3, reproduced from their paper, the data
are presented as a traditional meta-analysis, with
individual study results presented along with the
became statistically significant in 1977 and has
remained so ever since.
The process of cumulative meta-analysis was
applied by these authors to eight therapies for
acute myocardial infarction. In five of the six
instances in which the cumulative meta-analyses

Figure 38.3. Results of 17 randomized control trials of the effects of oral -blockers for secondary prevention of mortality
in patients surviving a myocardial infarction presented as two types of meta-analysis. On the left is the traditional one,
revealing many trials with nonsignificant results but a highly significant estimate of the pooled results on the bottom of the
panel. On the right, the same data are presented as cumulative meta-analyses, illustrating that the updated pooled
estimate became statistically significant in 1977 and has remained so up to the present. Note that the scale is changed on
the right-hand graph to improve clarity of the confidence intervals. Reprinted with permission from Antman et al., Journal
of the American Medical Association; July 8, 1992; volume 268; pages 240 ± 248. Copyright 1992, American Medical
Association.
{Jobs}0688jw/makeup/688ch38.3d

652 PHARMACOEPIDEMIOLOGY

revealed the therapies to be of statistically sig-
nificant benefit in reducing in-hospital mortality, it
was several years before experts recommended the
therapy with any consistency. An important
example was thrombolytic therapy, which did not
are often suggested by nonexperimental studies,
including cohort and case±control studies and
nonrandomized phase 2 clinical trials. The results
of these studies are not always confirmed by
subsequent, properly designed randomized trials.
begin to be recommended by more than half the For example, consider the case of -carotene in the
experts, even for specific indications, until 13 years prevention of cancer. A series of observational
after the cumulative meta-analysis would have studies (see Ziegler et al.157 for a review) examined
shown therapy to be effective. Six years passed the relation between dietary intake of foods rich in
between the publication, in a major journal,40, 155 -carotene and the risk of lung cancer. Overall, they
of the first meta-analysis showing an impressive showed a relatively consistent association between
reduction in mortality by thrombolytic therapy
and the year in which the majority of the experts
whose opinions were studied by the authors
recommended it for routine or specific use. In
1985, a 20% reduction in mortality was established
at the p < 0:001 level (OR = 0.78; CI 0.69, 0.90). A
total of 14 reviews after 1985 did not mention the
treatment or felt that it was still experimental. The
authors concluded that identifying and interpret-
ing the therapeutic trials in a given field is
extremely difficult, so that clinical experts need
access to better databases and new statistical
techniques (such as cumulative meta-analysis).
Some caution may be advised in interpreting
cumulative meta-analyses. The issue of multiple
statistical tests, for example, is considered by some
to be an important consideration. The problem is
that testing and estimation procedures may need to
make adjustments for the increased probability of
a spurious positive finding (type I, or  error)
introduced by the use of repeated statistical
tests.100, 156 At the least, one might wish to consider
using a more stringent criterion for statistical
significance than the traditional p < 0:05 cutoff.
Another consideration is that estimates of treat-
ment effect may not be stable over time, perhaps
diets rich in -carotene and reduced risk of lung
cancer. Subsequent randomized trials of this specific
nutrient as a supplement have failed to confirm a
protective effect against lung cancer.158±160
THE FUTURE
The examples above have raised several important
issues that will need to be addressed in the future.
A set of issues not fully addressed above relates to
the availability of individual level data. From the
above examples, it becomes increasingly apparent
that the pursuit of questions about subgroups of
patients is often an informative and important
element of a well conducted meta-analysis, at least
for certain therapies. One should certainly exercise
due caution in the interpretation of subgroup
effects, emphasizing those that are specified a
priori with biological justification. By assembling
large numbers of patients, meta-analysis can at
least begin to address the problems related to
statistical instability of subgroup effects. It is too
often the case, however, that results are not
reported separately for subgroups of patients.
Typically, some trials will exclude particular
due to changing clinical environments. In the -
blocker example, there is an apparent ``drift'' of
the effect estimate back toward the null in more
recent years, i.e., treatment appears to be less
effective in the most recent studies. Thus, it may be
important to re-evaluate therapies as other treat-
ment strategies evolve for the same conditions.
A final caution with regard to interpreting
cumulative meta-analyses relates to the continuing
need for well designed randomized controlled trials.
New indications for existing therapies, for example,
patients while others will not exclude them. At
the level of grouped data from published reports,
one is faced with analyzing the two groups of
studies separately as the only way of addressing
the subgroup question, or using meta-regression
techniques on what amounts to ecological data. As
a trivial hypothetical example, suppose one wanted
to perform separate analyses of the effect of
treatment X in men and women. Among the
existing randomized trials, six exclude women and
four do not, but the four also include men. Ideally,
{Jobs}0688jw/makeup/688ch38.3d
THE USE OF META-ANALYSIS IN PHARMACOEPIDEMIOLOGY
one would like to obtain information on the effect
in men alone from all of the ten trials, since all
include men. Similarly, one could obtain a
separate estimate of the effect of treatment in
women from the four studies including women. It
is possible to use the group level data only to show,
for example, that studies that include women tend
to show different effects than studies excluding
women, but one cannot perform a separate
analysis of women. One might alternatively regress
the treatment effect measure (e.g., log relative risk)
against the percent male (or female), but that is
still less satisfactory than obtaining patient level
data. For further practical discussions of the use of
individual patient data, see Stewart and Clarke161
and Stewart and Parmar.162 Mechanisms for the
sharing of person level data need to be promoted.
In the development of cumulative meta-analysis,
some of the most important issues will be
philosophical ones. Some of the same issues apply
to the approval process for new drugs. How much
evidence is required before a therapy can be
accepted as efficacious? Should we require the
existence of a certain minimum number of trials
showing a statistically significant benefit of a
therapy? Suppose ten studies all show a 20% (or
thereabouts) reduction in mortality in patients
treated with drug X compared to placebo, but
none of the individual studies shows a statistically
significant effect. If the combined analysis shows a
highly statistically significant 20% reduction in
mortality due to treatment X, and 20% is
considered clinically significant, should the com-
bined analysis be sufficient evidence for the
acceptance of drug X as beneficial? What would
an additional, large clinical trial contribute?
In this context, it is worth noting that several
empirical studies have examined discrepancies
between large trials and meta-analyses of the same
therapies.163 ± 168 The assumption made by some of
the authors of these studies, that larger studies are
necessarily better studies, may not be valid.
Replication of a finding by independent studies
must certainly be a key element to establishing
efficacy, as compared with a single trial. Large
trials may also be poorly designed. For example,
for BCG vaccine, a huge trial using passive
followup, and therefore missing (by the authors'
653
own admission) at least 50% of all cases of
tuberculosis, failed to show the protective effect
found in a number of other studies with more
complete followup.169, 170
When there is little or no heterogeneity of results
among trials, and the likelihood of serious pub-
lication bias is minimal, one might be willing to
accept meta-analytic evidence as helping to estab-
lish effectiveness. It is less obvious what to do with
the results of a meta-analysis when there is
substantial heterogeneity. If the heterogeneity is
adequately explained in the analysis in terms of
subgroup effects, or trial quality, meta-analysis
might still be an acceptable part of demonstrating
effectiveness, but such a conclusion might be
conditional on the type of patient or other factors.
Similarly, the technique of cumulative meta-
analysis could be applied to the analysis of adverse
events. As nonexperimental studies of adverse
effects are completed, the same approach could
be applied. The likelihood seems to be, however,
that such meta-analyses would be faced with much
more serious issues of heterogeneity of findings
than meta-analyses of randomized studies typically
have to confront. The acceptance of meta-analytic
results in this context might be extremely slow.
(Consider the slow pace, demonstrated by Antman
et al.,40 with which new therapies are accepted even
when the evidence is provided by randomized
trials.) Even if a meta-analysis of, for example, oral
contraceptive use and breast cancer risk were to
show a convincing, consistent duration± response
relationship, the issue of what to do with that
information is complex. If the relative risk for 10
years of use is 1.5, is that sufficient to warrant
removal of oral contraceptives from the market?
Would 2.5 be a sufficiently high relative risk? What
other factors, e.g., family history, etc., need to be
considered when prescribing oral contraceptives?
While these are clearly more general issues, not
restricted to the interpretation of meta-analyses,
the additional precision provided by meta-analyses
makes their interpretation all the more difficult.
The concept of prospective meta-analysis also
merits further attention. Along with registration of
trials, this closely related strategy has been
advocated as a means of avoiding publication
bias.130 ± 133 It may be possible, however, to go
{Jobs}0688jw/makeup/688ch38.3d
654 PHARMACOEPIDEMIOLOGY
beyond simply planning the logistics of multiple
trials and the collection of common data elements
to allow pooling of results upon the completion of
all trials. It may be possible to go further toward
planning the scientific questions to be addressed.
As a simple example, by regulation, sex and age
(adult versus pediatric) would need to be addressed
for a new analgesic. In addition, it would be
important to consider indication (emergency de-
partment, postoperative, etc.) and dose (cumula-
tive dose, daily dose, need for a loading dose, etc.).
How best to design the series of studies to address
all of these questions, either simultaneously or
sequentially, needs further consideration (see
Berlin and Colditz for a more complete discussion
of this issue).33
While there are no easy answers to many of
these questions, it is clear that meta-analysis will
play an increasingly important role in the for-
mulation of treatment and policy recommenda-
tions. Thus, the quality of the meta-analyses
performed is of the utmost importance and needs
to be reviewed by the scientific community in an
open, published forum. Meta-analyses, if they are
carefully interpreted in view of their strengths and
weaknesses, should prove to be extremely helpful
in pharmacoepidemiology research.
REFERENCES
1. DerSimonian R, Laird N. Meta-analysis in clinical
trials. Control Clin Trials 1986; 7: 177± 88.
2. Last JM. A Dictionary of Epidemiology. New York:
Oxford University Press, 1988.
3. Berlin JA, Longnecker MP, Greenland S. Meta-
analysis of epidemiologic dose-response data.
Epidemiology 1993; 4: 218± 28.
4. Feinstein AR. Para-analysis, faute de mieux, and
the perils of riding on a data barge. J Clin
Epidemiol 1989; 42: 929± 35.
5. Shapiro S. Is meta-analysis a valid approach to the
evaluation of small effects in observational studies?
J Clin Epidemiol 1997; 50: 223± 29.
6. Egger M, Schneider M, Davey Smith G. Spurious
precision? Meta-analysis of observational studies.
Br Med J 1998; 316: 140± 4.
7. Thacker SB. Meta-analysis. A quantitative ap-
proach to research integration. J Am Med Assoc
1988; 259: 1685± 9.
8. Buffler PA. The evaluation of negative epidemio-
logic studies: the importance of all available
evidence in risk characterization. Regul Toxicol
Pharmacol 1989; 9: 34 ± 43.
9. Sacks HS, Berrier J, Reitman D, Ancona-Berk VA,
Chalmers TC. Meta-analyses of randomized con-
trolled trials. New Engl J Med 1987; 316: 450± 5.
10. L'Abbe KA, Detsky AS, O'Rourke K. Meta-
analysis in clinical research. Ann Intern Med
1987; 107: 224± 33.
11. Colditz GA, Burdick E, Mosteller F. Heterogeneity
in meta-analysis of data from epidemiologic
studies: a commentary. Am J Epidemiol 1995;
142: 371± 82.
12. Berlin JA. Invited commentary: benefits of hetero-
geneity in meta-analysis of data from epidemiolo-
gic studies. Am J Epidemiol 1995; 142: 383±7.
13. Dickersin K, Berlin JA. Meta-analysis: state-of-
the-science. Epidemiol Rev 1992; 14: 154± 76.
14. Mulrow CD. The medical review article: state of
the science. Ann Intern Med 1987; 106: 485± 8.
15. Oxman AD, Guyatt GH. Guidelines for reading
literature reviews. Can Med Assoc J 1988; 138:
697±703.
16. Chalmers I. Improving the quality and dissemina-
tion of clinical research. In: Lock S, ed., The Future
of Medical Journals: in Commemoration of 150
Years of the British Medical Journal. London:
British Medical Society, 1991; 127± 46.
17. Kass EH. Reviewing reviews. In: Coping with the
biomedical literature: a primer for the scientist and
the clinician. Warren KS, ed. New York: Praeger,
1981. 71 ± 91.
18. Huth EJ. Needed: review articles with more
scientific rigor. Ann Intern Med 1987; 106: 470± 1.
19. Canner PL. An overview of six clinical trials of
aspirin in coronary heart disease. Stat Med 1987; 6:
255±67.
20. Romieu I, Berlin JA, Colditz G. Oral contra-
ceptives and breast cancer. Review and meta-
analysis. Cancer 1990; 66: 2253± 63.
21. Longnecker MP, Berlin JA, Orza MJ, Chalmers
TC. A meta-analysis of alcohol consumption in
relation to risk of breast cancer. J Am Med Assoc
1988; 260: 652± 6.
22. Longnecker MP, Orza MJ, Adams ME, Vioque J,
Chalmers TC. A meta-analysis of alcoholic bev-
erage consumption in relation to risk of colorectal
cancer. Cancer Causes Control 1990; 1: 59 ± 68.
23. Wong O, Raabe GK. Critical review of cancer
epidemiology in petroleum industry employees,
with a quantitative meta-analysis by cancer site.
Am J Ind Med 1989; 15: 283± 310.
24. Frumkin H, Berlin J. Asbestos exposure and
gastrointestinal malignancy: review and meta-
analysis. Am J Ind Med 1988; 14: 79 ± 95 [published
erratum appears in Am J Ind Med 1988; 14: 493].
{Jobs}0688jw/makeup/688ch38.3d
THE USE OF META-ANALYSIS IN PHARMACOEPIDEMIOLOGY
25. Berlin JA, Colditz GA. A meta-analysis of physical
activity in the prevention of coronary heart disease.
Am J Epidemiol 1990; 132: 612±28.
26. Gray A, Berlin JA, McKinlay JB, Longcope C. An
examination of research design effects on the
association of testosterone and male aging: results
of a meta-analysis. J Clin Epidemiol 1991; 44:
671± 84.
27. Booth-Kewley S, Friedman HS. Psychological
predictors of heart disease: a quantitative review.
Psychol Bull 1987; 101: 343± 62.
28. Morgenstern H, Glazer WM, Niedzwiecki D,
Nourjah P. The impact of neuroleptic medication
on tardive dyskinesia: a meta-analysis of published
studies. Am J Public Health 1987; 77: 717± 24.
29. Greenland S. Quantitative methods in the review
of epidemiologic literature. Epidemiol Rev 1987; 9:
1 ± 30.
30. Greenland S, Longnecker MP. Methods for trend
estimation from summarized dose ± response data,
with applications to meta-analysis. Am J Epidemiol
1992; 135: 1301± 9.
31. Greenland S, Salvan A. Bias in the one-step
method for pooling study results. Stat Med 1990;
9: 247± 52.
32. Fleiss JL, Gross AJ. Meta-analysis in epidemiol-
ogy, with special reference to studies of the
association between exposure to environmental
tobacco smoke and lung cancer: a critique. J Clin
Epidemiol 1991; 44: 127± 39.
33. Berlin JA, Colditz GA. The role of meta-analysis in
the regulatory process for foods, drugs, and
devices. J Am Med Assoc 1999; 281: 830± 4.
34. O'Neill RT, Anello C. Does research synthesis
have a place in drug regulatory policy? Synopsis of
issues: assessment of efficacy and drug approval.
Clin Res Regul Affairs 1996; 13: 23 ± 9.
35. Poole C, Greenland S. Random-effects meta-
analyses are not always conservative. Am J
Epidemiol 1999; 150: 469±75.
36. Jones DR. Meta-analysis of observational epide-
miological studies: a review. J R Soc Med 1992; 85:
165± 8.
37. Einarson TR, Leeder JS, Koren G. A method for
meta-analysis of epidemiological studies. Drug
Intell Clin Pharm 1988; 22: 813± 24.
38. Greenland S. Invited commentary: a critical look
at some popular meta-analytic methods. Am J
Epidemiol 1994; 140: 290±6.
39. Chalmers TC, Berrier J, Hewitt P, Berlin J,
Reitman D, Nagalingam R, Sacks H. Meta-
analysis of randomized controlled trials as a
method of estimating rare complications of non-
steroidal anti-inflammatory drug therapy. Aliment
Pharmacol Ther 1988; 2: 9 ± 26.
40. Antman EM, Lau J, Kupelnick B, Mosteller F,
Chalmers TC. A comparison of results of meta-
655
analyses of randomized control trials and recom-
mendations of clinical experts. Treatments for
myocardial infarction. J Am Med Assoc 1992;
268: 240±8.
41. Begg CB, Berlin JA. Publication bias and dissemi-
nation of clinical research. J Natl Cancer Inst 1989;
81: 107±15.
42. Easterbrook PJ, Berlin JA, Gopalan R, Matthews
DR. Publication bias in clinical research. Lancet
1991; 337: 867± 72.
43. Dickersin K, Min YI, Meinert CL. Factors
influencing publication of research results. Fol-
low-up of applications submitted to two institu-
tional review boards. J Am Med Assoc 1992; 267:
374± 8.
44. Dickersin K. The existence of publication bias and
risk factors for its occurrence. J Am Med Assoc
1990; 263: 1385± 9.
45. Chalmers I. Underreporting research is scientific
misconduct. J Am Med Assoc 1990; 263: 1405± 8.
46. Chalmers TC, Frank CS, Reitman D. Minimizing
the three stages of publication bias. J Am Med
Assoc 1990; 263: 1392± 5.
47. Moher D, Berlin J. Improving the reporting
of randomised controlled trials. In: Non-
random reflections on health services research.
Maynard A, Chalmers I, eds., London: BMJ, 1997.
250± 71.
48. Dickersin K, Min YI. NIH clinical trials and
publication bias. Online J Curr Clin Trials 1993;
Doc. No. 50.
49. Stern JM, Simes RJ. Publication bias: evidence of
delayed publication in a cohort study of clinical
research projects. Br Med J 1997; 315: 640± 5.
50. Ioannidis JP. Effect of the statistical significance of
results on the time to completion and publication
of randomized efficacy trials. J Am Med Assoc
1998; 279: 281± 6.
51. Berlin JA, Begg CB, Louis TA. An assessment of
publication bias using a sample of published
clinical trials. J Am Stat Assoc 1989; 84: 381± 92.
52. Chalmers TC, Berrier J, Sacks HS, Levin H,
Reitman D, Nagalingam R. Meta-analysis of
clinical trials as a scientific discipline. II: Replicate
variability and comparison of studies that agree
and disagree. Stat Med 1987; 6: 733± 44.
53. Early Breast Cancer Trialists' Collaborative
Group. Treatment of Early Breast Cancer. Vol I.
Worldwide Evidence. New York, NY: Oxford
University Press, 1990.
54. Hetherington J, Dickersin K, Chalmers I, Meinert
CL. Retrospective and prospective identification of
unpublished controlled trials: lessons from a survey
of obstetricians and pediatricians. Pediatrics 1989;
84: 374±80.
55. Gotzsche PC. Reference bias in reports of drug
trials. Br Med J Clin Res Edn 1987; 295: 654± 6.
{Jobs}0688jw/makeup/688ch38.3d
656 PHARMACOEPIDEMIOLOGY
56. Egger M, Zellweger-Zahner T, Schneider M,
Junker C, Lengeler C, Antes G. Language bias in
randomised controlled trials published in English
and German. Lancet 1997; 350: 326± 9.
57. Moher D, Fortin P, Jadad AR, Juni P, Klassen T,
Le Lorier J, Liberati A, Linde K, Penna A.
Completeness of reporting of trials published in
languages other than English: implications for
conduct and reporting of systematic reviews.
Lancet 1996; 347: 363± 6.
58. Davidson RA. Source of funding and outcome of
clinical trials. J Gen Intern Med 1986; 1: 155± 8.
59. Hemminki E. Study of information submitted by
drug companies to licensing authorities. Br Med J
1980; 280: 833± 6.
60. Cho MK, Bero LA. The quality of drug studies
published in symposium proceedings. Ann Intern
Med 1996; 124: 485± 9.
61. Chalmers TC. Problems induced by meta-analyses.
Stat Med 1991; 10: 971 ±9.
62. Conn HO, Blitzer BL. Nonassociation of adreno-
corticosteroid therapy and peptic ulcer. New Engl J
Med 1976; 294: 473± 9.
63. Messer J, Reitman D, Sacks HS, Smith H Jr,
Chalmers TC. Association of adrenocorticosteroid
therapy and peptic-ulcer disease. New Engl J Med
1983; 309: 21 ± 4.
64. Conn HO, Poynard T. Adrenocorticosteroid ad-
ministration and peptic ulcer: a critical analysis. J
Chron Dis 1985; 38: 457± 68.
65. Chalmers TC. Meta-analysis in clinical medicine.
Trans Am Clin Climatol Assoc 1987; 99: 144± 50.
66. DerSimonian R. Parenteral nutrition with
branched-chain amino acids in hepatic ence-
phalopathy: meta analysis. Hepatology 1990; 11:
1083± 4.
67. Erikkson LS, Conn HO. Branched-chain amino
acids in hepatic encephalopathy. Gastroenterology
1990; 99: 604± 7 [published erratum appears in
Gastroenterology 1990; 99: 1547].
68. Naylor CD, O'Rourke K, Detsky AS, Baker JP.
Parenteral nutrition with branched-chain amino
acids in hepatic encephalopathy. A meta-analysis.
Gastroenterology 1989; 97: 1033± 42.
69. Dickersin K, Hewitt P, Mutch L, Chalmers I,
Chalmers TC. Perusing the literature: comparison
of MEDLINE searching with a perinatal trials
database. Control Clin Trials 1985; 6: 306± 17.
70. Poynard T, Conn HO. The retrieval of randomized
clinical trials in liver disease from the medical
literature. A comparison of MEDLARS and
manual methods. Control Clin Trials 1985; 6:
271± 9.
71. Bernstein F. The retrieval of randomized clinical
trials in liver diseases from the medical literature:
manual versus MEDLARS searches. Control Clin
Trials 1988; 9: 23 ± 31.
72. Chalmers I, ed. Oxford database of perinatal
trials. Oxford: Oxford University Press, 1988.
73. Huston P, Moher D. Redundancy, disaggregation,
and the integrity of medical research. Lancet 1996;
347: 1024± 6.
74. Detsky AS, Naylor CD, O'Rourke K, McGeer AJ,
L'Abbe KA. Incorporating variations in the
quality of individual randomized trials into meta-
analysis. J Clin Epidemiol 1992; 45: 255± 65.
75. Berard A, Bravo G. Combining studies using effect
sizes and quality scores: application to bone loss in
postmenopausal women. J Clin Epidemiol 1998; 51:
801±7.
76. Chalmers TC, Smith H Jr, Blackburn B, Silverman
B, Schroeder B, Reitman D, Ambroz A. A method
for assessing the quality of a randomized control
trial. Control Clin Trials 1981; 2: 31 ± 49.
77. Imperiale TF, McCullough AJ. Do corticosteroids
reduce mortality from alcoholic hepatitis? A meta-
analysis of the randomized trials. Ann Intern Med
1990; 113: 299± 307.
78. Prendiville W, Elbourne D, Chalmers I. The effects
of routine oxytocic administration in the manage-
ment of the third stage of labour: an overview of
the evidence from controlled trials. Br J Obstet
Gynaecol 1988; 95: 3 ± 16.
79. Mahon WA, Daniel EE. A method for the
assessment of reports of drug trials. Can Med
Assoc J 1964; 90: 565± 9.
80. Verhagen AP, de Vet HCW, de Bie RA,
Kessels AGH, Boers M, Bouter LM, Knipschild
PG. The Delphi list: a criteria list for quality
assessment of randomized clinical trials for
conducting systematic reviews developed by
Delphi consensus. J Clin Epidemiol 1998; 51:
1235 ± 41.
81. Moher D, Jadad AR, Tugwell P. Assessing the
quality of randomized controlled trials. Current
issues and future directions. Int J Technol Assess
Health Care 1996; 12: 195± 208.
82. Moher D, Jadad AR, Nichol G, Penman M,
Tugwell P, Walsh S. Assessing the quality of
randomized controlled trials: an annotated biblio-
graphy of scales and checklists. Control Clin Trials
1995; 16: 62 ± 73.
83. Greenland S. Quality scores are useless and
potentially misleading. Am J Epidemiol 1994; 140:
300±1.
84. Juni P, Witschi A, Block R, Egger M. The hazards
of scoring the quality of clinical trials: lessons
for meta-analysis. J Am Med Assoc 1999; 282:
1054± 60.
85. Schulz KF, Chalmers I, Hayes RJ, Altman DG.
Empirical evidence of bias. Dimensions of metho-
dological quality associated with estimates of
treatment effects in controlled trials. J Am Med
Assoc 1995; 273: 408± 12.
{Jobs}0688jw/makeup/688ch38.3d
THE USE OF META-ANALYSIS IN PHARMACOEPIDEMIOLOGY
86. Berlin JA, Miles CG, Cirigliano MD, Conill AM,
Goldmann DR, Horowitz DA, Jones F, Scott E,
Hanchak NA, Williams SV. Does blinding of
readers affect the results of metaanalyses? Results
of a randomized trial. Online J Curr Clin Trials
1997; Doc. No. 205.
87. Kleinbaum DG, Kupper LL, Morgenstern H.
Epidemiologic research: principles and quantitative
methods. New York: Van Nostrand Reinhold, 1982.
88. Rothman KJ. Modern epidemiology. Boston, MA:
Little, Brown, 1986.
89. Yusuf S, Peto R, Lewis J, Collins R, Sleight P. Beta
blockade during and after myocardial infarction:
an overview of the randomized trials. Prog
Cardiovasc Dis 1985; 27: 335± 71.
90. Deeks J, Bradburn M, Localio R, Berlin J. Much
ado about nothing: statistical models for meta-
analysis with rare events. 6th International Co-
chrane Colloquium, Baltimore, MD, 1998.
91. Berlin JA, Laird NM, Sacks HS, Chalmers TC. A
comparison of statistical methods for combining
event rates from clinical trials. Stat Med 1989; 8:
141± 51.
92. Fleiss JL. The statistical basis of meta-analysis.
Stat Methods Med Res 1993; 2: 121± 45.
93. Hardy RJ, Thompson SG. Detecting and describ-
ing heterogeneity in meta-analysis. Stat Med 1998;
17: 841± 56.
94. Berlin JA, Antman EM. Advantages and limita-
tions of metaanalytic regressions of clinical trials
data. Online J Curr Clin Trials 1994; Doc. No. 134.
95. Berkey CS, Hoaglin DC, Mosteller F, Colditz GA.
A random-effects regression model for meta-
analysis. Stat Med 1995; 14: 395± 411.
96. Vanhonacker WR. Meta-analysis and response
surface extrapolation: a least squares approach.
Am Stat 1996; 50: 294±9.
97. Walker AM, Martin-Moreno JM, Artalejo FR.
Odd man out: a graphical approach to meta-
analysis. Am J Public Health 1988; 78: 961± 6.
98. Hedges LV, Vevea JL. Fixed- and random-effects
models in meta-analysis. Psychological Methods
1998; 3: 486±504.
99. Carlin JB. Meta-analysis for 2  2 tables: a
Bayesian approach. Stat Med 1992; 11: 141± 58.
100. Whitehead A, Whitehead J. A general parametric
approach to the meta-analysis of randomized
clinical trials. Stat Med 1991; 10: 1665± 77.
101. Thompson SG, Smith TC, Sharp SJ. Investigating
underlying risk as a source of heterogeneity in
meta-analysis. Stat Med 1997; 16: 2741± 58.
102. McIntosh MW. The population risk as an ex-
planatory variable in research synthesis of clinical
trials. Stat Med 1996; 15: 1713± 28.
103. Larose DT, Dey DK. Grouped random effects
models for Bayesian meta-analysis. Stat Med 1997;
16: 1817± 29.
657
104. Smith TC, Spiegelhalter DJ, Thomas A. Bayesian
approaches to random-effects meta-analysis: a
comparative study. Stat Med 1995; 14: 2685± 99.
105. Eddy DM, Hasselblad V, Shachter R. An intro-
duction to a Bayesian method for meta-analysis:
the confidence profile method. Med Decis Making
1990; 10: 15 ± 23.
106. Eddy DM, Hasselblad V, Shachter R. Meta-
analysis by the confidence profile method: the
statistical synthesis of evidence. Boston, MA:
Academic, 1990.
107. Goodman SN. Meta-analysis and evidence. Control
Clin Trials 1989; 10: 188±204. [published erratum
appears in Control Clin Trials 1989; 10: 435].
108. Fleiss JL. Analysis of data from multiclinic trials.
Control Clin Trials 1986; 7: 267± 75.
109. Simon R. Overviews of randomized clinical trials.
Cancer Treat Rep 1987; 71: 3 ± 5.
110. Davey Smith G, Egger M, Phillips AN. Meta-
analysis. Beyond the grand mean? Br Med J 1997;
315: 1610± 14.
111. Light RJ, Pillemer DB. Summing up: the science of
reviewing research. Cambridge, MA: Harvard
University Press, 1984.
112. Devine EC, Cook TD. Effects of psycho-educa-
tional interventions on length of hospital stay: a
meta-analytic review of 34 studies. In: Light RJ,
ed., Evaluation Studies Review Annual, Vol. 8.
Beverly Hills, CA: Sage, 1983.
113. Rosenthal R. The file drawer problem and tolerance
for null results. Psychol Bull 1979; 86: 638±41.
114. Iyengar S, Greenhouse JB. Selection models and
the file-drawer problem. Stat Sci 1988; 3: 109± 17.
115. Gleser LJ, Olkin I. Models for estimating the
number of unpublished studies. Stat Med 1996; 15:
2493± 507.
116. Egger M, Davey Smith G, Schneider M, Minder C.
Bias in meta-analysis detected by a simple,
graphical test. Br Med J 1997; 315: 629± 34.
117. Begg CB, Mazumdar M. Operating characteristics
of a rank correlation test for publication bias.
Biometrics 1994; 50: 1088± 101.
118. Dear KBG, Begg CB. An approach for assessing
publication bias prior to performing a meta-
analysis. Stat Sci 1992; 7: 237± 45.
119. Hedges LV, Olkin I. Statistical Methods for Meta-
Analysis. Orlando, FL: Academic, 1985.
120. Vevea JL, Hedges LV. A general linear model for
estimating effect size in the presence of publication
bias. Psychometrika 1995; 60: 419± 35.
121. Hedges LV, Vevea JL. Estimating effect size under
publication bias: small sample properties and
robustness of a random effects selection model. J
Educ Behav Stat 1996; 21: 299± 332.
122. Givens GH, Smith DD, Tweedie RL. Publication
bias in meta-analysis: a Bayesian data-augmenta-
tion approach to account for issues exemplified in
{Jobs}0688jw/makeup/688ch38.3d
658 PHARMACOEPIDEMIOLOGY
the passive smoking debate. Stat Sci 1997; 12:
221± 50.
123. Meinert CL. Toward prospective registration of
clinical trials. Control Clin Trials 1988; 9: 1 ±5.
124. Simes RJ. Publication bias: the case for an
international registry of clinical trials. J Clin Oncol
1986; 4: 1529± 41.
125. Dickersin K. Report from the panel on the Case
for Registers of Clinical Trials at the Eighth
Annual Meeting of the Society for Clinical Trials.
Control Clin Trials 1988; 9: 76 ± 81.
126. Dickersin K. Why register clinical trials? Ð revis-
ited. Control Clin Trials 1992; 13: 170± 7.
127. Savulescu J, Chalmers I, Blunt J. Are research
ethics committees behaving unethically? Some
suggestions for improving performance and ac-
countability. Br Med J 1996; 313: 1390± 3.
128. Anonymous. Making clinical trialists register.
Lancet 1991; 338: 244± 5.
129. Chalmers I, Dickersin K, Chalmers TC. Getting to
grips with Archie Cochrane''s agenda. Br Med J
1992; 305: 786± 8.
130. Valsecchi MG, Masera G. A new challenge in
clinical research in childhood ALL: the prospective
meta-analysis strategy for intergroup collabora-
tion. Ann Oncol 1996; 7: 1005± 8.
131. Margitic SE, Morgan TM, Sager MA, Furberg
CD. Lessons learned from a prospective meta-
analysis. J Am Geriatr Soc 1995; 43: 435± 9.
132. Simes RJ. Prospective meta-analysis of cholesterol-
lowering studies: the Prospective Pravastatin Pool-
ing (PPP) Project and the Cholesterol Treatment
Trialists (CTT) Collaboration. Am J Cardiol 1995;
76: 122C± 126C.
133. Whitehead A. A prospectively planned cumulative
meta-analysis applied to a series of concurrent
clinical trials. Stat Med 1997; 16: 2901± 13.
134. Temple RJ. The regulatory evolution of the
integrated safety summary. Drug Information J
1991; 25: 485± 92.
135. Gabriel SE, Jaakkimainen L, Bombardier C. Risk
for serious gastrointestinal complications related to
use of nonsteroidal anti-inflammatory drugs. A
meta-analysis. Ann Intern Med 1991; 115: 787± 96.
136. Carson JL, Strom BL, Soper KA, West SL, Morse
ML. The association of nonsteroidal anti-inflam-
matory drugs with upper gastrointestinal tract
bleeding. Arch Intern Med 1987; 147: 85± 8.
137. Bollini P, Garcia Rodriguez L, Perez Gutthann S,
Walker AM. The impact of research quality and
study design on epidemiologic estimates of the
effect of nonsteroidal anti-inflammatory drugs on
upper gastrointestinal tract disease. Arch Intern
Med 1992; 152: 1289± 95.
138. Henry D, Lim LL-Y, Garcia Rodriguez LA, Perez
Gutthann S, Carson JL, Griffin M, Savage R,
Logan R, Moride Y, Hawkey C, Hill S, Fries JT.
Variability in risk of gastrointestinal complications
with individual non-steroidal anti-inflammatory
drugs: results of a collaborative meta-analysis. Br
Med J 1996; 312: 1563± 6.
139. Taragin MI, Carson JL, Strom BL. Gastrointest-
inal side effects of the nonsteroidal anti-inflamma-
tory drugs. Dig Dis 1990; 8: 269± 80.
140. Carson JL, Strom BL. The gastrointestinal toxicity
of the non-steroidal anti-inflammatory drugs. In:
Side-Effects of Anti-Inflammatory Drugs 3. Rain-
sford KD, Velo GP, eds. Boston, MA: Kluwer,
1992; 1 ± 8.
141. Hine LK, Laird NM, Hewitt P, Chalmers TC.
Meta-analysis of empirical long-term antiarrhyth-
mic therapy after myocardial infarction. J Am Med
Assoc 1989; 262: 3037± 40.
142. Hine LK, Laird N, Hewitt P, Chalmers TC. Meta-
analytic evidence against prophylactic use of
lidocaine in acute myocardial infarction. Arch
Intern Med 1989; 149: 2694± 8.
143. MacMahon S, Collins R, Peto R, Koster RW,
Yusuf S. Effects of prophylactic lidocaine in
suspected acute myocardial infarction. An over-
view of results from the randomized, controlled
trials. J Am Med Assoc 1988; 260: 1910± 16.
144. Antman EM, Berlin JA. Declining incidence of
ventricular fibrillation in myocardial infarction.
Implications for the prophylactic use of lidocaine.
Circulation 1992; 86: 764± 73.
145. Coplen SE, Antman EM, Berlin JA, Hewitt P,
Chalmers TC. Efficacy and safety of quinidine
therapy for maintenance of sinus rhythm after
cardioversion. A meta-analysis of randomized
control trials. Circulation 1990; 82: 1106± 16
[published erratum appears in Circulation 1991;
83: 714].
146. Reimold SC, Chalmers TC, Berlin JA, Antman
EM. Assessment of the efficacy and safety of
antiarrhythmic therapy for chronic atrial fibrilla-
tion: observations on the role of trial design and
implications of drug-related mortality. Am Heart J
1992; 124: 924± 32.
147. Szczech LA, Berlin JA, Aradhye S, Grossman RA,
Feldman HI. Effect of anti-lymphocyte induction
therapy on renal allograft survival: a meta-analysis.
J Am Soc Nephrol 1997; 8: 1771± 7.
148. Szczech LA, Berlin JA, Feldman HI. The effect of
antilymphocyte induction therapy on renal allo-
graft survival. A meta-analysis of individual
patient-level data. Anti-Lymphocyte Antibody
Induction Therapy Study Group. Ann Intern Med
1998; 128: 817± 26.
149. Midgette AS, O'Connor GT, Baron JA, Bell J.
Effect of intravenous streptokinase on early
mortality in patients with suspected acute myocar-
dial infarction. A meta-analysis by anatomic
location of infarction. Ann Intern Med 1990; 113:
{Jobs}0688jw/makeup/688ch38.3d
THE USE OF META-ANALYSIS IN PHARMACOEPIDEMIOLOGY
961± 8 [published erratum appears in Ann Intern
Med 1991; 114: 522].
150. Andrews TC, Reimold SC, Berlin JA, Antman
EM. Prevention of supraventricular arrhythmias
after coronary artery bypass surgery. A meta-
analysis of randomized control trials. Circulation
1991; 84: 236±44.
151. Hommes DW, Bura A, Mazzolai L, Buller HR, ten
Cate JW. Subcutaneous heparin compared with
continuous intravenous heparin administration in
the initial treatment of deep vein thrombosis. A
meta-analysis. Ann Intern Med 1992; 116: 279±84.
152. Hull RD, Raskob GE, Hirsh J, Jay RM, Leclerc
JR, Geerts WH, Rosenbloom D, Sackett DL,
Anderson C, Harrison L. Continuous intravenous
heparin compared with intermittent subcutaneous
heparin in the initial treatment of proximal-vein
thrombosis. New Engl J Med 1986; 315: 1109± 14.
153. Cook RJ, Walter SD. A logistic model for trend in
2  2  kappa tables with applications to meta-
analyses. Biometrics 1997; 53: 352± 7.
154. Lau J, Antman EM, Jimenez-Silva J, Kupelnick B,
Mosteller F, Chalmers TC. Cumulative meta-
analysis of therapeutic trials for myocardial infarc-
tion. New Engl J Med 1992; 327: 248± 54.
155. Stampfer MJ, Goldhaber SZ, Yusuf S, Peto R,
Hennekens CH. Effect of intravenous streptoki-
nase on acute myocardial infarction: pooled results
from randomized trials. New Engl J Med 1982; 307:
1180± 2.
156. Berkey CS, Mosteller F, Lau J, Antman EM.
Uncertainty of the time of first significance in
random effects cumulative meta-analysis. Control
Clin Trials 1996; 17: 357± 71.
157. Ziegler RG, Mayne ST, Swanson CA. Nutrition
and lung cancer. Cancer Causes Control 1996; 7:
157± 77.
158. The Alpha-Tocopherol=Beta Carotene Cancer
Prevention Study Group. The effect of vitamin E
and beta carotene on the incidence of lung cancer
and other cancers in male smokers. New Engl J
Med 1994; 330: 1029± 35.
159. Hennekens CH, Buring JE, Manson JE, Stampfer
M, Rosner B, Cook NR, Belanger C, LaMotte F,
Gaziano JM, Ridker PM, Willett W, Peto R. Lack
of effect of long-term supplementation with beta
carotene on the incidence of malignant neoplasms
and cardiovascular disease. New Engl J Med 1996;
334: 1145± 9.
659
160. Omenn GS, Goodman GE, Thornquist MD,
Balmes J, Cullen MR, Glass A, Keogh JP,
Meyskens FL, Valanis B, Williams JH, Barnhart
S, Hammar S. Effects of a combination of beta
carotene and vitamin A on lung cancer and
cardiovascular disease. New Engl J Med 1996;
334: 1150± 5.
161. Stewart LA, Clarke MJ. Practical methodology of
meta-analyses (overviews) using updated indivi-
dual patient data. Cochrane Working Group. Stat
Med 1995; 14: 2057± 79.
162. Stewart LA, Parmar MK. Meta-analysis of the
literature or of individual patient data: is there a
difference? Lancet 1993; 341: 418± 22.
163. LeLorier J, Gregoire G, Benhaddad A, Lapierre J,
Derderian F. Discrepancies between meta-analyses
and subsequent large randomized, controlled trials.
New Engl J Med 1997; 337: 536± 42.
164. Ioannidis JP, Cappelleri JC, Lau J. Issues in
comparisons between meta-analyses and large
trials. J Am Med Assoc 1998; 279: 1089± 93.
165. Cappelleri JC, Ioannidis JP, Schmid CH, de
Ferranti SD, Aubert M, Chalmers TC, Lau J.
Large trials vs meta-analysis of smaller trials: how
do their results compare? J Am Med Assoc 1996;
276: 1332± 8.
166. Peto R, Collins R, Gray R. Large-scale rando-
mized evidence: large, simple trials and overviews
of trials. J Clin Epidemiol 1995; 48: 23 ±40.
167. Villar J, Piaggio G, Carroli G, Donner A. Factors
affecting the comparability of meta-analyses and
largest trials results in perinatology. J Clin
Epidemiol 1997; 50: 997± 1002.
168. Borzak S, Ridker PM. Discordance between meta-
analyses and large-scale randomized, controlled
trials. Examples from the management of acute
myocardial infarction. Ann Intern Med 1995; 123:
873± 7.
169. Colditz GA, Brewer TF, Berkey CS, Wilson ME,
Burdick E, Fineberg HV, Mosteller F. The efficacy
of bacillus Calmette ± Guerin vaccination in the
prevention of tuberculosis: meta-analysis of the
published literature. J Am Med Assoc 1994; 271:
698± 702.
170. Colditz GA, Berkey CS, Mosteller F, Brewer TF,
Wilson ME, Burdick E, Fineberg HV. The efficacy
of bacillus Calmette ± Guerin vaccination of new-
borns and infants in the prevention of tuberculosis:
meta-analyses of the published literature. Pedia-
trics 1995; 96: 29 ±35.