PERSPECTIVE
minous information about ad-
verse drug reactions in large
populations. The challenge will
be to distinguish adverse drug
reactions from events not caused
by drugs.
The FDA must also decide
when to publicize preliminary
findings about adverse events.
Historically, it has generally not
announced its findings until it
has fully reviewed the data and
decided to take regulatory action.
Recently, there has been increas-
ing pressure to announce prelim-
inary findings earlier. With the
help of its new tools for identify-
ing adverse reactions, the FDA
will have to develop new stan-
dards for publicizing findings,
In Defense of Pharmacoepidemiology
Embracing the Yin and Yang of Drug Research
Jerry Avorn, M.D.
T he past decade has not been
kind to observational stud-
ies of medications. The damage
began in 1998 with the publica-
tion of the Heart and Estrogen
Progestin Replacement Study, a
randomized controlled trial show-
ing that hormone replacement
increased the risk of cardiac
events among postmenopausal
women with heart disease. Like
many physicians, I had been
teaching the gospel that estro-
gen use prevented heart disease
an idea based on observation-
al studies1 showing that post-
menopausal women who regular-
ly took estrogen were less likely
to have heart disease than appar-
ently similar women who did not
take hormones. It now appeared
that this had been a misleading
conclusion that may have led to
drug-induced disease in millions
The New England Journal of Medicine
Downloaded from nejm.org at Harvard Library on August 8, 2016. For personal use only. No other uses without permission.
Copyright 2007 Massachusetts Medical Society. All rights reserved.
Bolstering the FDAs Drug-Safety Authority
separate from its standard for timely technical assistance to
taking regulatory action.
The bill also expands the
clinical trial registry put into
place by the National Institutes
of Health to include trials from
additional sources, additional in-
formation including results of
trials, and trials on a wider range
of diseases.
For some time now, FDA ex-
perts have recognized that the
agency needed new legal authori-
ties and funding to act expedi-
tiously when new safety data
became available. The Bush ad-
ministration blocked agency of-
ficials from publicly supporting
such new authorities and resourc
es and even from providing
of patients. A heavier blow came
in 2002 with publication of an
even larger randomized trial from
the Womens Health Initiative
(WHI), demonstrating that wom
en without preexisting heart dis-
ease who were given estrogen
also had an increase in cardiac
events along with expected in-
creases in breast cancer, throm-
bophlebitis, stroke, and pulmo-
nary emboli. Other WHI trial data
refuted the conclusions of obser-
vational studies that estrogen us-
ers were less likely than nonusers
to develop dementia, depression,
or incontinence.
More disconnects followed.
Epidemiologic findings that de-
mentia or cancer was less likely
to develop in statin users than in
nonusers were not borne out by
clinical trials2; other observation
al studies suggesting that patients
n engl j med 357;22 www.nejm.org november 29, 2007
ensure that legislation would be
drafted to meet congressional
goals. Although Congress has
acted, whether the FDA will be
permitted to use its new authori-
ties to protect the public health
remains to be seen.
Mr. Schultz is a partner at Zuckerman
Spaeder LLP in Washington, DC, and a for-
mer deputy commissioner for policy at the
Food and Drug Administration.
1. Drug Amendments Act of 1962, Pub. L.
87-781, 76 Stat. 780 (1962).
2. Prescription Drug User Fee Act of 1992,
Pub. L. 102-571, 106 Stat. 4491 (1992).
3. Institute of Medicine. The future of drug
safety: promoting and protecting the health
of the public. Washington, DC: National
Academies Press, 2007.
4. Harris G. F.D.A. official admits lapse on
Vioxx. New York Times. March 2, 2005.
Copyright 2007 Massachusetts Medical Society.
treated with statins really had
higher cancer rates didnt pan out
either. Next, clinicians, policy
makers, and families were alarmed
by the contention based on a
limited number of cases and so
far unsubstantiated that chil-
dren given stimulants for atten-
tion-deficit disorder are at in-
creased risk for potentially fatal
heart disease. In January 2007,
the BMJ ran an article arguing
that, as its headline stated, Ob-
servational studies should carry a
health warning, on the grounds
that analyses in such studies can
produce unreliable findings. In
September, the New York Times
Magazine published a controver-
sial cover story suggesting that
epidemiologic studies of drugs
and diet often arrive at bogus
conclusions. At the Food and Drug
Administration (FDA), drug-epi-
2219
PERSPE C T I V E In Defense of Pharmacoepidemiology Embracing the Yin and Yang of Drug Research

many are hardwired into the na-

Strengths and Weaknesses of Randomized Controlled Trials
and Observational Studies of Medications.
Strengths
Randomized, controlled trial
Study groups very similar before
treatment
Conducted by well-established
methodologic rules
Considered gold standard for as-
sessing efficacy
Can be registered to prevent selec-
tive reporting
Observational study
Can involve large numbers of typi-
cal patients in settings of rou-
tine care
Can focus on specific vulnerable
populations
Can be performed relatively quickly
and at modest cost
Can identify rare adverse events
Can follow patients over many
years
Can compare outcomes of several
treatment alternatives
demiology activities have long
been relegated to second-class
status, resulting in a failure to
identify many important drug
risks in a timely manner.3
So is this the end of the line
for such research? Not by a long
shot. Although the traditional
means of assessing drugs, the
randomized, controlled trial, is
the rightfully enshrined standard
for determining efficacy, such
studies often have important
drawbacks (see box): limited size,
making it difficult to detect un-
common adverse events; short
duration, even for drugs designed
to be taken for a lifetime; fre-
quent reliance on placebos as
the comparator, limiting clinical
relevance; termination after a
surrogate end point is achieved,
without measurement of real clin-
ical effects; and underrepresen-
tation of patients with complex
Weaknesses
Costly, cumbersome
Involve limited number of participants
Often underrepresent key patient groups
Short duration
Comparator (or placebo) often irrelevant
May measure surrogate end points rather
than clinical outcomes
Protocol may not reflect typical care
Susceptible to confounding caused by un-
derlying differences among patients
treated with different drugs
Confounding (especially due to patient se-
lection and differences in compliance)
can generate drugoutcome associa-
tions that are not truly causal
Methodologically difficult to do well
Difficult to identify selective reporting of
findings
Difficult to require registration
health problems, especially the
elderly. Some of these limitations
result from lax study protocols
that are proposed by manufac-
turers and are too readily accept
ed by the FDA. But others are
inherent in the randomized, con-
trolled study design. No trial could
ever be large enough to gather
enough data to quantify the risks
of all uncommon side effects,
and studies lasting long enough
to document all clinical outcomes
would be impractical if they re-
quired many years of follow-up
before approval could be grant-
ed. In addition, since the FDA
generally doesnt require head-
to-head comparisons of similar
drugs, preapproval trials are un-
likely to provide the comparative
data that physicians, patients,
and payers need. Improvement
in study designs could address
some of these problems, but
ture of randomized, controlled
trials (see box).
Fortunately, much of the in-
formation we need can be found
through careful epidemiologic
analysis of data being routinely
amassed through the computer-
ization of nearly all health care
transactions, from prescriptions
to hospitalizations to laboratory
tests a wealth of readily mo-
bilized insights about real-world
drug use and outcomes in mil-
lions of patients who have been
followed for years. Data can also
be collected through interviews,
registries, and other methods that
can be rigorous, practical, and
cost-effective. New FDA regula-
tions and funding that took ef-
fect in October4 may give post-
marketing drug surveillance a
more central role both inside and
outside the agency.
But will that merely generate
more misleading findings of il-
lusory benefits and phantom
risks? Not if we do things right.
Pharmacoepidemiology is still in
its adolescence, with all the char-
acteristics that implies: expansive
energy, huge potential, limited
experience, a sense of infallibil-
ity, accident-proneness, and occa-
sionally impaired judgment. Many
of us who work in this area rec-
ognize the need to advance the
disciplines methodologic sophis-
tication to prevent the sort of
glib conclusions that have be-
deviled the field; that arcane work
is making important strides.5 We
are learning key lessons: that
without randomization, physicians
and patients select therapies in
ways that can introduce sub-
stantial confounding; that the
trendy approach of propensity-
score analysis, in which research-

2220 n engl j med 357;22 www.nejm.org november 29, 2007

The New England Journal of Medicine

Downloaded from nejm.org at Harvard Library on August 8, 2016. For personal use only. No other uses without permission.
Copyright 2007 Massachusetts Medical Society. All rights reserved.
PERSPECTIVE In Defense of Pharmacoepidemiology Embracing the Yin and Yang of Drug Research
ers try to statistically align the
characteristics of users and non-
users of a drug, often is not
powerful enough to eliminate
such confounding; that the best
observational research mimics
the randomized, controlled trial
by studying new users of a drug
in comparison with new users
of another medication, not just
analyzing the survivors who
have been filling their prescrip-
tions consistently and have prob-
ably had the most success with
their treatment regimens; that the
presence of a diagnostic code on
a billing form does not necessar-
ily correspond to the presence of
a given disease; that occasional
mailed surveys are not an ade-
quate means of assessing actual
medication exposure; and that
subtle misdefinition of clinical
outcomes or periods of drug use
can introduce substantial bias.
Many of the gaffes of errant
pharmacoepidemiologic studies
can be traced to such methodo-
logic lapses, which we are grad-
ually learning how to prevent.
When done correctly, epide-
miologic studies of drug effects
can be both more conceptually
demanding and more powerful
than the average randomized,
controlled trial, especially in as-
sessing drug safety. Undertak-
ing a randomized trial requires
considerable resources, plan-
ning, human-studies approvals,
and time; the design standards
for such trials are well estab-
n engl j med 357;22 www.nejm.org november 29, 2007
The New England Journal of Medicine
Downloaded from nejm.org at Harvard Library on August 8, 2016. For personal use only. No other uses without permission.
Copyright 2007 Massachusetts Medical Society. All rights reserved.
lished, and these studies are in-
creasingly subject to registration
and public scrutiny. But anyone
with a few hundred thousand
dollars can buy access to a data-
base and perform an observa-
tional study, analyzing the data
in myriad ways. Since such stud-
ies are not currently registered,
there is no way of knowing how
many analyses were attempted
enormous numbers of alter-
native designs can be run on a
computer before a preferred re-
sult is selected for publication.
This is particularly worrisome,
because such research is often
sponsored by stakeholders who
may have billions of dollars rid-
ing on the outcome.
We forget how difficult it was
to establish the rules of the road
for conducting randomized tri-
als. In terms of design theory
and pubic policy, drug-epidemi-
ology research is now where ran-
domized trials were in the 1950s.
We have much to learn about
methods, transparency, and pro-
tecting the publics interest. But
that work can be done, and we
often have no other way of gath-
ering vital insights.
A Zen garden in Kyoto, Japan,
contains 15 large stones surround
ed by gravel. Monks meditate on
the fact that one cannot see all
15 rocks from any one point; no
matter where you sit, at least 1
stone is blocked by another. Like
one of these perspectives, ran-
domized trials offer one kind of
knowledge but prevent us from
seeing other properties of a drug.
Epidemiologic studies can help
elucidate those properties but may
introduce new blind spots.
The two approaches are a lit-
tle like rocket ships and tele-
scopes. Space flights got us to
the moon and beyond, but astron-
omy noninterventionally re-
vealed some of the basic secrets
of the universe and made space
flight possible. Is there one bet-
ter way of knowing aeronau-
tics or astronomy, invasive or
analytic method, yang or yin?
This is a nonchoice: to under-
stand everything we should know
about a drug, we must do both
kinds of research with rigor and
with humility.
No potential conflict of interest relevant
to this article was reported.
Dr. Avorn is a professor of medicine at Har-
vard Medical School and chief of the Divi-
sion of Pharmacoepidemiology and Phar-
macoeconomics at Brigham and Womens
Hospital both in Boston.
1. Barrett-Connor E, Grady D. Hormone re-
placement therapy, heart disease, and other
considerations. Annu Rev Public Health
1998;19:55-72.
2. Shepherd J, Blauw GJ, Murphy MB, et al.
Pravastatin in elderly individuals at risk of
vascular disease (PROSPER): a randomised
controlled trial. Lancet 2002;360:1623-30.
3. Mathews AW. Bayer halts U.S. sale of Tra-
sylol. Wall Street Journal. November 5, 2007.
4. Harris G. House passes bill giving more
power to the F.D.A. New York Times. Sep-
tember 20, 2007.
5. Schneeweiss S, Patrick AR, Strmer T, et
al. Increasing levels of restriction in pharma-
coepidemiologic database studies of elderly
and comparison with randomized trial re-
sults. Med Care 2007;45:Suppl 2:S131-S142.
Copyright 2007 Massachusetts Medical Society.
2221