Publication Bias in Appellate Court Decisions

Jonathan D. Colan
At a recent Eleventh Circuit oral argument addressing, among other issues, whether a
particular instruction to the jury during deliberations was coercive, one of the judges on the panel
noted that when he was a trial judge, whenever he gave the jury an Allen charge,1 the jury always
acquitted the defendant. The prosecutor responded to the judges observation by stating that
those decisions are not appealed and are thus not reported and found when lawyers research this
issue.
This sort of publication bias has been recognized as a problem in scientific and medical
fields, where studies producing negative results tend not to be published as often as studies
yielding positive results.2

The absence of literature on one side of an issue can leave a

misleading impression. Its not that fewer studies find negative results, its that fewer authors
and journals bother to publish negative results. Whether this is because, in the scientific context,
the entities behind the studies have less incentive to publicize negative results or because the
journals themselves find negative results less interesting is an issue best left for others to explore.
For our purpose as members of the Bench and Bar, it is worth remembering that only
certain scenarios will lead to appealable issues and thus to reported decisions for us to find and
apply. I recall a law school classmate who defended police search procedures on the grounds
1 See Allen

v. United States, 164 U.S. 492 (1896).

See, e.g., Annie Franco, Neil Malhotra, Gabor Simonovits, Publication Bias in the Social
Sciences: Unlocking the File Drawer, 345 Science 1502 (Sept. 19, 2014), abstract available at:
http://www.sciencemag.org/content/345/6203/1502.abstract?sid=169ff50d-8761-43d3-83c6f4473555d064.
2

that the police must know what they are doing, because every case we read was one in which the
police found illegal drugs. Of course, the incidents where the police found no drugs did not lead
to arrests, convictions, appeals, and reported decisions. We dont find those cases when we
research Fourth Amendment precedent.
This Eleventh Circuit judges experience as a former trial judge was that Allen charges
are not coercive, because they tended to lead to acquittals. A lawyer researching this issue before
drafting a brief or memorandum might not come to that conclusion if all he or she has to go on is
the body of reported decisions resolving appeals of cases where Allen charges were given. The
cases ending in acquittals werent appealed.
Beyond the fact that appellate courts do not choose to publish more of their
unpublished and yet still easily accessible decisions, we must remember that the body of
reported decisions suffers from its own sort of publication bias. [P]ublished opinions are not
representative of all opinions; opinions are not representative of all district court decisions; and
adjudicated cases are not representative of all filed cases. 3 Not every trial court decision is
immediately appealable and not every decision will remain at issue if the matter settles, pleads
out, or is otherwise resolved in a partys favor.4
Pauline T. Kim, Margo Schlanger, Christina L. Boyd, and Andrew D. Martin, How Should We
Study District Judge Decision-Making, 29 WASH. U. J. L. & POLY 83, 97 (2009).
3

See id. at 107-08; see also Andrew E. Henry, The ADA Amendments Act of 2008: Why the
Qualified Individual Analysis is the New Battleground for Employment Discrimination Suits, 67
Okla. L. Rev. 111, 121 (Fall 2014) (noting the tendency of [trial] courts to issue a higher
number of written opinions for decisions granting summary judgment than those denying
summary judgment); Thomas A. Cooper, Jurisdictional, Procedural, and Economic
Considerations for Non-Party Electronic Discovery, 59 Emory L. J. 1339, 1346 n.49 (2010)
(noting the publication bias in the dearth of reported decisions resolving discovery disputes).
4

Just as we appellate lawyers like to remind our trial lawyer colleagues that they should
consult with us and keep in mind the appellate implications of events in trial court, perhaps we
appellate lawyers should remember to consult with our trial lawyer colleagues about their
experiences in trial court that would not be reflected in reported appellate court decisions. There
are more things in law and jurisprudence than are found in reported decisions.