Political Polling

EMILY SWANSON, ASSOCIATED PRESS

SCOTT CLEMENT, WASHINGTON POST
So you’re thinking
of writing about a
poll....
DISCLOSURE & TRANSPARENCY
You should be able to figure out when, how and by whom a poll was
conducted
How many people were interviewed? When? What was the target
population and how were they selected and interviewed? What
questions were asked and in what order?
If you can’t answer these questions, you have no starting point to judge
a poll’s quality or newsworthiness
Who conducted the poll and
who sponsored it?
Is there a strategic reason for releasing this poll?
◦ Political campaigns sometimes selectively release
polls that suggest their candidate is competitive
Are the questions asked in a fair and unbiased way?
◦ It’s always important to know a question’s wording
and context, but especially take care when the poll
sponsor has a stake in the outcome
Do you support or oppose offshore drilling for domestic oil and natural gas resources?
Poll’s sponsor – the American Petroleum Institute.
Comes after:

• How important to you is producing more oil and natural gas here at home?
• How important is energy when it comes to the issues the federal government should be
focusing on?
• Do you think the federal government does enough to encourage the development of oil
and natural gas resources right here in the U.S.?
• Please indicate if you agree or disagree:
• Increased production of domestic oil and natural gas could lead to more jobs in the U.S.
• Increased production of domestic oil and natural gas could help stimulate the economy.
• Increased production of domestic oil and natural gas could help strengthen America’s
energy security.
• Producing more domestic oil and natural gas could help lower energy costs for
consumers.
• Producing more domestic oil and natural gas could help strengthen America’s national
security by lessening the negative impacts of political instability occurring in other parts
of the world
• Producing more domestic oil and natural gas could benefit federal and state budgets
through lease payments, royalty fees and other sources of revenue.
How were people contacted to
take part in the poll?
Phone
Online
Mail
In-person
PHONE POLLS
Major considerations:
Is everyone covered?
◦ 52% of American adults don’t have a landline phone at home
◦ They differ from adults overall: 71% of those age 25-34 only have a cell
phone

IVR/”Robopolls”
Where do phone numbers come from?
◦ Random digit dialing
◦ Voter lists
ONLINE POLLS
Probability based panels: GfK KnowledgePanel, NORC AmeriSpeak
Panel, Pew American Trends Panel
Opt-in panels: ongoing research and debate about ability to be
representative of general population: Major players include
SurveyMonkey, YouGov, Ipsos
Web votes and email/text message campaigns 
Margin of sampling error
“plus or minus X percentage points”
The fewer people interviewed, the larger the margin of error
◦ Higher for subgroups like women, African Americans, and likely voters

What it means: For a poll with a margin of error of plus or minus 4

percentage points, we should expect the measured value to fall within 4
percentage points of the true value 95 times out of 100
◦ BUT not the only source of error in polls: Question wording and order,
refusal to participate, imperfect likely voter models, etc.
When is a candidate leading?
A survey margin of sampling error applies to every candidate or poll
response
If the difference between two response options is more than twice the
margin of error, then the poll shows one candidate is leading or one group
is larger than another.
If the difference is at least equal to the margin of error, but no more than
twice the margin of error, then one candidate can be said to be
“apparently leading” or “slightly ahead,” or one group can be said to be
slightly larger than another.
If the difference is less than the margin of error, the poll says a race is close
or about even or that two groups are of similar size.
Do not use the term “statistical dead heat,” which is inaccurate if there is
any difference between the candidates. If the poll finds the candidates are
tied, say they are tied.
TIMELINESS
It’s important to know when a poll was conducted!
Consider if public opinions could have changed because of events
occurring between when the poll was conducted and when it was
released (or you’re reporting on it)
If so, the poll might still be usable but the context should be noted (“the
poll was conducted the week before Congress passed the new health
care legislation.”)
PUSH FOR ANSWERS
Pollsters don’t always release all the details, but you’re in a good
position to ask
The American Association for Public Opinion Research has disclosure
standards that can be a useful tool to push reticent pollsters (and a
good resource for you)
What about the aggregators?
Poll aggregation: Benefits and
drawbacks
Easy to quickly get a sense of where public opinion stands
Can be useful to compare to trends from individual polls: If one poll
shows a change but the rest don’t, the one showing a change might be
an outlier
Usually include polls of varying quality
Important to look at multiple polls
Look at the spread: If polls disagree, wording or methods may be
instructive
Lessons of 2016
 What actually happened?
National polls were actually pretty accurate by historical standards!

Source: American Association for Public Opinion Research report on 2016 poll accuracy
 But state polls were less accurate, especially in key states Trump ended up winning

Source: American Association for Public Opinion Research report on 2016 poll accuracy
Why?
People changed their minds during the final week, according to exit
polls
Education
Was there a “shy Trump” effect?
Turnout?
Forecasts assigning a win probability may be part of the problem
 AAPOR task force report: “There is no consistent partisan favoritism in recent
U.S. polling” – “whether polls tend to miss in the Republican direction or the
Democratic direction is tantamount to a coin flip”

Source: American Association for Public Opinion Research report on 2016 poll accuracy
2018 and the
lessons of 2017
What was going on in Virginia?
Polls in Virginia showed a narrow edge for Democrat Ralph Northam,
ranging from Northam +9 to Gillespie +3. Northam ended up winning by
9 points, with polls underestimating his support by 6 points on average

However, in 2013 and 2014, Virginia polls underestimated the Republican

candidate
Source: Real Clear Politics
 And in Alabama?
Polls showed everything from a 10-point Jones advantage to a 9-point
Moore lead; Jones ended up winning by 1.5 points.
Most polls employed automated methods, which were less accurate in
2017’s elections than in 2016

Source: Real Clear Politics

The likely voter challenge
illustrated
So what’s going to happen in
2018?
There won’t be polls in every congressional district – even the competitive ones
(and it’s really early to do these anyway!)
National polls can give a rough indication of how many seats a party is expected
to win
The “generic ballot”
Measures support for an unnamed Democrat or Republican candidate
for Congress, for example: If the election for the U.S. House of
Representatives were being held today, would you vote for the
Democratic candidate or the Republican candidate in your congressional
district?
For Democrats, it’s not enough to be ahead: Forecasters and analysis
estimate they need at least a 6 to 8 point advantage to win a House
majority
Most are now measuring among registered voters and will switch to
likely voters in the fall. This can make a big difference if turnout heavily
favors one party!
It’s still early! A sitting president’s party often loses seats, but there’s a
lot of time between now and November
Resources
AAPOR online course for journalists: http://www.aapor.org/Education-
Resources/For-Media/Online-Course-for-Journalists.aspx
AAPOR transparency initiative:
http://www.aapor.org/Transparency_Initiative.htm
Roper Center: https://ropercenter.cornell.edu/
National Council on Public Polls “20 questions”:
http://www.ncpp.org/?q=node/4
Polling Report pollingreport.com