NBC News|SurveyMonkey Weekly Election Tracking poll

Hispanic Weighting
August 5, 2016
The role of Hispanics in the 2016 primaries and general elections is important. There has been a
considerable amount of research by academics and professional survey researchers into the question of
how to best capture Hispanics in surveys. When NBC News and SurveyMonkey first entered into a
strategic partnership in 2014, one goal was to produce research on best survey practices that would
improve the quality of our own online polls and help inform those of others. We have recently
conducted research on how to best capture Hispanic respondents in our joint weekly election tracking
poll. This memo describes the findings of our research.
The question of how to best include Hispanic respondents in surveys has been studied extensively by
many and it has been analyzed in great detail by the U.S. Census Bureau. The Census does not treat
Hispanic as a race, but an ethnicity. What this means is that someone could be of Hispanic origin but
identify as white, black or any other race for that matter. Conceptually this is a clear idea, but putting it
into practice is a little trickier.
The NBC News|SurveyMonkey Weekly Election Tracking poll interviews about 10,000 registered voters
each week and at this massive scale, we are able to continually iterate and refine the methods that we
use in each poll. Since the beginning of May, weve been systematically examining and testing different
aspects of our methodology regarding Hispanic interviews and as a result have made two small changes
in recent weeks which we think improves the quality of our tracking poll. The first change, which was
rolled out several weeks ago on July 11, was to begin to incorporate Spanish language respondents in
our releases. The second change, effective with the results released just this week involved moving to a
single, combined race/ethnicity question for weighting and crosstabs. Below we explain the research
that motivated us to make the change.
Spanish language respondents
Our testing clearly showed differences between Hispanic respondents who completed the English
version of our survey compared to those who responded in Spanish. This is consistent with what other
survey researchers have found. Over the past three weeks, an average of 98 registered voters per week
have completed the survey in Spanish. These Spanish-language respondents are recruited the same way
as the English-language respondents, on the backs of user-generated surveys at SurveyMonkey.
Respondents who have completed a user-created survey in Spanish are eligible to take the NBC
News|SurveyMonkey Weekly Election Tracking poll in Spanish. Data collected after July 11th
incorporated respondents from the Spanish-language sample.
Over a three-week period from July 11-31, 2016, Hispanic voters who completed the poll in English were
four times more likely to support Donald Trump compared to Hispanic voters who were recruited from a
Spanish-language survey.

The NBC News|SurveyMonkey Weekly Election Tracking poll will continue to be offered in Spanish going
forward. We consider this move to be in line with best practices in survey research and our own
research confirmed it is important to offer the tracking poll in both languages.
Questionnaire designs and weighting
Prior to the latest tracking poll, race and ethnicity were measured in two separate questions. The first
question asked respondents to select one race, with Hispanic offered as a choice. We asked a second,
follow-up question of those who did not select Hispanic on the race question whether they are of
Hispanic or Latino origin or descent. Using that questionnaire design, those who selected Yes in the
Hispanic ethnicity follow-up were tagged as Hispanic for the purposes of weighting and also for
crosstabs, even if they had selected a different race in the previous question.
Are you:
White
Black
Hispanic
Asian
Other

Are you of Hispanic or Latino origin or descent,

or not?
Yes
No

After examining our horse race data among Clinton and Trump, we found that our estimates of
Hispanics supporting Trump in the primaries and the early general election polling were higher on
average than the bulk of other election polls. We decided to investigate whether that was an artifact of
questionnaire design. We also wanted to compare our approach to other ways of identifying Hispanics,
including the method used by the Census. We evaluated the different approaches by randomly
assigning respondents two different ways of identifying as Hispanics. Our hypothesis was that our
current methodology might be measuring Hispanic ancestry too broadly. In other words, by asking
respondents twice whether they are Hispanic, we might be coaxing some respondents to answer
affirmatively who might not do so on other surveys.
We divided our interviews conducted from June 27-30 so that we asked our standard race and ethnicity
questions of a random half of the sample. The second half received the questions similar to those used
in the U.S. Census and the Bureau of Labor Statistics Current Population Survey, which we use as targets
for weighting and does not include Hispanic as a choice on the race question.

Are you of Hispanic or Latino origin or descent,

or not?
Yes
No

Is your race:
White
Black
Asian
Other

Our unweighted control interviews showed that 7.1 percent of registered voters opted for Hispanic on
the initial race question, and another 2.5 percent identified as being of Hispanic origin on the follow-up,
for a total of 9.6 percent that we had typically classified as Hispanic for purposes of weighting and
analysis. On the experimental control that used the Census format, just 6.6 percent identified as
Hispanic. Thus the data confirmed our hypothesis: By asking respondents if they were Hispanic twice,
we had classified some respondents as Hispanic when their primary identification was different.
CONTROL
Unweighted N (RVs)
Choose Hispanic on race
question
Choose Hispanic on follow-up
Total
EXPERIMENT
Unweighted N (RVs)
Choose Hispanic on CPS Hispanic
ancestry question, asked first

4796
7.1
2.5
9.6
4371
7.1

Source: NBC News|SurveyMonkey Weekly

Election Tracking poll conducted June 2730, 2016
In testing, we found that Trump/Clinton support looks very similar if asked the Census way (with the
Hispanic ethnicity question asked among all first followed by the race question without the option for
Hispanic) or if asked as one race question which includes the option for Hispanic. The results of that are
shown below.

The control version is what we had used for our tracking poll until the results of this experiment. Our
finding led us to make a change. The two other optionsasking the race question with Hispanic as an
option and no follow up or asking Hispanic ethnicity first followed by the race question without Hispanic
as an option (the Census method)produced similar results. Trumps support among Hispanics in the
control is 33% whereas his support among Hispanics in both experiments is 31%.
Trumps support among Hispanics was lower for both experiments than it was in the control, however
not by a substantively large amount. Using one of the experiments going forward was the better option
for that reason. The Census battery and the one-question race option including Hispanic produced
similar results. There are limitations to the number of questions we can ask on an online questionnaire
and the space savings with a one-question approach vs. two-question approach are significant. Thus, we
will use the one-question race format which includes Hispanic as an option to capture Hispanic
respondents going forward.