WGBH Higher Education Poll August 2018

Methodology Report

Submitted to:
WGBH Boston

Prepared by:
Abt Associates
Marci Schalk
Andrew Burkey
David Ciemnecki
Seth Brohinsky

www.abtassociates.com

August 31, 2018

1
I. SUMMARY
The WGBH National Poll August 2018, fielded for WGBH Boston by Abt Associates, obtained
telephone interviews with a representative sample of 1,002 adults living in the United States (350
respondents were interviewed on a landline telephone and 652 were interviewed on a cell
phone). Interviewing was conducted from August 21st through 25th, 2018 in English and Spanish.
Samples were drawn from both the landline and cell phone RDD frames. Persons with residential
landlines were not screened out of the cell phone sample. Both the landline and cell phone
samples were provided by Survey Sampling International. The combined sample is weighted to
match demographic parameters from the Current Population Survey and telephone status
parameters from the National Health Interview Survey. The weighting procedure also accounts
for the fact that respondents with both a landline and cell phone had a greater probability of
selection.

II. SAMPLE DESIGN

The target population for the study is non-institutionalized persons age 18 and over, living in the
US. Samples were drawn from both the landline and cellular random digit dial (RDD) frames to
represent people with access to either a landline or cell phone. Both samples were provided by
Survey Sampling International, LLC according to Abt Associates specifications.

Numbers for the landline sample were drawn with equal probabilities from active blocks (area
code + exchange + two-digit block number) that contained one or more residential directory
listings. The cellular sample was drawn by Survey Sampling International through a systematic
sampling from 100‐blocks dedicated to cellular service according to the Telcordia database.

III. CALLING PROTOCOL

Landline numbers were called as many as 5 times, and cell phone numbers were called as many
as 6 times. Refusal conversion was attempted on soft refusal cases. The sample was released for
interviewing in replicates, which are representative subsamples of the larger sample. Using
replicates to control the release of sample ensures that complete call procedures are followed
for the entire sample.

For the landline sample, interviewers asked to speak with either the youngest male or youngest
female at home right now. For the cell sample, interviews were conducted with the person who
answered the phone. Interviewers verified that the person was an adult and in a safe place before
administering the survey.

IV. WEIGHTING
Each case was assigned a base weight (BSW) to account for the probability of selection from the
landline or cell phone sampling frame. The base weight for respondents who have both a landline
and cell phone accounted for their greater probability of selection than individuals who are cell
only or landline only.

The data were weighted using iterative proportional fitting, also known as raking or rim
weighting. The sample was adjusted sequentially to population targets one variable at a time,
continuing until the optimum distribution is achieved. The population targets were drawn from
the 2017 Current Population Survey March Supplement for Census Region, educational
attainment, age, race/ethnicity, and sex. CDC’s wireless substitution report for July – December
2017 was used to define household telephone service (cell only, landline only, or dual service).
Weights were trimmed at the 2.5th and 97.5th percentile to reduce the design effect. The final
trimmed weight (Weight) is available for analysis.

V. DESIGN EFFECT AND MARGIN OF ERROR

Weighting and survey design features that depart from simple random sampling tend to result in
an increase in the variance of survey estimates. This increase, known as the design effect or deff,
should be incorporated into the margin of error, standard errors, and tests of statistical
significance. The overall design effect for a survey is commonly approximated as the 1 plus the
squared coefficient of variation of the weights. Using the final weight (Weight), the margin of
error (half-width of the 95% confidence interval) incorporating the design effect for full-sample
estimates at 50% is ± 3.47 percentage points. A summary of the final weights and their
approximate design effect and margin of error is reported in Table 1 below. Estimates based on
subgroups will have larger margins of error. It is important to remember that random sampling
error is only one possible source of error in a survey estimate. Other sources, such as question
wording and reporting inaccuracy, may contribute additional error.
Table 1. Design Effect and Effective Sample Size

Number of Minimum Maximum Approx. Effective

Weight Variable MOE
cases (n) weight weight Design effect n

WEIGHT 1,002 0.3240 2.7978 1.26 798 3.47%

VI. DISPOSITIONS
Table 2 reports the disposition of all sampled telephone numbers dialed for the survey. Overall,
the response rate (AAPOR RR3) was 5.36% for the landline sample and 4.08% for the cell sample.
The response rate calculations use two eligibility terms for estimating the proportion of cases of
unknown eligibility that are in fact eligible. The first (e1) addresses uncertainty about the working
and residential status of certain numbers and the second (e2) addresses uncertainty about
whether unscreened working and residential numbers were eligible for the survey (for example,
whether cell phone numbers belonged to someone age 18 and older).

Table 2. Final Dispositions and Rates, by Sample

Landline Cell
Sample Sample
Interview (Category 1)
Complete 1.000 350 652
Screen-outs 1.100 0 0
Partial 1.200 69 180

Eligible, non-interview (Category 2)

Refusal and breakoff 2.100 107 157
Refusal 2.110 1,718 0
Respondent never available 2.210 5 0
Answering machine household-no message left 2.221 3,436 0
Physically or mentally unable/incompetent 2.320 57 0
Household-level language problem 2.331 36 0

Unknown eligibility, non-interview (Category 3)

Always busy 3.120 699 3,781
No answer 3.130 4,258 2,558
Answering machine-don't know if household 3.140 94 0
Call blocking 3.150 31 387
Technical phone problems 3.160 0 25
No screener completed: No live contact made 3.210 0 11,662
No screener completed: Live contact made 3.210 0 3,448
Other: "cell phone" dispo used in error 3.910 0 4
Other: Cell case physically or mentally unable/incompetent 3.920 0 51
Other: Cell case language problem 3.930 0 226

Not eligible (Category 4)

Fax/data line 4.200 880 71
Non-working/disconnect 4.300 30,341 5,226
Temporarily out of service 4.330 703 1,070
Cell phone 4.420 1 0
Business, government office, other organizations 4.510 1,306 509
No eligible respondent (e.g., child phone) 4.700 0 342
Other 4.900 0 0
Total phone numbers used 44,091 30,349
Completes (1.0) I 350 652
Partial Interviews (1.2) P 69 180
Eligible Non-Interview: Refusal (2.1) R 1,825 157
Eligible Non-Interview: Non-Contact (2.2) NC 3,441 0
Eligible Non-Interview: Other (2.3) O 93 0
Undetermined If Working and Residential (3.1) UH 5,082 6,751
Working and Residential But Undetermined Eligibility (3.2,3.9)
Live contact was made UOC 0 3,729
Live contact not made UONC 0 11,662
Not Eligible: Nonworking, Nonresidential, or Ported (4.1-4.5,4.9) NWC 33,231 6,876
Screen Out: Working and Residential but Not Eligible (4.7) SO 0 342
TOTAL 44,091 30,349
e1=(I+P+R+NC+O+UOC+OUNC+SO)/(I+P+R+NC+O+UOC+OUNC+SO+NWC) 14.8% 70.9%
e2=(I+P+R)/(I+P+R+SO) 100.0% 74.3%
AAPOR RR3 = 5.36% 4.08%
I / (I+P+R+NC+O+[e1*e2*UH]+[e2*(UOC +UONC)])
AAPOR RR4 = 6.42% 5.21%
(I+P) / (I+P+R+NC+O+[e1*e2*UH]+[e2*(UOC +UONC)])