Survey methodology

References
Groves R.M. et al. Survey methodology, Wileyinterscience, 2004
Canada national statistical agency:
http://www.statcan.gc.ca/
Eurostat methodological publications
European Values Study (EVS)

INTRODUCTION :
What is a survey ?
A survey is a systematic method of gathering information from (a sample
of) entities for the purpose of constructing quantitative descriptors of the
attributes of the larger population of wich the entities are members
A survey usually originates when an individual or institution is confronted
with an information need and the existing data are insufficient

What is survey methodology ?

Survey methodology is the study of survey methods

How survey work to produce statistics

Characteristics of
the population

Inference

Characteristics of a
respondent

Inference

Respondent
answers to
quenstions

Statistical
computing

Characteristics of
the sample

The life cycle of a survey

A survey from a process perspective

Define research objectives
Choose mode of
collection

Choose
sampling frame

Construct and
pretest a
questionnaire

Design and
select sample
Recruit and
measure sample
Code and edit data

Make postsurvey adjustments

Perform analysis

The life cycle of a survey from a

design perspective
Target Population
Construct
Sampling Frame
Measurement

Response

What
is the
survey
about?

Who
is the
survey
about?

Sample

Respondent
Edited
Response

Postsurvey Adjustments
Survey statistics

The Measurement dimension describes

what data are to be collected about the
observational units in the sample

The Representational dimention

concerns what population are
described by the survey

The measurement dimension

Construct

Measurement

Constructs are the elements of information that are

sought by the researcher :
How many incidents of crimes with victims there were in the last year;
The consumption of beer in the last month;
The degree of knowledge of mathematics of childrens

Measurements are ways to gather information about

constructs :
Questions posed to a respondent (During the last 6 month, did you call the police
to report something that happened to you that you thought was a crime?)
NB: the critical task for maesurement is to design questions that produce answers
reflecting perfectly the construct we are trying to measure.

Response

Edited
Response

Response could be produced in a variety of means

But in general the nature of the response is determined by the
nature of the measurement

Editing of data may examine the full distribution of

answers and look for atypical patterns of responses
Edited responses are the data from wich inference is made about the values of
the construct for an individual respondent

The representational dimension

Target Population

Sampling Frame

The target population is the set of unit to be studied

The adult population living in households in 2009;

The frame population if the set of target population members

that has chance to be selected into the survey sample :

In a simple case it is a list of all units in the target population, but sometimes it is a s
of units imperfectly linked to population members.
i.e. a list of telephone numbers when the target population is the adult population

Sample

The sample is the group from wich measurement will be

sought. In many case it is a very small fraction of the the sampling frame
Respondents are the elements successfully measures.

Respondent

Postsurvey
Adjustments

Non respondents is the complement

Postsurvey adjustments consist on weighting up the

underrepresented groups in order to improve the survey
estimate
Because of mismatches of the sampling frame and the target population
(coverage problems) statistics based on the respondents can differ from
caracteristics of the target population. Examination of non response patterns
may suggest an underrepresentation of some groupes relative to the sampling
frame

The life cycle of a survey from a

quality perspective
Construct

Target Population
Coverage
error

Validity
Sampling Frame
Measurement

Sampling
error

Measurement
error

Sample
Nonresponse
error

Response
Respondent

Processing
error

Adjustments
error
Edited
Response

Postsurvey Adjustments
Survey statistics

Coverage of a target
population by a frame
Undercoverage
Elements in the
target
population
missing from the
frame
i.e.:non telephone
household, using a
telephone frame to
cover the full
household
population

Ineligible units

Covered population

Undercoverage

Target population

Frame population

Ineligible units
Elements in the
frame that are no
member of the
target population
i.e.:business telephone
numbers, using a
telephone frame to
cover the full household
population

Evaluating survey questions:

Are the answers good measures of the
intended construct?
Exemple of methods that can be used
to evaluate draft survey questions
Expert reviews

Focus groups

The substantive expert review the wording, the

order and the structure of questions, the
response alternatives etc.
A small number of target population participate in a systematic
discussion about the survey topic. The researcher learn
about the nomenclature of the concept, the common
perspective taken by the target population on key issues etc

Questionnaire pretest

Researcher test how questions are read

and answered. A behaviour coding is
often used

Evaluating survey questions:

Exemple of behavior codes for interviewer
and respondent behaviors
Interview Questioning Behavior (choose one)
1.
Reads questions exacty as worded
2.
Reads questions with minor changes
3.
Reads questions so that meaning is altering
Respondent Behaviors (check as many as apply)
1.
Interrupt question reading
2.
Asks for clarification of question
3.
Gives adequate answer
4.
Gives answer qualifies about accuracy
5.
Gives answer inadequate for questions
6.
Answers dont know
7.
Refuses to answer

Methods of data collection

Traditional
data collection methods
Mailing paper questionnaires to respondents,
who fill them out and mail them back
Having interviewers call to respondents on the
telephone and ask them the question in a
telephone interview
Sending the interviewers to the respondents
home or office to administer the questions in
face-to-face (FTF) interviews

Alternatives methods of data collection

OCR/ICR

FAX

Optical/intelligent
caracter recognition

Mail

Disk by Mail

Telephone
Face to face
SAQ
Self administered
questionnaire

Web

E-mail

Computerised Self
Administered
Questionnaires

CATI

TDE

IVR

computer assisted
telephone interviewing

Touchtone
data entry

Interactive
voice response

CAPI computer assisted personal interviewing

Text
CASI

Walkman

Audio
CASI

Video
CASI

Alternatives methods of data

collection (a)
OCR/ICR
Optical/intelligent
caracter recognition

Mail

Disk by Mail

FAX

E-mail

Web

Alternatives methods of data

collection (b)

Telephone

CATI

TDE

IVR

computer assisted
telephone interviewing

Touchtone
data entry

Interactive
voice response

Alternatives methods of data

collection (c)

Face to face
SAQ
Self administered
questionnaire

CAPI computer assisted personal interviewing

Text
CASI

Walkman

Audio
CASI

Video
CASI

Chapter III
DESIGNING
A
QUESTIONNAIRE

QUESTIONS AND ANSWERS

IN SURVEYS
A questionaire
is a standardised set of questions administered to the
respondents in a survey
Respondents are required to interpret a preestablished set
of questions and to supply the information these questions
seek.

The cognitive processes in

answering questions
A simple model of the survey
response process
Comprehension
of the question

BUT
Respondents often take shortcuts to get
through the interview more quickly

OR

Retrieval of
information

they have motives that override their

desire to provide accurate information

Judgment and
estimation
Reporting an
answer

Responses could be biased by

acquiescence (the tendency to
agree)
social desirability (the tendency to
present oneself in a favourable light by
underreporting undesirable attributes and
overreporting desirable one)

The cognitive processes in

answering questions (2)
The satisficing model
(Krosnik and Alwin, 1987)
Some respondents try to satisfy (to take a low road answering more
superficially)
whereas others try to optimise (to take an high road by careful
answering questions)

Satisficing respondents do not seek to understand the question completely, just

well enough to provide a reasonable answer

Problems in answering
survey questions
Failure to encode the information sought
Misinterpretation of the questions
Forgetting and other memory problems
Estimation strategies
Problems in formatting answer
More or less deliberate misreporting
Failure to follow instruction

FORMATTING THE ANSWER

Survey items can take a variety of formats;
the most common are:
1) Open-ended qustions that call for
numerical answers
2) Closed questions with ordered response
scales
3) Closed questions with categorial
response options

1 - Open-ended qustions that call

for numerical answers
Now, thinking about your physical health,
which includes physical illness and injury,
for how many days during the past 30
was your physical health not good?
Note that:
Open-ended items yield more exact information than closed items

2 - Closed questions with ordered

response scales
1
2
3
4
5

Would you say that in general your health is:

Excellent
Very good
The interviewer is instructed to
Good
please read the answer
Fair
categories, but not the number
Poor
attached to them!
Note that :

with some type of rating respondents seem to shy away from the negative end of the scale

When the scale points have numerical labels, the label can affect the answer (e.g. if respondents are asked to rate their success in life)

3 - Closed questions with categorial

response options
1
2
3
4
5
6

Are you:
Married
Divorced
Widowed
Separated
Never married
A member of an unmarried couple
Note that :

The respondent may not wait to hear or read all the option; they may select the firs reasonable answer they consider (primacy effect)

The opposite coul happen: the last option the interviewer read may be the first one that respondent think about (recency effect)

GUIDELINES FOR WRITING GOOD

QUESTIONS (Sudman and Bradburn)
Non sensitive questions about behavior
The key problem with many questions about behavios is that respondents may forget some or
all of the relevant information, or that their answer may reflect inaccurate estimate

In order to reduce memory problems it is essential to play attention

to the wording of the question and to provide memory help

Attitude questions
Attitude questions are a very commen class of survey questions. The most frequent problems
deals with the wording of questions, the question order and the format of response scales

Non sensitive questions about behavior

Play attention to the wording
With closed questions, include all reasonable
possibilities as explicit response options

Are you:
Married
Divorced
Widowed
Separated
Never married

Are you:
Married
Single

Non sensitive questions about behavior

Play attention to the wording
Make the question as specific as possible
(about who it covers, what time period, which behaviours)

Over the last month,

that is .. how
often do you read a
newspaper in a
tipical week?

In a tipical
week, how
often do you
read a
newspaper?

Non sensitive questions about behavior

Play attention to the wording
Use words that virtually all respondents will understand
Have you ever
had a heart
attack?

Have you ever

had a miocardial
infarction?

Non sensitive questions about behavior

Provide memory help
Uses aided recall
(or ask separate questions about subcategories)
Please look cerefully at the
following list of volountary
organisations: which, if any,
do you belong to?
A
B
C
D

Religious organisations
Cultural organisations
Political groups
Other

To which volountary
organisation do you
belong to?

Attitude questions
Play attention to the wording
Clearly specify the attitude object of interest

Do you think the

Government is spending too
litte, about the right amount,
or too much on higher
education?

Do you think the

Government is spending too
litte, about the right amount,
or too much on education?

Attitude questions*
play attention to the wording
Measure the strength of the attitute
using a response scale, a separate item
or multiple items that can be combined into a scale

Do you agree or disagree with the following statement?

Government is spending too little on education
1
Agree strongly
2
Agree
3
Neither agree nor disagree
4
Disagree
5
Disagree strongly

*note that an attitude have generally a direction (agree or disagree)

and an intensity (strongly disagree.strongly agree)

Attitude questions
reduce impact of question order
When asking general and specific questions about a
topic, ask the general question first
(otherwise, the answer to the general question is likely to be affected by the
number and content of specific questions)

When asking questions about about multiple items,

start with the least popular
(the unpopular questions are likely to seem even less appealing
when they follow more popular questions)

When asking general and specific questions about

a topic, ask the general question first
Please tell me whether or not you think it should be
possible for a pregnant woman to obtain a legal
abortion if: the woman wants it for any reason?
1. Yes
2. No
3. Don't know
Please tell me whether or not you think it should be possible
for a pregnant woman to obtain a legal abortion if: there is
a strong chance of a serious defect in the baby?
1. Yes
2. No
3. Don't know
USA GSS

Attitude questions
play attention to the response scale
Use closed questions for measuring aptitudes;
(open answers are difficult to code)

Use five-to seven-point response scales

and label every scale point
(verbal label ensure that interpret the scale in the same way)

Now is up to you!