INSTITUTIONS,

GOVERNMENT AND
SOCIETY
Piero Stanig

INSTITUTIONS, GOVERNMENT AND

SOCIETY
Piero Stanig
Assistant Professor of Political Science, PAM Department, Bocconi

September 18, 2014

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

When experiments are not an

option

Piero Stanig

Last week: experiment gets rid of baseline and

self-selection biases
what if experiment not an option
estimating credible causal effects from observational

data
various families of strategies
1 creating an adequate control group based on available
observations
2 using time to estimate causality
3 exploiting randomness given to us by nature
4 exploiting known, even though not random, treatment
assignment mechanism

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Biases and non-random

assignment

Piero Stanig

biases derive from fact that the treated are not like the

controls
in other words, being assigned to treatment depends

on potential outcomes
and in again other words, assignment depends on

characteristics of unit
these characteristics can be observed or unobserved

(and unobservable)

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Finding a good control group

Piero Stanig

observe a vector Xi of characteristics for unit i...

for every treated unit j {1, 2, . . . , NT } find one unit i

among the untreated with exact same values of X

(exact matching)
in practice hard to find one exact match for every

treated

[ involgarito, difettoso ]

could do coarsened exact matching (Iacus, King and

Porro)
as it turns out, no need to match exactly: just need a

summary of the X variables

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

The propensity score

Piero Stanig

problem is that treatment status T = {0, 1} correlated

with X
trick: model P(T = 1|X )

the predicted probability P(T

= 1|X ) of being treated is
called propensity score
can be estimated using standard method for binary

dependent variable (logit/probit) [cooler methods exist

though]
Rosenbaum and Rubin show that using propensity

score is equivalent to using full vector of X

??

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Matching

Piero Stanig

once we have propensity score (for both treated and

controls) can find, for each treated unit, match with

similar score among the controls
intuition: these are pairs that have (according to model)

same probability of being treated

assumptions:
1
2

CIA (conditional independence assumption)

common support

notice that matching throws out some of the data

various ways to perform in practice

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Checking balance

Piero Stanig

propensity score matching works because it provides

balanced groups (somewhat tautological...)

fortunately, balance (on observables) can be checked
in practice, compare summaries of the X variables in

control and treatment groups

if two groups are sufficiently similar, one can run and

claim credibility

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Checking balance: an example

Piero Stanig

do municipalities with mayor from presidents party

receive more transfers from federal govt?

study of Mexico 2000-2006
outcome: (per capita) transfers from federal

government to municipality
treatment: mayor is from the PAN (given that president

is from PAN)
observables: political and demographic info about

municipality

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Checking balance: an example

Piero Stanig

PAN Vote 2000

PRI Vote 2000
PRD Vote 2000
Less 5000 Residents
Illiteracy Rate
Log Population

Control
24.89
48.71
21.41
75.01
17.20
9.21

SD
1.58
3.53
1.55
2.16
1.57
5.96

Treatment
36.68
46.55
12.87
65.17
14.28
9.83

SD
2.33
3.37
0.93
1.87
1.31
6.36

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Checking balance: an example

Piero Stanig

PAN Vote 2000

PRI Vote 2000
PRD Vote 2000
Less 5000 Residents
Illiteracy Rate
Log Population

Control
36.66
46.57
12.87
65.18
14.28
9.83

SD
2.33
3.37
0.93
1.87
1.31
6.36

Treatment
36.68
46.55
12.87
65.17
14.28
9.83

SD
2.33
3.37
0.93
1.87
1.31
6.36

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Problems with matching

Piero Stanig

[ salto ]

leap of faith that balance on observables implies

balance on unobservables
less credible than (good) RDD and IV
inefficient (throws out some data)

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Alternatives to matching

Piero Stanig

weight controls by some function of propensity score

choice of function determines if estimand is ATT or ATE

(or ATC)
notice more complicated spins on this intuition (most

prominent is Abadies work on syntethic controls)

INSTITUTIONS,
GOVERNMENT AND
SOCIETY
Piero Stanig

Using variation over time:

Difference in Difference
simple intuition: look at unit before and after treatment
problem: time trends as confounders (ancient problem

of causation vs. time sequencing)

examples :
patient takes flu medicine, but flu was fading anyway
UN sends peacekeepers, but civil war was already

fading (deplete resources, too many people killed, etc)

one simple fix: compare pre- and post- for treated to

pre- and post- for untreated

assumption: parallel trends
equivalently: all unit-specific confounders are

time-invariant
hidden assumption: causal effects are the same in

treated and control

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

DD in practice

Piero Stanig

in practical terms, call Yt the outcome at time t

estimate

DD = E(Y1 Y0 |T = 1) E(Y1 Y0 |T = 0)
if needed, add controls and estimate regression like

Yit = + Tit + Ait + Tit Ait + Xit + it

with A dummy for the after period, and T dummy for

the treated units

estimates the causal effect from DD
again, remember assumption that unobserved

heterogeneity is additive and constant in time

DD = [ a + g + fi + b +d*x] - [ a + g + d*x ] - { [ a + fi + d*x ] - [ a + d*x ]
= fi + b - { fi } = b

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

D-D: An example

Piero Stanig

effect of gender quotas on quality of politicians

conventional wisdom: affirmative action reduces

quality
exploit historical event in Italy: gender quotas in local

elections introduced in 93 and then struck down by

Constitutional Court in 95
(notice importance of context-specific knowledge)
quotas in place only for 2 years
municipal elections non-concurrent only some

municipalities hold election with quotas in place

compare pre- and post- in municipalities that did run an

election under quotas with those that ran them without

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Corroborating plausibility

Piero Stanig

do quotas bind?
are treated and controls alike?
can one claim parallel trends?

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Outcomes and results

Piero Stanig

main outcome of interest: years of education of elected

additional outcomes:
proportion of high-skilled among elected
reelection rates
elected pursuing higher office
quotas increase quality of politicians
decomposing the effect:
more women elected
higher quality men elected
quotas dont affect average quality of elected women

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Summary and plan for tomorrow

Piero Stanig

if experiment not possible, try to mimic effects of

random assignment
use observational data and statistical model to create

adequate control group

otherwise, use pre- and post notice: D-D and matching can be used together
tomorrow: natural experiments proper
exploit randomness that is out there
exploit known assignment mechanisms

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Natural experiments

Piero Stanig

experiment proper requires

1 random assignment to treatment by researcher
2 intervention to manipulate value of treatment variable
what happens if nature performs assignment and

manipulation?
actual randomization (e.g., lottery, weather, death)
as-if (or A(s) G(ood) A(s) R(andom)): can claim (but not

test) that assignment does not depend on potential

outcomes

INSTITUTIONS,
GOVERNMENT AND
SOCIETY
Piero Stanig

The classic natural experiment:

Snows study of cholera

theoretical question: do miasmas or germs cause

cholera?
natural experiment: households in parts of London

assigned to one of two water providers

one water company draws water upstream, one

downstream

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Instrumental variables

Piero Stanig

workhorse method in econometrics

originally devised to reconstruct demand curves
use weather as instrument for fish price...
IV can be recast in Rubin causal framework
standard way of analyzing encouragement design: ITT

(intention to treat)
instead of estimating effect of T itself, estimate effect of

encouragement: covariance of outcome and

encouragement
example: instead of effect of smoking on health, effect

of campaign to quit smoking on health

INSTITUTIONS,
GOVERNMENT AND
SOCIETY
Piero Stanig

IV as encouragement design:
LATE
Encouragement Z=0 Z=1
Never-takers
T=0 T=0
Compliers
T=0 T=1
Always-takers
T=1 T=1
Defiers
T=1 T=0
if info about compliance available, can estimate effect
of actual treatment by:
W =

cov(Y , Z )
cov(T , Z )

numerator is ITT, denominator is compliance rate

same as IV estimator
estimates L(ocal) A(verage) T(treatment) E(ffect)
only require that there are no defiers (monotonicity

assumption)

(1)

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

An example: the effect of

military service

Piero Stanig

what is the effect of military service on future income?

treatment endogenous: features of the person that

affect both decision to serve and future behavior

during Vietnam war, lottery to select who is drafted
not all the lottery winners go to war; some non-drafted

volunteer
result: serving in Vietnam reduces future earnings
notice LATE here: effect of serving in the military for

those who were randomly drafted

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Why it works and what can go

wrong

Piero Stanig

what would be the problem with cov(Y , T )?

always-takers 6= never-takers and compliers!
e.g., people who volunteer for the military are not like

people who go because of draft

random assignment (on expectation): among each

group defined by assignment to treatment, same

proportion of types
notice two distinct requirements:
1
2

random assignment of instrument (or AGAR)

exclusion restriction: no other path from instrument to
treatment

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

RDD: when treatment

assignment mechanism is
known

Piero Stanig

assignment to treatment based on whether some

individual score s is above threshold v

examples
1
2
3

to be mayor, win election

to receive a fellowship, need good average
to get anti-poverty program, income below threshold

units with forcing variable s just below and just above

threshold very similar

if potential outcomes are smooth around the threshold
[ Note that 1 and 0 denote treatment, not time!! ]

lim E(Y (1)|s) lim E(Y (0)|s) =

(2)

E(Y (1)|s = v ) E(Y (0)|s = v ) = v

(3)

sv +

sv

Differenza tra DD e RDD: Es nel paper sul voto dei polacchi lungo il confine ovvio che devi usare un rdd
method perch non vuoi misurare come cambiata nel tempo un oggetto ma vuoi vedere gli effetti del
trattamento su un gruppo di ex prussiani e uno di ex russi.

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

RDD in action

500

1000

1500

2000

Piero Stanig

20

40

60
s

80

100

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

RDD in action: using window

500

1000

1500

2000

Piero Stanig

window

20

40

60
s

80

100

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

RDD in action

500

1000

1500

2000

Piero Stanig

20

40

60
s

80

100

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

RDD in action: parametric

500

1000

1500

2000

Piero Stanig

20

40

60
s

80

100

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

RDD: cautionary notes and

good practices

Piero Stanig

effect estimated for specific sub-population: units at the

threshold
this not necessarily effect one is after: e.g.,
effect of holding office for those who barely win 6= effect
of holding office after landslide (mandate)
effect of fellowship on students that barely made it 6=
effect of[ fellowship
on very good students
ordinare ]
problem of sorting: e.g., scores are manipulated to be

just above/below
mean reversion: negative shock (or bad luck) does not

persist forever
if Y = f (s) has lot of discontinuities, easy to find jumps
good practice:
show that theres no sorting
use placebo discontinuities, show nothing happens
if fit complicated polynomial, less credible than with
window

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Effect of foreign vs. local rule

Piero Stanig

does devolving power to locals reduce resistence to

foreign rule?
nice natural experiment: parts of France under Vichy

(puppet state), parts under direct German rule

close enough to the border between the two, border is

as-if random
or, in technical terms, potential outcomes are

continuous
(if you want: because people near the border have

continuous features)

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Main analysis

Piero Stanig

select some departments

measure outcomes (sabotage and fighting)
measure covariates (distance from train station,

telegraph, political leaning in 1936 election, etc.)

show that close to discontinuity, treated and controls

similar
2 show that close to discontinuity, resistence differs
between treated and controls
3 show robustness to specific estimator
4 claim causality
1

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Additional remarks

Piero Stanig

notice importance of robustness checks using many

methods
particularly important:
1
2

estimate with some fake (placebo) locations of border

show that there is no effect there

understand the mechanism (cooptation)

with interaction
[ propensione ]

between treatment and political leanings of area

in German-occupied area: both left and right join

resistence
in Vichy only left resists (because right is coopted)
notice caution of authors: the mechanism evidence is

not causal

[ La cooptazione un metodo per

la scelta dei nuovi membri di un
organo collegiale, consistente nella
loro elezione da parte dell'organo
stesso ]

INSTITUTIONS,
GOVERNMENT AND
SOCIETY

Summary and preview

Piero Stanig

nature provides situations that resemble experiments

can also exploit known treatment assignment
next week Wednesday:
lab experiments
questions and answers regarding causal estimation
methods
next week Thursday: using surveys