This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License.

Your use of this

material constitutes acceptance of that license and the conditions of use of materials on this site.

Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. Use of these materials
permitted only in accordance with license rights granted. Materials provided AS IS; no representations or
warranties provided. User assumes all responsibility for use, and all liability related thereto, and must independently
review all materials for accuracy and efficacy. May contain materials owned by others. User is responsible for
obtaining permissions for use from third parties as needed.

Fundamentals of Program Evaluation

Course 380.611
Developing a Conceptual Framework,
and Introduction to Formative
Research

Topics to cover today

Importance of a conceptual framework

Key points from Earp and Ennett article
Examples of conceptual frameworks
In-class group discussion on CFs
Discussion of Assignment #1
Introduction to formative research

Importance of a conceptual
framework (program theory)

Articulates the pathways by which an

intervention is expected to cause the
desired outcomes
Provides evaluator with specific
elements to assess
Other names:

Logic model, program model, outcome line,

cause map, action theory

Conceptual Framework of Family

Planning Demand and Program
Impact on Fertility
Societal and
Individual
factors

Value and
Demand
for Children

FP Demand
Spacing
Limiting

Other
Intermediate
Variables

Fertility
Wanted
Unwanted

Contraceptive
Practice
Development
Programs
Family Planning
Supply Factors

Service Outputs:
Access
Quality
Image/
Acceptability

Service
Utilization

Other Health
and Social
Improvements

Conceptual models:
Earp and Ennett (1991)

Definition of a conceptual model:

Diagram of proposed causal linkages among a set

of concepts believe to be related to a particular
public health problem.

Concepts = in boxes
Processes = shown by arrows
Can reflect factors at multi levels (macro to
micro)

Conceptual models

Often draw on:

One or more theories

Empirical evidence
Knowledge specific to the particular case

Serve to:

Summarize and integrate knowledge

Provide explanations for causal linkages
Generate hypotheses

Building a conceptual model

Start with the endpoint (dependent

variable, outcome, or target point for
intervention)
Identify potential correlates, based on
empirical or theoretical evidence
Show antecedent or mediating variables
by proximity to dependent variables

Conventions for drawing a

conceptual model
1)

2)
3)
4)
5)

Only include concepts that will be

operationally defined and measured
Present left-to-right or top-to-bottom
Use arrows to imply causality
Label concepts succinctly
Do not include operational definitions
or values of variables in the model

How to think through a

conceptual framework

Example: Tobacco Prevention and

Control

Interventions to Reduce Exposure to

Environmental Tobacco Smoke
http://www.thecommunityguide.org/tobacc
o/tobac.ppt

Tobacco use is the single largest cause of preventable premature mortality in the United States. It also
represents an enormous cost burden to the nation. The question is, what works to make tobacco use
prevention and control at the population or community level? The Guide to Community Preventive
Services addresses the effectiveness of community-based interventions for three strategies to promote
tobacco use prevention and control: 1) prevent tobacco product use initiation, 2) increase
cessation and 3) reduce exposure to environmental tobacco smoke (ETS). The findings
strengthen and complement existing guidelines (hyperlink table and text to existing guidelines) on
tobacco prevention and control.

Analytic Framework

For every intervention that we evaluate in the

Community Guide, we develop an analytic
framework, in which we postulate how we think the
intervention works and what outcomes we think are
important to capture information on. In many cases,
our analytic frameworks change over the course of
our reviews as we learn more about the intervention,
the potential outcomes, and the body of the evidence
in the literature.
Lets start with our intervention: Smoking bans
And our goal: A reduction in morbidity and
mortality.. Our analytic framework will connect these
two.

Analytic Framework: Smoking Bans

Smoking
Bans

Reduced
Morbidity
and
Mortality

Smoking Bans Might Result from

Community Education Efforts

Now its important to recognize that smoking bans

might be the result or outcome of interventions, such
as a community-wide education and/or political
campaign.
The experience from the state of California with a
broad smoking ban has been described quite well in
the literature, giving you a blow by blow account of
how that state managed to adopt and implement and
extend a statewide clean indoor air laws.
Its also important to note that in many states, preemption legislation precludes local governments from
strengthening clean indoor air laws. These efforts
have been described as a industry-sponsored effort
to obstruct efforts to extend protections from ETS to
workers.

Smoking Bans Might Result from Community

Education Efforts

Community
Education

Smoking
Bans

Reduced
Morbidity
and
Mortality

Bans Might Reduce ETS Exposure

Now back to how smoking bans work.

They work in one or three ways.
First, directly by reducing exposure to
ETS in the restricted environment. This
alone will have health effects.

Bans Might Reduce ETS Exposure

Reduced
Exposure to ETS

Smoking
Bans

Reduced
Morbidity
and
Mortality

Bans Might Increase Smoking

Cessation

Second, smoking bans might work by affecting the

tobacco use behaviors of smokers
In response to a smoking ban, they might think twice
about continuing their habit. They might reduce their
daily consumption of tobacco, and these two effects
might increase the number who attempt to quit.
Since smoking bans also reduce ques to smokers to
relapse, more smokers attempting to quit will be
successful. This will result in fewer tobacco users
and a reduction in adverse health outcomes.

Bans Might Increase Smoking Cessation

Reduced
Exposure to ETS

Fewer
Tobacco
Users

Smoking
Bans

Change
In
Attitudes

Reduced
Consumption

Increased
Quit
Attempts

Increased
Cessation

Reduced
Morbidity
and
Mortality

Bans Might Reduce Smoking Initiation

Finally, we acknowledge that smoking

bans might directly affect tobacco
consumption by youth or affect their
impressions of the social desirability of
smoking. These will reduce tobacco
use prevalence among adolescents and
contribute to fewer tobacco users.

Bans Might Reduce Smoking Initiation

Reduced
Exposure to ETS

Smoking
Bans

Change
In
Attitudes

Change
In
Attitudes

Reduced
Initiation

Fewer
Tobacco
Users

Reduced
Consumption

Increased
Quit
Attempts

Increased
Cessation

Reduced
Morbidity
and
Mortality

Bans Might Increase ETS in the Home

One unintended effect described in the literature, at

least initially, was a concern that smokers might
respond to workplace smoking restrictions by
compensating at home, smoking more and thus
increasing ETS exposures in the home.
We looked for evidence of this potential harm in our
review.
We also examined the evidence, not shown here,
that smoking bans in restaurants and hotels
adversely affects business revenue and tourism.

Bans Might Increase ETS in the Home

Reduced
Exposure to ETS

Smoking
Bans

Change
In
Attitudes

Change
In
Attitudes

Reduced
Consumption

Reduced
Initiation

Fewer
Tobacco
Users

Increased
Quit
Attempts

Increased
Cessation

Diverted
Consumption

Increased
Home Exposure

Reduced
Morbidity
and
Mortality

Body of Evidence: Bans and

Restrictions

We did a series of electronic database searches, and

screened titles and abstracts and ended up with the
following body of evidence:

56 studies were reviewed

17 studies measured differences or changes in ETS
exposure, of which 10 met our criteria for good or fair
quality
51 studies measured smoking habits of employees exposed
to bans or restrictions, of which only 9 met our criteria for
good or fair. In most cases, the excluded studies did not
included concurrent comparison groups.

Study Measurements by Outcome

If you take all of the qualifying studies, and

plotted their outcomes to slots on our analytic
framework this is what we found across this
body of evidence.
For example we have 12 measurements of
differences or changes in exposure to ETS, 6
studies of changes in tobacco use prevalence
among employees, 4 measurements of
cessation by smoking employees etc.

Study Measurements by Outcome

12

Reduced
Exposure to ETS

0*
Change
In
Attitudes

Smoking
Bans

0*

9
Change
In
Attitudes

Reduced
Consumption

Reduced
Initiation

6
Fewer
Tobacco
Users

Increased
Quit
Attempts

Increased
Cessation

Diverted
Consumption

Increased
Home Exposure

Reduced
Morbidity
and
Mortality

Examples of different conceptual

frameworks

Model of Program Impact

Socioeconomics
Status

Knowledge

Gender
Income
Education
Psychographic
Characteristics

Program
Exposure

Practice

Family
Characteristics
Interpersonal
Contacts

Attitude

Determinants of Domestic Violence

(no intervention)
Contextual and
Community Factors

Household and IndividualLevel Factors

Socioeconomic
development

Socioeconomic
status

Domestic violence
norms

Life cycle factors

Gender inequality

Intergenerational
exposure to violence

Crime levels

Risk behaviors

Domestic
violence

Womens status/
autonomy

Conceptual
Framework
Conceptual
Framework
Communication
to a Health-Competent
Pathways toPathways
a Health-Competent
SocietySociety
Domains for
Communication Interventions

Communication to
Strengthen the
Social Political
Environment

Communication
for Effective
Service
Delivery
Systems

Communication
to Create
Health Literate
Communities
and Individuals

Conceptual Framework
Communication Pathways to a Health-Competent Society
Domains for
Communication Interventions

CommunicationSocial Political
Environment

Communication
for Service
Delivery
System

Initial Outcomes

Environment

Service Systems

Community
Communication
for Community/
Individual

Individual

Conceptual Framework
Communication Pathways to a Health-Competent Society
Domains for
Communication Interventions

CommunicationSocial Political
Environment

Communication
for Service
Delivery
System

Initial Outcomes

Environment

Behavioral
Outcomes

Supportive
Environment

Service
Performance
Service Systems

Client
Behaviors:
Community
Community
Communication
for Community/
Individual

Individual

Individual

Conceptual Framework
Communication Pathways to a Health-Competent Society
Domains for
Communication Interventions

CommunicationSocial Political
Environment

Communication
for Service
Delivery
System

Initial Outcomes

Environment

Behavioral
Outcomes

Supportive
Environment

Service
Performance
USAID 5 SOs

Service Systems

Client
Behaviors:
Community
Community
Communication
for Community/
Individual

Sustainable
Health Outcomes

Individual

Individual

Conceptual Framework
Communication Pathways to a Health-Competent Society
Underlying
Conditions

Domains for
Communication Interventions

CommunicationSocial Political
Environment

Initial Outcomes

Environment

Behavioral
Outcomes

Sustainable
Health Outcomes

Supportive
Environment

Context

Communication
for Service
Delivery
System

Service
Performance
USAID 5 SOs

Service Systems

Client
Behaviors:
Community
Resources
Community
Communication
for Community/
Individual

Individual

Individual

Conceptual Framework
Communication Pathways to a Health-Competent Society

Environment

Social

CommunicationSocial Political
Environment

Initial Outcomes

Political will
Resource allocation
policy changes
Institutional capacity
building
National coalition
National communication
strategy

Service Systems

Context
Disease Burden

Domains for
Communication Interventions

Availability
technical competence
Information to client
Interpersonal
communication
Follow-up of clients
Integration of services

Community

Underlying
Conditions

Leadership
Participation equity
Information equity
Priority consensus
Network cohesion
Ownership
Social norms
Collective efficacy
Social capital

Economic
Communication
Technology
Political
Legal

Resources

Communication
for Service
Delivery
System

Human and
Financial
Resources
Strategic
Plan/Health
Priorities
Other Development
Programs
Policies

Communication
for Community/
Individual

Individual

Cultural

Message recall
Perceived social
support/stigma
Emotion and values
Beliefs and attitudes
Perceived risk
Self-efficacy
Health literacy

Behavioral
Outcomes

Sustainable
Health Outcomes

Supportive
Environment:
Multi-sectoral
partnerships
Public opinion
Institutional
performance
Resource acquisition
Media support
Activity level

Service
Performance:
Access
Quality
Client volume
Client satisfaction

Client
Behaviors:
Community
Sanitation
Hospice/PLWA
Other actions

Individual
Timely service use
Contraception
Abstinence/partner
reduction
Condom use
Safe delivery
BF/nutrition
Child care/immuniz.
Bednet use

USAID 5 SOs
Reduction in:
Reduction in:
Unintended/mistimed
Pregnancies
Morbidity/mortality
From pregnancy/
Childbirth
Infant/child
morbidity/mortality
HIV transmission
Threat of infectious
diseases

Requirement for
exercise #1

Present the diagram in terms of initial,

intermediate, and long-term outcomes
Note: this is NOT a standard
requirement of conceptual frameworks
but it is a useful way to look at program
effects.

Criteria for grading conceptual

framework on exercise #1

Diagram respects the 5 conventions for

drawing a conceptual model
The model presented is:

Conceptually clear (explains to the reader

how you expect the program to achieve its
objectives)
Visually pleasing
Concise but covers key factors (suggestion:
include 10-15 concepts in your model)

Rules relating to confounding and

modifying variables (Earp & Ennett)

See page 169 of the article

Technically fine, but not necessarily
used among all researchers
In exercise #1, dont feel bound by
these two rules.

Formative evaluation

Guides the design of a program

Different types:

Needs assessment (esp. in U.S.)

Diagnostic (formative) research
(Specific to media) Pretesting

Needs assessment in the

program cycle (McDavid)

Strategic Planning

Program Development

Stakeholder Input

Program Implementation
Program Evaluation

Environment Scanning
Stakeholder Input

Stakeholder assessments of services/outputs in

relation to needs (relevance)

Program Accountability

Steps in conducting a needs

assessment (McDavid & Hawthorn)

Become familiar with political context

Identify users and uses
Identify target pop. (geographic, socio-dem)
Inventory existing services (what gaps exist?)
Identify needs
Prepare document

Evidence, benchmarks, conclusions, recs

Communicate findings, implement

Use of benchmarks in needs

assessment

Compare current levels and types of

services to benchmarks (or reference
points)

Conceptions of human needs

Moral/ethical values (no child left behind)
Levels of service provided elsewhere
Service provider opinions/preference
Client (current, prospective) opinions

Sources of data: primary

(new) & secondary (existing)

Lit reviews

Similar studies
Demographic statistics
Government reports

Surveys (mail, phone, in person)

Focus groups
Interviews
Direct observation

Diagnostic research (very

similar to needs assessment)

Also called formative research or

formative evaluation
Learn more about all aspects of the
problem, population, and context

Diagnostic research uses both

quantitative & qualitative

Quantitative (demographic,
epidemiological):

To quantify the extent of the problem

To identify subgroups most affected
To identify explain determinants

Qualitative:

To understand problem from user

perspective, identify barriers

Great diversity in types of

formative research

Examples:

Formative research for Stop Aids Love Life

Louisiana study on teen smoking behavior

Investigation of places with high rates of

new partner acquisition (PLACE
methodology)

Publication of formative
research in peer-reviewed lit

Quite rare
Results often presented in a report

More likely in form of baseline findings

Ex: Stop AIDS Love Life

Louisiana adolescent smoking study

If value goes beyond study location

PLACE methodology in S. Africa

Key points from Louisiana

smoking study

National surveys of adolescent smoking

didnt provide adequate data on target
population
Survey of 4808 students provided data:

Smoking patterns by ethnic group, gender

Social relationships related to smoking

Friends, family; smoking and alcohol

Example of a baseline survey

as a two-fer (two for one)

Formative research in form of baseline

survey serves two purposes:

Establishing a baseline level against which

to evaluate program after intervention
Providing insights into the problem that
help to guide the design of the program

Findings from LA study useful

in developing intervention

LA rate higher than national rate for

adolescents
Who was most likely to smoke:

Among whites: no male/female differences

Among blacks: males more likely to smoke
Both black and whites:

Discretionary $$
Low academic achievers

Findings from LA study useful

in developing intervention

Strong relationship of smoking to:

Smoking of family & friends, alcohol use

Authors discuss challenges of designing

a program with these dynamics
Formative research doesnt give all the
answers to program design!

PLACE (priorities for local AIDS

control activities) methodology

MEASURE Evaluation Project (UNC)

Identifies where to access sexual
networks with individuals with high
rates of new partner acquisition
Provides information on availability of
preventive services (info, condoms)

Methods of PLACE: 3 phases

of data collection

Key informants: where do people meet

new sexual partners?

Visit to sites compiled from interviews

Community leaders, health care providers,

youth on street, taxi drivers, STD clients
Type of site, patrons, AIDS prevention?
Sites marked on aerial map

Interviews with people at these sites

Useful information for

designing an intervention

Key locations: taverns and shebeens

<2% of sites had on-site anti-AIDS info
<10% didnt have condoms onsite
Almost 60% of owners/managers would
be willing to have condoms onsite
Patrons at these locations frequent
visitors (regulars)

Indicators useful for

monitoring programs

# new sites identified sexual activity

% of sites with condoms (verified)
Mean rate of new partnership formation
at site in past 4 weeks, by gender
Portion of patrons who ever used a
condom
Portion used condom at last sex

Use of qualitative research to

inform quantitative

To learn vocabulary used by local

population to describe problem

Yoder study on diarrhea: 9 different words

To identify new concepts that

researchers hadnt considered
To generate hypotheses to be tested
through subsequent research