Kel 7 3DHM EvaluationMethods

Module 9 Quality Improvement
and Evaluation
Unit 3 Evaluation Methods

Sylvia Sax, Barbara Kloss-Quiroga
Facilitators Manual
Dr. Barbara Kloss-Quiroga (Ed.), Berlin 2003
Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.
Evaluation
To enable participants
to understand and apply methods for
evaluation and
to improve their personal and technical
skills in evaluation
District Health Management Tools, T 931
Specific Objectives
At the end of the unit the participants will be able to
identify concepts key to evaluation
list key questions for evaluation
describe the differences and linkages between monitoring
and evaluation activities
identify three types of evaluation methods used within the
health sector for evaluating quality
plan and practice an evaluation at the health district level
using one of these methods
describe the strengths and weaknesses inherent in these
evaluation methods
History and Introduction

Quantitative basis of many evaluations
General nature of evaluation
Judgement and value
Evaluation: A Working Definition

Attributing value to an intervention
Gathering reliable and valid information in a
systematic way
by making comparisons (i.e. to standards,
guidelines, pathways)
for the purposes of making more informed
decisions
and/or understanding causal mechanisms or
general principles
and/or assessing if goals have been met

Source: Adapted from Ovretveit, John: Evaluating health interventions, Open University Press,
Buckingham, 1998
Evaluation
A process which attempts to determine,

as systematically and objectively as
possible, the relevance, adequacy,
effectiveness, efficiency, and/or health
impact of health services or their
components in the light of predetermined
objectives (standards, criteria or the like)

Source: Adapted from Schrettenbrunner, A., Harpham, T.: A different approach to evaluating PHC
projects in developing countries, Health Policy and Planning 8/1993, pp. 128-135
Evaluation
Pre-determined purpose for evaluation

Often one-off process, clear beginning and end
Involves collection and analysis of set of
information that leads to a decision
Usually involves some form of comparison
(i.e. to standards, guidelines, pathways)
For the purposes of making more informed
decisions

Source: Adapted from Schrettenbrunner, A., Harpham, T.: A different approach to evaluating PHC
projects in developing countries, Health Policy and Planning 8/ 1993, pp. 128-135
Monitoring
Regular and routine collection
Information to be collected has clear definitions/
parameters
Can be multiple uses for the information
Tends to be quantitative data (numbers, units etc.)
A purpose for the data is often for comparing similar
organisations/units
Assess achievement of activities of the operational plan
The Planning (Management) Cycle

Evaluating
Implementing
Situational Analysis
MONITORING
Priority Setting
uncertain events
Programming
1. Situational Analysis
Planning
2. Priorit y Setting
INPUT
Income/
Vision
Input/
Structure
3. Planning
4. Progr amming
5. Implementing
6. Evaluating
PROCESS
Direction
Supervision
Targets
OUTPUT
Activites
Output/
Results
Outcome/
Impact
MONITORING
Morbidity
Mortality
Image of
Services
Accessibility
Availability
Baseline
Productivity
Use
Utilisation
Quality
Progress control
Coverage
Efficiency
Efficacy
Objectives based
evaluation
Effectiveness
Morbidity
Mortality
Image
Impact
evaluation

Source: Barbara Kloss-Quiroga
Questions for Evaluation Design

Why undertake the evaluation?
Who is the evaluation for?
When do we evaluate?
How do we evaluate?
Who evaluates?
What will become of the results?
Why Evaluate?
Needs assessment
Situational analysis
Evaluation of objectives
Change management
Quality assurance
Comparison of results and impacts of health programs
Comparison of health care structures
Establishment of new initiatives
Suggest more purposes for evaluation in health.
Who Evaluates?
Peers (including organisations and professionals)
Private professional auditors/consultants
Purchasers/Funders/Insurance
Consumers/Patients
Governments
The organisation or group itself
How Do We Evaluate?
Multiple Methods
Administrative Cycle is one method
1. identifying the problem
2. clarifying and analysing the problem
3. collecting the necessary information
4. implementing judgment-based changes
5. controlling and advising the changes
Evaluation Framework
Types of Evaluation
Questions Answered by the

Different Types of Evaluation
Formative Evaluation
Is an intervention needed?
(Determines Concept and Design)
Who needs the intervention?

How should intervention be carried out?
Process Evaluation
(Monitors Inputs and Outputs;
Assesses Service Quality)
To what extent are planned activities

actually realised?
How well are the services provided?
Effectiveness Evaluation
What outcomes are observed?
(Assesses Outcome and Impact)
What do the outcomes mean?

Does the program make a difference?
Cost-Effectiveness Analysis
(Including Sustainability Issues)
Should program priorities be changed or

expanded?
To what extent should resources be
reallocated?

Source: Rehle, T., Saidel, T., Mills, S., Magnani, R. (editors): Evaluating Programs for HIV/AIDS
prevention and care in developing countries, Family Health International, Washington D.C., 2003
Who Evaluates?
Individuals
Technical knowledge and skills
Professional and personal qualifications
Evaluation knowledge and skills
Teams
Complimentarity
Team building
Four Phases of Evaluation
Desk Study
An examination of key provider documents relevant to
the standards/criteria being assessed
Conducted by the evaluation team prior to the site visit
Provides an understanding of the organisation,
programme or persons activity
Provides theoretically adequate assessment as a basis
for an on-site visit but incomplete as an assessment tool
in itself
Documents should show evidence that they are
approved, implemented, reviewed, and change
controlled
Other Important Terminology:

Policy
Provides direction and/or guidance for
the actions of those working in the
organisation/system
Other Important Terminology:

Procedure
Outlines a process for key activities, often includes the
following headings:
Purpose:
objective/intent of the procedure
Scope:
application/boundary of what the

procedure will accomplish
References:
other documents, people/links

relevant to this procedure
Definition of key terms used in the description of the procedure

Procedure/method:
who, what, where, when, how; the

step-by-step description of the procedure
Associated documents: names of important forms, records etc.

associated with the procedure

Kel 7 3DHM EvaluationMethods

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Kel 7 3DHM EvaluationMethods

Uploaded by

Copyright:

Available Formats

Module 9 Quality Improvement

Unit 3 Evaluation Methods

District Health Management Tools, T 931

District Health Management Tools, T 932

History and Introduction

District Health Management Tools, T 933

Evaluation: A Working Definition

District Health Management Tools, T 934

A process which attempts to determine,

District Health Management Tools, T 935

Pre-determined purpose for evaluation

District Health Management Tools, T 936

District Health Management Tools, T 937

The Planning (Management) Cycle

District Health Management Tools, T 938

Questions for Evaluation Design

District Health Management Tools, T 939

District Health Management Tools, T 9310

District Health Management Tools, T 9311

District Health Management Tools, T 9312

Questions Answered by the

(Determines Concept and Design)

Who needs the intervention?

To what extent are planned activities

What outcomes are observed?

(Assesses Outcome and Impact)

What do the outcomes mean?

Should program priorities be changed or

District Health Management Tools, T 9313

District Health Management Tools, T 9314

Four Phases of Evaluation

District Health Management Tools, T 9315

Other Important Terminology:

District Health Management Tools, T 9317

Other Important Terminology:

objective/intent of the procedure

application/boundary of what the

other documents, people/links

Definition of key terms used in the description of the procedure

who, what, where, when, how; the

Associated documents: names of important forms, records etc.

District Health Management Tools, T 9318

You might also like