HPPH Systematic Review Handbook

CochraneHealthPromotion
andPublicHealthField
www.vichealth.vic.gov.au/cochrane
COURSE WORKBOOK
APublicHealthEducationand
ResearchProjectfundedbythe
AustralianGovernment
DepartmentofHealthand
Ageing
Author
NickiJackson
CochraneHealthPromotionandPublicHealthField
VictorianHealthPromotionFoundation
Acknowledgements
The production of this handbook was funded by a grant from the Australian Government Public
HealthEducationandResearchProgram(PHERP).
Thanks to those listed below for providing comments on drafts of the material included in the
Handbook:
ProfessorElizabethWaters
Director, Cochrane Health Promotion and Public Health Field, Chair in Public Health, Deakin
University
DrCeliaMcMichael
SchoolofPublicHealth,LaTrobeUniversity
DrLucieRychetnik
SydneyHealthProjectsGroup,TheUniversityofSydney
JohnBennett
Projectcoordinator,SchoolofPublicHealth,TheUniversityofSydney
The Victorian Health Promotion Foundation, Australia (www.vichealth.vic.gov.au), is also
acknowledgedforthesupportprovidedduringthecompletionoftheHandbook.
The author has also utilised the work conducted by the Guidelines for Systematic Reviews in Health
PromotionandPublicHealth.Thistaskforceincludes:
Anderson L. (Centers for Disease Control and Prevention, USA); Bailie R. (Menzies School of Health
Research and Flinders University NT Clinical School, Australia); Brunton G. (Evidence for Policy and
Practice Information and Coordinating (EPPI) Centre, UK); Hawe P. (University of Calgary, Canada);
JacksonN.(CochraneHealthPromotionandPublicHealthField,Australia);KristjanssonE.(University
ofOttawa,Canada);NaccarellaL.(UniversityofMelbourne,Australia);NorrisS.(AgencyforHealthcare
Research and Quality, USA); Oliver S. (EPPICentre, UK); Petticrew M. (MRC Social and Public Health
Sciences Unit, UK); Pienaar E. (South African Cochrane Centre); Popay J. (Lancaster University, UK);
RobertsH.(CityUniversity,UK);RogersW.(FlindersUniversity,Australia);ShepherdJ.(Universityof
Southampton,UK);SowdenA.(CentreforReviewsandDissemination,UniversityofYork,UK);Thomas
H.(McMasterUniversityandtheEffectivePublicHealthPracticeProject,Canada);WatersE.(Cochrane
HealthPromotionandPublicHealthFieldandDeakinUniversity,Australia).
Copyright
The copyright for the handbook lies with Deakin University and the Australian Department of
Health and Aging. The course materials may be reproduced and used to conduct non-profit
systematic review courses for the Australian public health workforce. The materials should not be
used for any commercial or profit-making activity unless specific permission is granted by the
copyright owners.
Contents
Introduction ......................................................................................................................................... 1
UnitOne:BackgroundtoSystematicReviews................................................................................ 3
UnitTwo:InternationalSystematicReviewInitiatives ................................................................. 7
UnitThree:ResourcesRequired........................................................................................................ 9
UnitFour:DevelopingaProtocol ................................................................................................... 13
UnitFive:AskinganAnswerableQuestion .................................................................................. 17
UnitSix:FindingTheEvidence....................................................................................................... 25
UnitSeven:DataAbstraction .......................................................................................................... 43
UnitEight:PrinciplesofCriticalAppraisal ................................................................................... 45
UnitNine:SynthesisingtheEvidence ............................................................................................ 79
UnitTen:InterpretationofResults ................................................................................................. 83
UnitEleven:WritingtheSystematicReview ................................................................................ 91
Introduction
Thishandbookprovidesaworkingframeworktoconductasystematicreviewofahealthpromotion
orpublichealthintervention.Thepurposeofthishandbookistodescribethestepsofthesystematic
reviewprocessandprovidesomeworkingexamplestopracticepriortocommencingareview.The
handbook, however, is not intended to be used as a single resource for conducting reviews.
Reviewers are recommended to source additional information from other review
handbooks/guidancemanuals(highlightedbelow),particularlyforissuesrelatingtodataanalysis.
Note: This handbook is useful for both Cochrane reviewers and reviewers who are completing a
systematicreviewfortheirworkplace,studies,etc.IfyouwishtocompleteaCochranereview,please
visit www.cochrane.org (About us Contact: Groups and Centres) to find the appropriate
CollaborativeReviewGrouptoregisteryourinterestorcontacttheCochraneHealthPromotionand
PublicHealthFieldforfurtherinformationcochrane@vichealth.vic.gov.au.
Overalllearningoutcomes
Workingthroughthishandbookwillenableyouto:
Befamiliarwithsomeofthekeychallengesofconductingsystematicreviewsofhealth
promotionandpublichealthinterventions
Formulateananswerablequestionabouttheeffectivenessofinterventionsinhealth
promotionandpublichealth
Identifyprimarystudies,includingdevelopingevidencebasedstrategiesforsearching
electronicdatabases
Evaluatethequalityofbothanindividualhealthpromotionorpublichealthstudyanda
systematicreview
Synthesisethebodyofevidencefromprimarystudies
Formulateconclusionsandrecommendationsfromthebodyofevidence
Additionalreading:
Textbooks:
Oliver S, Peersman P. Using Research for Effective Health Promotion. Open University Press, UK.
2001.
Brownson R, Baker E, Leet T, Gillespie K. Evidencebased Public Health. Oxford University Press,
USA.2003.
Egger M, Smith G, Altman D. Systematic Reviews in Health Care: Metaanalysis in context. BMJ
PublishingGroup,UK.2001.

Manuals/Handbooks:
Cochrane Collaboration OpenLearning Materials for Reviewers. Version 1.1, November 2002.
http://www.cochranenet.org/openlearning/
Clarke M, Oxman AD, editors. Cochrane Reviewers Handbook 4.2.0 [updated March 2003].
http://www.cochrane.org/resources/handbook/index.htm
Undertaking Systematic Reviews of Research on Effectiveness. CRDs Guidance for those Carrying
OutorCommissioningReviews.CRDReportNumber4(2ndEdition).NHSCentreforReviewsand
Dissemination,UniversityofYork.March2001.http://www.york.ac.uk/inst/crd/report4.htm
Evidence for Policy and Practice Information and Coordinating Centre Review Group Manual.
Version1.1,SocialScienceResearchUnit,InstituteofEducation,UniversityofLondon.2001.
http://eppi.ioe.ac.uk/EPPIWebContent/downloads/RG_manual_version_1_1.pdf
HedinA,andKallestalC.Knowledgebasedpublichealthwork.Part2:Handbookforcompilationof
reviews on interventions in the field of public health. National Institute of Public Health. 2004.
http://www.fhi.se/shop/material_pdf/r200410Knowledgebased2.pdf
Unit One: Background to Systematic

Reviews
LearningObjectives
Tounderstandthetermssystematicreviewandmetaanalysis
Tobefamiliarwithdifferenttypesofreviews(advantages/disadvantages)
Tounderstandthecomplexitiesofreviewsofhealthpromotionandpublichealthinterventions
Typesofreviews
Generally,reviewsmaybegroupedintothefollowingtwocategories(seeTableOne):
1) Traditionalliteraturereviews/narrativereviews
2) Systematicreviews(withorwithout)metaanalysis
Narrativeortraditionalliteraturereview
Theauthorsofthesereviews,whomaybeexpertsinthefield,useinformal,unsystematicand
subjectivemethodstocollectandinterpretinformation,whichisoftensummarisedsubjectively
andnarratively.2Processessuchassearching,qualityappraisalanddatasynthesisarenotusually
described and as such, they are very prone to bias. Although an advantage of these reviews is
thattheyareoftenconductedbyexpertswhomayhaveathoroughknowledgeoftheresearch
field, but they are disadvantaged in that the authors may have preconceived notions or biases
andmayoverestimatethevalueofsomestudies.3
Note: A narrative review is not to be confused with a narrative systematic review the latter
referstothetypeofsynthesisofstudies(seeUnitNine).
Systematicreview
ManyofthetoolsofsystematicresearchsynthesisweredevelopedbyAmericansocialscientists
in the 1960s.4 However, todays systematic evidence reviews are very much driven by the
evidencebasedmedicinemovement,inparticular,fromthemethodsdevelopedbytheCochrane
Collaboration. A systematic review is defined as a review of the evidence on a clearly
formulated question that uses systematic and explicit methods to identify, select and critically
appraise relevant primary research, and to extract and analyse data from the studies that are
includedinthereview.1
Whatisametaanalysis?
Ametaanalysisisthestatisticalcombinationofatleast2studiestoproduceasingleestimateof
the effect of the health care intervention under consideration.2Note: a metaanalysis is simply
thestatisticalcombinationofresultsfromstudiesthefinalestimateofeffectmaynotalwaysbe
theresultofasystematicreviewoftheliterature.Therefore,itshouldnotbeconsideredasatype
ofreview.
TableOne.Comparingdifferenttypesofreviews
Review
Traditional
literature
review/
narrative
review
Characteristics
Describesandappraises
previousworkbutdoesnot
describespecificmethodsby
whichthereviewedstudies
wereidentified,selectedand
evaluated
Systematic
review
Thescopeofthereviewis
identifiedinadvance(eg
reviewquestionandsub
questionsand/orsubgroup
analysestobeundertaken)
Comprehensivesearchto
findallrelevantstudies
Useofexplicitcriteriato
include/excludestudies
Applicationofestablished
standardstocritically
appraisestudyquality
Explicitmethodsof
extractingandsynthesising
studyfindings
Uses
Overviews,
discussions,
critiquesof
previousworkand
thecurrentgapsin
knowledge
Oftenusedas
rationalefornew
research
Toscopethetypes
ofinterventions
availableto
includeinareview
Identifies,
appraisesand
synthesisesall
availableresearch
thatisrelevanttoa
particularreview
question
Collatesallthatis
knownonagiven
topicandidentifies
thebasisofthat
knowledge
Comprehensive
reportusing
explicitprocesses
sothatrationale,
assumptionsand
methodsareopen
toscrutinyby
externalparties
Canbereplicated/
updated
Limitations
Thewriters
assumptionsand
agendaoften
unknown
Biasesthatoccurin
selectingand
assessingthe
literatureare
unknown
Cannotbe
replicated
Systematicreviews
withnarrowly
definedreview
questionsprovide
specificanswersto
specificquestions
Alternative
questionsthat
havenotbeen
answeredusually
needtobe
reconstructedby
thereader
Advantagesofsystematicreviews
Reducesbias
Replicable
Resolvescontroversybetweenconflictingfindings
Providesreliablebasisfordecisionmaking
Reviewsofclinicalinterventionsvs.reviewsofpublichealthinterventions
Someofthekeychallengespresentedbythehealthpromotionandpublichealthfieldareafocusor
emphasison;
populationsandcommunitiesratherthanindividuals;
combinationsofstrategiesratherthansingleinterventions;
processesaswellasoutcomes;
involvementofcommunitymembersinprogramdesignandevaluation;
healthpromotiontheoriesandbeliefs;
theuseofqualitativeaswellasquantitativeapproachestoresearchandevaluation;
thecomplexityandlongtermnatureofhealthpromotioninterventionoutcomes.5
REFERENCES
1.
Undertaking Systematic Reviews of Research on Effectiveness. CRDs Guidance for those

CarryingOutorCommissioningReviews.CRDReportNumber4(2ndEdition).NHSCentre
forReviewsandDissemination,UniversityofYork.March2001.
2.
KlassenTP,JadadAR,MoherD.GuidesforReadingandInterpretingSystematicReviews.1.
GettingStarted.ArchPediatrAdolescMed1998;152:700704
3.
Hedin A, and Kallestal C. Knowledgebased public health work. Part 2: Handbook for
compilation of reviews on interventions in the field of public health. National Institute of
PublicHealth.2004.http://www.fhi.se/shop/material_pdf/r200410Knowledgebased2.pdf
4.
Chalmers I, Hedges LV, Cooper H. A brief history of research synthesis. Eval Health Prof
2002;25:1237.
5.
Jackson SF, Edwards RK, Kahan B, Goodstadt M. An Assessment of the Methods and
ConceptsUsedtoSynthesizetheEvidenceofEffectivenessinHealthPromotion:AReviewof
17Initiatives.
http://www.utoronto.ca/chp/chp/consort/synthesisfinalreport.pdf
ADDITIONALREADING
MulrowCD.Systematicreviews:Rationaleforsystematicreviews.BMJ1994;309:597599.
McQueenD.Theevidencedebate.JEpidemiolCommunityHealth2002;56:8384.
PetticrewM.Whycertainsystematicreviewsreachuncertainconclusions.BMJ2003;326:7568.
Petticrew M. Systematic reviews from astronomy to zoology: myths and misconceptions. BMJ
2001;322:98101.
GrimshawJM,FreemantleN,LanghorneP,SongF.Complexityandsystematicreviews:reporttothe
USCongressOfficeofTechnologyAssessment.Washington,DC:OfficeofTechnologyAssessment,
1995.
RychetnikL,HaweP,WatersE,BarrattA,FrommerM.Aglossaryforevidencebasedpublichealth.J
EpidemiolCommunityHealth2004;58:53845.
Unit Two: International Systematic

Review Initiatives
LearningObjective
Tobefamiliarwithinternationalgroupsconductingsystematicreviewsoftheeffectivenessofpublichealth
andhealthpromotioninterventions
Thereareanumberofgroupsaroundtheworldconductingsystematicreviewsofpublichealthand
health promotion interventions. Reviews are often published on the groups internet website, and
followguidelines/methodsdevelopedbytheindividualorganisation.Itisusefultovisiteachofthe
organisations listed below to view the different styles of systematic reviews. Reviewers seeking to
conduct a Cochrane Review should visit the Cochrane website for more information
(http://www.cochrane.org) or contact the Cochrane Health Promotion and Public Health Field
(http://www.vichealth.vic.gov.au/cochrane/).
Usefulwebsitesofsystematicreviewinitiatives:
1.
TheCochraneCollaborationTheCochraneLibrary:
http://www.thecochranelibrary.com
ReviewsrelevanttohealthpromotionandpublichealtharelistedontheCochraneHealthPromotionand
PublicHealthFieldwebsite:
http://www.vichealth.vic.gov.au/cochrane
2.
GuidetoCommunityPreventiveServices:
http://www.thecommunityguide.org
3.
TheEvidenceforPracticeInformationandCoordinatingCentre(EPPICentre):
http://eppi.ioe.ac.uk/
4.
EffectivePublicHealthPracticeProject:
http://www.city.hamilton.on.ca/PHCS/EPHPP/EPHPPResearch.asp
5.
HealthDevelopmentAgency(HDA):
http://www.hdaonline.org.uk/html/research/effectiveness.html
Note: These reviews are systematic reviews of systematic reviews (not reviews of individual
primarystudies).
6.
CentreforReviewsandDissemination:
http://www.york.ac.uk/inst/crd/
7.
TheCampbellCollaboration
http://www.campbellcollaboration.org/
ADDITIONALREADING
SheaB,MoherD,GrahamI,PhamB,TugwellP.AcomparisonofthequalityofCochranereviews
andsystematicreviewspublishedinpaperbasedjournals.EvalHealthProf2002;25(1):11629.
Unit Three: Resources Required

LearningObjective
Tobefamiliarwiththeresourcesrequiredtoconductasystematicreview
Conductingasystematicreviewcanbeatimeconsumingtask.Ideally,aminimumofsixmonthsis
required to complete a review (fulltime). However, there will be times which are less busy, for
example,whenawaitingtheretrievaloffulltextarticles.Thefollowinglistoutlinestherequirements
tocompleteasystematicreview:
Topicofrelevanceorinterest
Teamofcoauthors(toreducebias)
Trainingandsupport
Accessto/understandingofthelikelyusersofthereview
Funding
Time
Accesstoelectronicsearchingdatabasesandtheinternet(forunpublishedliterature)
Statisticalsoftware(ifappropriate)
Bibliographicsoftware(eg.Endnote)
Wordprocessingsoftware
TheCochraneCollaborationsoftware,RevMan(abbreviationforReviewManager),canbeusedfor
boththetextofthereviewandmetaanalysis,andcanbedownloadedforfreefromhttp://www.cc
ims.net/RevMan.
Time
Althoughnoresearchhasbeencompletedontheoveralltimeittakestocompleteahealthpromotion
or public health systematic review, we are given some insight from an analysis of 37 medically
relatedmetaanalyses1.TheanalysisbyAllenandOlkin1foundthattheaveragehoursforareview
were1139(~6months),butrangedfrom216to2518hours.
Thecomponentmeantimeswere:
588hours
Protocoldevelopment,searches,retrieval,abstractmanagement,paperscreeningand
blinding,dataextractionandqualityscoring,dataentry
144hours
Statisticalanalysis
206hours
Reportandmanuscriptwriting
201hours
Other(administrative)
Therewasanobservedassociationbetweenthenumberofinitialcitations(beforeexclusioncriteria
areapplied)andthetotaltimeittakestocompleteametaanalysis.
Note: The time it takes to complete a health promotion and public health review may be longer due to less
standardised definitions (eg. concepts, language, terminology) for public health interventions compared to
clinicalinterventionsresultinginalargernumberofcitationstoapplytheinclusionandexclusioncriteria.
Searching
The EPPICentre2 documented the time it took an experienced health promotion researcher in
developingandimplementingaMedlinesearchstrategytoidentifysexualhealthpromotionprimary
studies.
40hours
DevelopingandtestingasensitivesearchstrategyforMedline
8hours
Implementing the search for the most recent Medline period available at the time
(January1996toSeptember1997)anddownloadingcitations
7hours
Scanningthroughthe1048retrievedrecords
Ifsuchasearchstrategywastobeimplementedoverthe30yearscoveredbyMedline,thenumberof
retrievedrecordswouldbearound10,000.Consequently,about70hourswouldbeneededtoidentify
therelevantcitationsforthereview.Overall,thisMedlinesearchstrategywouldtakeapproximately
120hours.
A preliminary literature search and contact with relevant experts in the area might help assist in
calculatingtheapproximatetimerequiredtocompletethereview.
REFERENCES
1. Allen IE, Olkin I. Estimating Time to Conduct a Metaanalysis From Number of Citations
Retrieved.JAMA1999;282(7):6345.
2. Evidence for Policy and Practice Information and Coordinating Centre. Research Report.
EffectivenessReviewsinHealthPromotion.1999.
10

Formulate review
question
Establish an
Advisory Group
Develop review
protocol
Initiate search strategy
Download citations to
bibliographic software
Apply inclusion and

exclusion criteria
Cite reasons for

exclusion
Obtain full reports and

re-apply inclusion and
exclusion criteria
Data abstraction
Quality appraisal
Synthesis of studies
Interpret findings
Full report
FigureOne.Flowchartofasystematicreview
11
12
Unit Four: Developing a Protocol

LearningObjectives
Tounderstandtherationalefordocumentingthereviewplanintheformofastructuredprotocol
Tounderstandtheimportanceofsettingtheappropriatescopeforthereview
Whatisaprotocol?
A protocol is the plan the reviewers wishes to follow to complete the systematic review. It allows
thinkingtobefocusedandallocationoftaskstobedetermined.Methodstobeusedinthesystematic
review process must be determined at the outset. The Cochrane Reviewers Handbook1 states that
thereviewersknowledgeoftheresultsofthestudymayinfluence:
Thedefinitionofthesystematicreview
Thecriteriaforstudyselection
Thecomparisonsforanalyses
Theoutcomestobereportedinthereview.
Furthermore,spendingtimeatthisstagepreparingaclearprotocolwillreducetimespentduringthe
systematicreviewprocess.
Informationtoincludeintheprotocol
Examples of protocols (of Cochrane systematic reviews) can be found in The Cochrane Library
(http://www.thecochranelibrary.com).
1) Background
This section should address the importance of conducting the systematic review. This may include
discussionoftheimportanceorprevalenceoftheprobleminthepopulationandtheresultsofany
similarreviewsconductedonthetopic.
Thebackgroundshouldalsodescribewhy,theoretically,theinterventionsunderreviewmighthave
animpactonpotentialrecipients.
Reviewersmayrefertoabodyof:
empirical evidence such as similar interventions having an impact, or identical interventions
havinganimpactonotherpopulations.
theoreticalliteraturethatjustifiesthepossibilityofeffectiveness.
Ifreviewerschoosetoexaminemoreproximaloutcomes(knowledgeandattitudes),theoryshouldbe
usedtoexplaintherelationshiptomoredistaloutcomes(changesinbehaviour).
2) Objectives
Reviewerswillneedtodeterminethescopeofthereview.Thescopeofareviewreferstothetypeof
questionbeingaskedandwillaffectthekindofstudiesthatneedtobereviewed,intermsofstudy
topic,populationandsetting,and,ofcourse,studydesign.2
Thescopeofthereviewshouldbebasedonhowtheresultsofthereviewwillbeused.Itisusefulto
consult with the potential users of the review when determining the reviews scope. For example,
manyhealthpromotionpractitionersandpolicymakerswouldfinditmoreusefultohavesystematic
13
reviews of approaches to health promotion (eg. community development or peerdelivered

interventions),ratherthantopicfocusedreviews(eg.healthyeatingoraccidentprevention).
Thescopeisalsolikelytodependonhowmuchtimeisavailableandthelikelyvolumeofresearch
literature.
Lumping the review question, i.e. addressing a wide range of interventions (eg. prevention of
injuriesinchildren):
likelytobetimeconsumingbecauseofthesearchingandselectingprocesses
will better inform decisions about which interventions to implement when there may be a
rangeofoptions
maybeultimatelyofmoreusetopolicydecisions
Splittingthereview,i.e.addressinganarrowrangeofinterventions,(eg.preventionofdrowning
intoddlers)
maybelesstimeconsuming
will only inform decisions about whether or not to implement narrowly focused
interventions
maybemoreusefulforpractitioners
3) Predeterminedselectioncriteria
TheselectioncriteriawillbedeterminedbythePICO(T)question,whichisdescribedinthefollowing
unit(UnitFive.AskinganAnswerableQuestion).Itisimportanttotakeaninternationalperspective
donotrestricttheinclusioncriteriabynationalityorlanguage,ifpossible.1
4) Plannedsearchstrategy
List the databases that are to be searched and if possible, document the search strategy including
subjectheadingsandtextwords.Methodstoidentifyunpublishedliteratureshouldalsobedescribed
(eg.handsearching,contactwithauthors,scanningreferencelists,internetsearching).
5) Planneddataextraction
Reviewers should describe whether they are going to extract process, outcome and contextual data
and state how many reviewers will be involved in the extraction process. The quality assessment
checkliststobeusedforappraisingtheindividualstudiesshouldalsobespecifiedatthisstage.
6) Proposedmethodofsynthesisoffindings
Describethemethodstobeusedtosynthesisethedata.Forexample,reviewersofhealthpromotion
andpublichealthinterventionsoftentabulatetheincludedstudiesandperformanarrativesynthesis
due to expected heterogeneity. It is worthwhile at this stage to consider the likely reasons for
heterogeneityinthesystematicreview.
EstablishanAdvisoryGroup
Systematicreviewsaremorelikelytoberelevantandofhigherqualityiftheyareinformedbyadvice
from peoplewith a rangeof experiences,in terms of both the topicand the methodology.2 Gaining
significantinput from thepotentialusers of the review will helpbringabouta review that is more
meaningful,generalisableandpotentiallymoreaccessible.
Preferably, advisory groups should include persons with methodological and subject/topic area
expertiseinadditiontopotentialreviewusers.
14
Establish an Advisory Group whose members are familiar with the topic and include policy,
funders, practitioners and potential recipients/consumers perspectives. Also include
methodologiststoassistinmethodologicalquestions.
Thebroaderthereview,thebroadertheexperiencerequiredofAdvisoryGroupmembers.
Toensureinternationalrelevanceconsulthealthprofessionalsindevelopingcountriestoidentify
prioritytopics/outcomes/interventionsonwhichreviewsshouldbeconducted.
TheEffectivePublicHealthPracticeProjecthasfoundthatsixmembersonanAdvisoryGroup
cancoverallareasandismanageable.
Develop Terms ofReference for theAdvisory Group to ensure there is clarity about the task(s)
required.Tasksmayinclude:
y making and refining decisions about the interventions of interest, the populations to be
included,prioritiesforoutcomesand,possibly,subgroupanalyses
y providing or suggesting important background material that elucidates the issues from
differentperspectives
y helpingtointerpretthefindingsofthereview
y designingadisseminationplanandassistingwithdisseminationtorelevantgroups
Develop job descriptions and person specifications for consumers and other advisors to clarify
expectations.Furtherinformation,includinghowtoinvolvevulnerableandmarginalisedpeople
inresearch,isalsoavailableatwww.invo.org.uk.
AnexampleofthebenefitsofusinganAdvisoryGroupintheplanningprocess
A review of HIV prevention for men who have sex with men (MSM)
(http://eppi.ioe.ac.uk/EPPIWebContent/hp/reports/MSM/MSMprotocol.pdf)
employed
explicit
consensus methods to shape the review with the help of practitioners, commissioners and
researchers.
An Advisory Group was convened of people from research/academic, policy and service
organisations and representatives from charities and organisations that have emerged from and
speakonbehalfofpeoplelivingwith,oraffectedby,HIV/AIDS.Thegroupmetthreetimesoverthe
courseofthereview.
The group was presented with background information about the proposed review; its scope,
conceptual basis, aims, research questions, stages, methods. Discussion focused on the policy
relevance and political background/context to the review; the inclusion criteria for literature
(interventions,outcomes,subgroupsofMSM);disseminationstrategies;andtimescales.Tworounds
ofvotingidentifiedandprioritisedoutcomesforanalysis.Opendiscussionidentifiedsubgroupsof
vulnerable MSM. A framework for characterising interventions of interest was refined through
AdvisoryGroupdiscussions.
The review followed this guidance by adopting the identified interventions, populations and
outcomestorefinetheinclusioncriteria,performingametaanalysisaswellassubgroupanalyses.
Thesubsequentproductincludedsynthesisedevidencedirectlyrelatedtohealthinequalities.
REFERENCES
15
1. Clarke M, Oxman AD, editors. Cochrane Reviewers Handbook 4.2.0 [updated March 2003].
2. EvidenceforPolicyandPracticeInformationandCoordinatingCentreReviewGroupManual.
Version1.1,SocialScienceResearchUnit,InstituteofEducation,UniversityofLondon,2001.
ADDITIONALREADING
SilagyCA,MiddletonP,HopewellS.PublishingProtocolsofSystematicReviews:ComparingWhat
WasDonetoWhatWasPlanned.JAMA2002;287(21):28312834.
HanleyB,BradburnJ,GorinS,etal.InvolvingConsumersinResearchandDevelopmentintheNHS:
briefingnotesforresearchers.Winchester:HelpforHealthTrust,2000.
16
Unit Five: Asking an Answerable Question

LearningObjectives
Tounderstandtheimportanceofformulatingananswerablequestion
Tobeabletoformulateananswerablequestion
Reviewersshouldseektoanswertwoquestionswithintheirreview:
1.Doestheinterventionwork(notwork)?
2.Howdoestheinterventionwork?
Importanceofgettingthequestionright
Aclearlyframedquestionwillguide:
thereader
o intheirinitialassessmentofrelevance
therevieweronhowto
o collectstudies
o checkwhetherstudiesareeligible
o conducttheanalysis.
Therefore, it is important that the question is formulated before beginning the review. Posthoc
questions are also more susceptible to bias than those questions determined a priori. Although
changes to the review question may be required, the reasons for making the changes should be
clearlydocumentedinthecompletedreview.
Componentsofananswerablequestion(PICO)
The formula to creating an answerable question is following PICO; Population, Intervention,
Comparison,Outcome.Itisalsoworthwhileatthisstagetodeterminethetypesofstudydesignsto
includeinthereview;PICOT.
Qualitative research can contribute to framing the review question (eg. selecting interventions and
outcomesofinteresttoparticipants).TheAdvisoryGroupcanalsoprovidevaluableassistancewith
thistask.
Population(s)
In health promotion and public health this may include populations, communities or individuals.
Considerwhetherthereisvalueinlimitingthepopulation(eg.streetyouth,problemdrinkers).These
groups are often understudied and may be different in all sorts of important respects from study
populationsusuallyincludedinhealthpromotionandpublichealthreviews.
Reviewsmayalsobelimitedtotheeffectsoftheinterventionsondisadvantagedpopulationsinorder
toinvestigatetheeffectoftheinterventionsonreducinginequalities.Furtherinformationonreviews
addressinginequalitiesisprovidedbelow.
17
Intervention(s)
As described earlier, reviewers may choose to lump similar interventions in a review, or split the
review by addressing a specific intervention. Reviewers may also consider approaches to health
promotion rather than topicdriven interventions, for example, peerled strategies for changing
behaviour.Inaddition,reviewersmaywanttolimitthereviewbyfocusingontheeffectivenessofa
particulartypeoftheorybasedintervention(eg.Transtheoreticalmodel)forachievingcertainhealth
outcomes(eg.smokingcessation).
Comparison(s)
Itisimportanttospecifythecomparisoninterventionforthereview.Comparisoninterventionsmay
be no intervention, another intervention or standard care/practice. The choice of comparison or
controlhaslargeimplicationsfortheinterpretationofresults.Aquestionaddressingoneintervention
versusnointerventionisadifferentquestionthanonecomparingoneinterventionversusstandard
care/practice.
Example:DiCensoA,GuyattG,WillanA,GriffithL.Interventionstoreduceunintendedpregnancies
amongadolescents:systematicreviewofrandomisedcontrolledtrials.BMJ2002;324:142634.
The majority of the studies included in this review address primary prevention of unintended
pregnancy versus standard care/practice. Therefore, this review is not addressing whether primary
prevention is effective, it is simply investigating the effect of specific interventions compared to
standardpractice.Thisisamuchsmallergaptoinvestigateaneffect,asitisusuallyeasiertofinda
differencewhencomparingoneinterventionversusnointervention.
Intervention
Effect
Effect
Standardpractice
Effect
Nointervention
Effect
Figure Two. The difference between comparing the effect of one intervention versus no
interventionandoneinterventionversusstandardpractice.
For example, many of the schoolbased interventions in the review are compared to normal sexual
educationintheschools,andareshowntobeineffectiveforreducingunintendedpregnancies.Yet
the interpretation of the results read primary prevention strategies do not delay the initiation of
sexual intercourse or improve the use of birth control among young men and women. This reads
that the review question has sought to address primary prevention versus no intervention. Rather,
the review addressed whether theoryled interventions are more effective than standard
care/practice.
Outcome(s)
Theoutcome(s)chosenforthereviewmustbemeaningfultotheusersofthereview.Thediscrepancy
between the outcomes and interventions that reviewers choose to include in the review and the
outcomesandinterventionsthatlaypeopleprefertobeincludedhasbeenwelldescribed.1
18

To investigate both the implementation of the intervention and its effects reviewers will need to
includebothprocessindicatorsaswellasoutcomemeasures.Unanticipated(sideeffects)aswellas
anticipatedeffectsshouldbeinvestigatedinadditiontocosteffectiveness,whereappropriate.
Reviewerswillalsoneedtodecideifproximal/immediate,intermediateordistaloutcomesaretobe
assessed. If only intermediate outcomes are measured (eg. blood sugar levels in persons with
diabetes,changeinknowledgeandattitudes)reviewersneedtodeterminehowstrongthelinkageis
tomoredistaloutcomes(eg.cardiovasculardisease,behaviourchange).Theuseoftheorycanassist
with determining this relationship. In addition, reviewers should decide if only objective measures
are to be included (eg. one objective measure of smoking status is saliva thiocyanate or alveolar
carbonmonoxide)orsubjectivemeasures(eg.selfreportedsmokingstatus),oracombinationofboth
(discussingtheimplicationsofthisdecision).
Examplesofreviewquestions
Poorlydesignedquestions:
1. ArecondomseffectiveinpreventingHIV?
2. WhichinterventionsreducehealthinequalitiesamongpeoplewithHIV?
Answerablequestions:
1. Inmenwhohavesexwithmen,doescondomusereducetheriskofHIVtransmission?
2. InwomenwithHIV,dopeerbasedinterventionsreducehealthinequalities?
Aremassmediainterventionseffectiveinpreventingsmokinginyoungpeople?
Problem,
Intervention
Comparison
Outcome
Typesof
population
studies
1. RCT(and
Young
1. Television
No
1. objective
quasiRCT)
people,
2. Radio
intervention
measuresof
2. Controlled
under25
3. Newspapers
smoking
yearsofage
4. Billboards
2. selfreported
beforeand
5. Posters
smoking
after
6. Leaflets
behaviour
studies
7. Booklets
3. Intermediate
3. Timeseries
measures
designs
(intentions,
attitudes,
knowledge)
4. Process
measures(eg.
mediareach)
Typesofstudydesignstoinclude
Thedecisionsaboutwhichtype(s)ofstudydesigntoincludewillinfluencesubsequentphasesofthe
review,particularlythesearchstrategies,choiceofqualityassessmentcriteria,andtheanalysisstage
(especiallyifastatisticalmetaanalysisistobeperformed).
The decision regarding which study designs to include in the review should be dictated by the
intervention (the review question) or methodological appropriateness, and not vice versa.2,3 If the
reviewquestionhasbeenclearlyformulatedthenknowledgeofthetypesofstudydesignsneededto
19
answeritshouldautomaticallyfollow.3Ifdifferenttypesofstudydesignsaretoincludedinthesame
reviewthereasonsforthisshouldbemadeexplicit.
Effectivenessstudies
WhereRCTsarelacking,orforissuesrelatingtofeasibilityandethicsarenotconducted,otherstudy
designs such as nonrandomised controlled trials, before and after studies, and interrupted time
seriesdesignsshouldalsobeconsideredforinclusioninthereview.
Comparisonswithhistoricalcontrolsornationaltrendsmaybeincludedwhenthisistheonlytypeof
evidence that is available, for example, in reviews investigating the effectiveness of policies, and
shouldbeaccompaniedbyanacknowledgementthattheevidenceofevidenceisnecessarilyweaker.
Randomisedcontrolledtrial
Subjects are randomly allocated to groups either for the intervention being studied or the control
(using a random mechanism, such as coin toss, random number table, or computergenerated
randomnumbers)andtheoutcomesarecompared.1
Each participant or grouphas the same chance of receiving eachinterventionand the investigators
cannotpredictwhichinterventionisnext.
Quasirandomisedcontrolledtrial/pseudorandomisedcontrolledtrial
Subjects are allocated to groups for intervention or control using a nonrandom method (such as
alternateallocation,allocationofdaysoftheweek,oroddevenstudynumbers)andtheoutcomesare
compared.1
Controlledbeforeandafterstudy/cohortanalytic
Outcomes are compared for a group receiving the intervention being studied, concurrently with
controlsubjectsreceivingthecomparisonintervention(eg,usualornocare/intervention).1
Uncontrolledbeforeandafterstudy/cohortstudy
The same group is pretested, givenan intervention, and tested immediately after the intervention.
Theinterventiongroup,bymeansofthepretest,actastheirowncontrolgroup.2
Interruptedtimeseries
Atimeseriesconsistsofmultipleobservationsovertime.Observationscanbeonthesameunits(eg.
individualsovertime)orondifferentbutsimilarunits(eg.studentachievementscoresforparticular
gradeandschool).Interruptedtimeseriesanalysisrequiresknowingthespecificpointintheseries
when an intervention occurred.2 These designs are commonly used to evaluate mass media
campaigns.
Qualitativeresearch
Qualitativeresearchexploresthesubjectiveworld.Itattemptstounderstandwhypeoplebehavethe
way they do and what meaning experiences have for people. Qualitative research relevant to
effectivenessreviewsmayincludethefollowing:
Qualitativestudiesofexperience:thesestudiesmayusearangeofmethods,butfrequentlyrelyonin
depthtaperecordedinterviewsandnonparticipantobservationalstudiestoexploretheexperience
ofpeoplereceivinganintervention.
20
Processevaluations:thesestudiescanbeincludedwithinthecontextoftheeffectivenessstudies.These
evaluationsuseamixtureofmethodstoidentifyanddescribethefactorsthatpromoteand/orimpede
theimplementationofinnovationinservices.3
References:
1.
2.
3.
NHMRC (2000). How to review the evidence: systematic identification and review of the
scientificliterature.Canberra:NHMRC.
Thomas H. Quality assessment tool for quantitative studies. Effective Public Health Practice
Project.McMasterUniversity,Toronto,Canada.
CarryingOutorCommissioningReviews.CRDReportNumber4(2ndEdition).NHSCentrefor
Reviews
and
Dissemination,
University
of
York.
March
2001.
http://www.york.ac.uk/inst/crd/report4.htm
ClusterRCTsandclusternonrandomisedstudies
Allocation of the intervention by group or cluster is being increasingly adopted within the field of
public health because ofadministrative efficiency, lessened risk of experimental contamination and
likely enhancement of subject compliance.4 Some studies, for example a classbased nutrition
intervention,dictateitsapplicationattheclusterlevel.
Interventionsallocatedatthecluster(eg.school,class,worksite,community,geographicalarea)level
have particular problems with selection bias where groups are formed not at random but rather
throughsomephysical,social,geographic,orotherconnectionamongtheirmembers.5,6 Clustertrials
also require a larger sample size than would be required in similar, individually allocated trials
because the correlation between cluster members reduces the overall power of the study.5 Other
methodologicalproblemswithclusterbasedstudiesincludethelevelofinterventiondifferingfrom
the level of evaluation (analysis) and the often small number of clusters in the study.7 Issues
surrounding cluster trials have been well described in a Health Technology Assessment report7,
which should be read for further information if cluster designs are to be included in a systematic
review.
Theroleofqualitativeresearchwithineffectivenessreviews
- toprovideanindepthunderstandingofpeoplesexperiences,perspectivesandhistoriesin
thecontextoftheirpersonalcircumstancesandsettings8
Qualitativestudiescancontributetoreviewsofeffectivenessinanumberofwaysincluding9:
- Helpingtoframethereviewquestion(eg.selectinginterventionsandoutcomesofinterestto
participants).
- Identifying factors that enable/impede the implementation of the intervention (eg. human
factors,contextualfactors)
- Describingtheexperienceoftheparticipantsreceivingtheintervention
- Providingparticipantssubjectiveevaluationsofoutcomes
- Helpingtounderstandthediversityofeffectsacrossstudies,settingsandgroups
- Providing a means of exploring the fit between subjective needs and evaluated
interventionstoinformthedevelopmentofnewinterventionsorrefinementofexistingones.
Methods commonly used in qualitative studies may include one or a number of the following;
interviews(structuredaroundrespondentspriorities/interests),focusgroups,participantand/ornon
participantobservation,conversation(discourseandnarrativeanalysis),anddocumentaryandvideo
analysis.Theunitofanalysiswithinqualitativestudiesisnotnecessarilyindividualsorsinglecases;
communities, populations or organisations may also be investigated. Anthropological research,
21
whichmayinvolvesomeorallofthesemethodsinthecontextofwiderangingfieldworkcanalso
be a valuable source of evidence, although may be difficult to subject to many aspects of the
systematicreviewprocess.
Healthinequalities
Healthinequalitiesaredefinedasthegapinhealthstatus,andinaccesstohealthservices,between
differentsocialclassesandethnicgroupsandbetweenpopulationsindifferentgeographicalareas.10
There is a need for systematic reviews to consider health inequalities in the assessment of
effectiveness of interventions. This is because it is thought that many interventions may not be
equally effective for all population subgroups. The effectiveness for the disadvantaged may be
substantiallylower.
EvansandBrown(2003)11suggestthatthereareanumberoffactorsthatmaybeusedinclassifying
healthinequalities(capturedbytheacronymPROGRESS):
Placeofresidence
Race/ethnicity
Occupation
Gender
Religion
Education
Socioeconomicstatus
Socialcapital
Therefore, it may be useful for a review of public health interventions to measure the effect across
differentsubgroups(asdefinedbyanyofthePROGRESSfactors).
Exampleofareviewaddressinginequalities:KristjanssonE,RobinsonVA,MacDonaldB,KrasevecJ,
GreenhalghT, McGowan J,Francis D, TugwellP,Petticrew M,Shea B,Wells G. Schoolfeedingfor
improving the physical and psychosocial health of disadvantaged elementary school children
(ProtocolforaCochraneReview).In:TheCochraneLibrary,Issue3,2004.Chichester,UK:JohnWiley
&Sons,Ltd.Disadvantageinthisreviewisdefinedbyincome(SESstatus).
Datarequiredforreviewsaddressinginequalities:
Avalidmeasureofhealthstatus(orchangeinhealthstatus)
Ameasureofdisadvantage(i.e.,definesocioeconomicposition)
Astatisticalmeasureforsummarisingthedifferentialeffectiveness.
The above review chose to define interventions effective in reducing inequalities as interventions
which were more effective for people in lower SES. A potentially effective intervention was one
whichwas:
o equally effective across the socioeconomic spectrum (potentially reducing health
inequalities due to the higher prevalence of health problems among the
disadvantaged).
o targetedonlyatdisadvantagedgroupsandwaseffective.
22
REFERENCES
1.
Oliver S. 1997. Exploring lay perspectives on questions of effectiveness; IN: Maynard A,

ChalmersI(eds).Nonrandomreflectionsonhealthservicesresearch.LondonBMJPublishing
Group.
2.
Nutbeam D, Harris E. (2004). Theory in a Nutshell. A practical guide to health promotion

theories.Sydney,Australia:McGrawHillAustraliaPtyLtd,vii9.
3.
Petticrew M, Roberts H. (2003). Evidence, hierarchies, and typologies: horses for courses. J
EpidemiolCommunityHealth,57,5279.
4.
Donner A, Klar N. Pitfalls of and controversies in cluster randomization trials. Am J Public

Health.2004Mar;94(3):41622.
5.
Torgerson DJ. Contamination in trials: is cluster randomisation the answer? BMJ. 2001 Feb
10;322(7282):3557.
6.
MurrayDM,VarnellSP,BlitsteinJL.Designandanalysisofgrouprandomizedtrials:areview
ofrecentmethodologicaldevelopments.AmJPublicHealth.2004Mar;94(3):42332.
7.
UkoumunneOC,GullifordMC,ChinnS,SterneJA,BurneyPG.Methodsforevaluatingarea
wide and organisationbased interventions in health and health care: a systematic review.
HealthTechnolAssess.1999;3(5):iii92.
8.
Spencer L, Ritchie J, Lewis J, Dillon L. Quality in Qualitative Evaluation: A framework for

assessing research evidence. Government Chief Social Researchers Office. Crown Copyright,
2003.
9.
Centre for Reviews and Dissemination (Undertaking Systematic Reviews of Research on

Effectiveness.CRDsGuidanceforthoseCarryingOutorCommissioningReviews.CRDReport
Number4(2ndEdition)March2001),athttp://www.york.ac.uk/inst/crd/report4.htm
10.
PublicHealthElectronicLibrary.
http://www.phel.gov.uk/glossary/glossaryAZ.asp?getletter=H.AccessedJune29,2004
11.
EvansT,BrownH.Roadtrafficcrashes:operationalizingequityinthecontextofhealthsector
reform.InjuryControlandSafetyPromotion2003;10(2):1112.
ADDITIONALREADING
Richardson WS, Wilson MC, Nishikawa J, Hayward RSA. The wellbuilt clinical question: a key to
evidencebaseddecisions[Editorial].ACPJClub1995;123(3):A123.
RichardsonWS.Ask,andyeshallretrieve[EBMNote].EvidenceBasedMedicine1998;3:1001.
23
EXERCISE
1. Writeananswerablereviewquestion(willbeusedinlaterexercises)
P=
I=....
C=.
O=.
Q..
Theeffectivenessof(I)versus(C)for(0)in(P)
2. What type(s) of study design(s) should be included to investigate the effectiveness of the
intervention?
Randomisedcontrolledtrial/clusterrandomisedcontrolledtrial
Quasirandomisedcontrolledtrial/pseudorandomisedtrial
Controlledbeforeandafterstudy/cohortanalytic/concurrentlycontrolledcomparativestudy
Uncontrolledbeforeandafterstudy/cohortstudy
Interruptedtimeseriesdesigns
Qualitativeresearch
24
Unit Six: Finding The Evidence

LearningObjectives
Tounderstandthecomplexitiesofsearchingforhealthpromotionandpublichealthstudies
Togainknowledgeofhowtolocateprimarystudiesofhealthpromotionandpublichealthinterventions
Togainbasicskillstocarryoutasearchforprimarystudies
Identifyinghealthpromotionandpublichealthprimarystudies
The inclusion of an unbiased sample of relevant studies is central to the validity of systematic
reviews. Timeconsuming and costly literature searches, which cover the grey literature and all
relevantlanguagesanddatabases,arenormallyrecommendedtopreventreportingbiases.1
Searching for primary studies on health promotion and public health topics can be a very time
intensive task, as search strategies will need to be adapted for a number of databases, and broad
searchesusingawiderangeoftermsmayresultinalargenumberofcitationsrequiringapplication
of the inclusion and exclusion criteria. This is party due to health promotion and public health
terminology being very nonspecific or nonstandardised; day to day words are often used to
describeinterventionsandpopulations.Inaddition,itmaynotbeappropriatetoaddarandomised
controlledtrial(RCT)filtertolimitthesearchbecausethequestionmaybebestansweredusingother
typesofstudydesigns.
Componentsofthesearchingprocess
Thekeycomponentsofthesearchstrategycompriseofsubjectheadingsandtextwordsthatdescribe
eachelementofthePICO(T)question.
However, it is usually recommended not to include the O (outcome) of the PICO question in the
searchstrategybecauseoutcomesaredescribedinmanydifferentwaysandmaynotbedescribedin
the abstract of the article. Search terms to describe outcomes should only be used if the number of
citationsistoolargetoapplytheinclusionandexclusioncriteria.
Pilotthesearchstrategyfirstcompleteascopingsearchonadatabasemostlikelytoyieldstudies
usingasampleofkeywordstolocateafewrelevantstudies.Checkthesubjectheadingsthatareused
toindexthestudiesandtherelevanttextwordsintheabstractofthecitation.Also,itmaybeusefulto
findthecitationsofkeyarticlesinPubMed(http://www.ncbi.nlm.nih.gov/entrez/query.fcgi)andclick
on Related Articles to find other relevant studies in order to determine additional relevant subject
headingsandtextwords.
Thesearchstrategydevelopedtoidentifystudieswillnotsearchtheentirefulltextofthearticle.The
followingcompletereferenceforthecitationdemonstratestheinformationthatisavailableforeach
citation(exampleprovidedusingtheOVIDinterface):thereforesearchingthesubjectheadingsand
textwords in the abstract will help us to find this study. Always use a combination of subject
headingsandtextwordsforeachPICOelement.
25
UniqueIdentifier
2014859
RecordOwner
NLM
Authors
BaumanKE.LaPrelleJ.BrownJD.KochGG.PadgettCA.
Institution
Department of Health Behavior and Health Education, School of Public Health, University of North Carolina,
ChapelHill275997400.
Title
Theinfluenceofthreemassmediacampaignsonvariablesrelatedtoadolescentcigarettesmoking:resultsofa
fieldexperiment.
Source
AmericanJournalofPublicHealth.81(5):597604,1991May.
AbbreviatedSource
AmJPublicHealth.81(5):597604,1991May.
PublicationNotes
Thepublicationyearisfortheprintissueofthisjournal.
NLMJournalCode
1254074,3xw
JournalSubset
AIM,IM
LocalMessages
HeldatRCH:1985onwards,Someyearsonlinefulltextlinkfromlibraryjournallist
CountryofPublication
UnitedStates
MeSHSubjectHeadings
Adolescent
*AdolescentBehavior
Child
*HealthEducation/mt[Methods]
Subjectheadings
Human
*MassMedia
Pamphlets
PeerGroup
Radio
RegressionAnalysis
*Smoking/pc[Prevention&Control]
Textwordsinabstract,eg.
SoutheasternUnitedStates
television,adolescent,
Support,U.S.Govt,P.H.S.
massmedia,smoking
Television
Abstract
BACKGROUND: This paper reports findings from a field experiment that evaluated mass media campaigns
designedtopreventcigarettesmokingbyadolescents.METHODS:Thecampaignsfeaturedradioandtelevision
messagesonexpectedconsequencesofsmokingandacomponenttostimulatepersonalencouragementofpeers
nottosmoke.SixStandardMetropolitanStatisticalAreasintheSoutheastUnitedStatesreceivedcampaignsand
fourservedascontrols.Adolescentsandmothersprovidedpretestandposttestdataintheirhomes.RESULTS
ANDCONCLUSIONS:Theradiocampaignhadamodestinfluenceontheexpectedconsequencesofsmoking
and friend approval of smoking, the more expensive campaigns involving television were not more effective
than those with radio alone, the peerinvolvement component was not effective, and any potential smoking
effectscouldnotbedetected.
ISSN
00900036
PublicationType
JournalArticle.
GrantNumber
26
CA38392(NCI)
Language
English
EntryDate
19910516
RevisionDate
20021101
UpdateDate
20031209
Subjectheadings/descriptors(eg.MESHheadingsinMedline)
Subjectheadingsareusedindifferentdatabasestodescribethesubjectofeachjournalarticleindexed
in the database. For example, MeSH (Medical Subject Headings) are used within the Medline
database; there are more than 22,000 terms used to describe studies and the headings are updated
annuallytoreflectchangesinmedicineandmedicalterminology.
Examplesofsubjectheadingsrelevanttohealthpromotionandpublichealth:
Massmedia,smoking,adolescent,healthpromotion,healtheducation,students,sports
Remember,eachdatabasewillhavedifferentcontrolledvocabulary(subjectheadings).Also,subject
headings are assigned by human beings, so mistakes can be made. For example, the mass media
articlewasnotassignedwiththemassmediasubjectheadinginthePyscINFOdatabase.Therefore,
searchstrategiesshouldalwaysincludetextwordsinadditiontosubjectheadings.
Formanyhealthpromotiontopicstheremaybefewsubjectheadingsavailable(eg.communitybased
interventions).Therefore,thesearchstrategymaycomprisemainlyoftextwords.
Textwords
Thesearewordsthatareusedintheabstractofarticles(andtitle)toassistwithfindingtherelevant
literature. Textwords in a search strategy always end in .tw, eg. adolescent.tw will find the word
adolescentintheabstractandtitleofthearticle.Ageneralruleistoduplicateallsubjectheadingsas
textwords,andaddanyotherwordssuchmayalsodescribethecomponentofPICO.
Truncation$thispicksupvariousformsofatextword.
Eg.teen$willpickupteenage,teenagers,teens,teen
Eg.Smok$willpickupsmoke,smoking,smokes,smoker,smokers
Wildcards?and#
Thesesyntaxcommandswillpickupdifferentspellings.
?willsubstituteforoneornocharacters,soisusefulforlocatingUSandEnglishspellings
Eg.colo?r.twwillpickupcolorandcolour
#willsubstituteforonecharactersoisusefulforpickinguppluralorsingularversionsofwords
Eg.wom#nwillpickupwomenandwoman
AdjacentADJn
This command retrieves two or more query terms within n words of each other, and in any order.
Thissyntaxisimportantwhenthecorrectphraseologyisunknown.
Eg.sportADJ1policywillpickupsportpolicyandpolicyforsport
Eg.mentalADJ2healthwillpickupmentalhealthandmentalandphysicalhealth
27
Note: Databases may use different syntax to retrieve records (eg. $ or * may be used in different
databases or interfaces). Therefore, reviewers will need to be become wellacquainted with the
idiosyncrasies of each database. Due to the different subject headings used between databases,
reviewers will also need to adapt their search strategy for each database (only adapt the subject
headings,nottextwords).
CombiningeachelementofthePICOquestions
Elementofquestion
PPopulation
Subjectheadings
IIntervention
Subjectheadings
CComparison(ifnecessary)
Subjectheadings
OOutcome
Subjectheadings
OR
OR
OR
OR
TTypeofstudy(ifnecessary)
Subjectheadings
Useavalidatedfilter
OR
Textwords
Textwords
Textwords
Textwords
Textwords
TofindstudiesusingallofthePICOelements
PANDIANDCANDO(ANDT)
A lumped review (review of a number of different interventions) is simply a review comprising a

number of different PICO(T) questions. This is exemplified in the following pages outlining the
searchstrategytolocateInterventionsforpreventingobesityinchildren.
Usingstudydesigntolimitsearch
RCTs:IfthereviewislimitedtoevidencefromRCTsastudydesignfiltercanbeaddedtothesearch
strategy.TheCochraneReviewersHandbook2detailstheappropriatefiltertoadd.
NonRCTs: Limiting the search strategy by using nonrandomised study terms can be very
problematic,andisgenerallynotrecommended.Thisisbecause:
Fewstudiesmaybeindexedbystudydesign
The vocabulary required to identify different study designs can vary extensively between
electronic databases. Terms vary from control groups to followup studies, to longitudinal
studiesorevenprogrameffectivenessorprogramevaluation,toindexthesamestudies
Somedatabases,eg.PsycINFO,arepoorlyindexedwithrespecttomethodology.
Therefore,afteraPICOsearchiscompletedallcitationswillrequireapplicationoftheinclusionand
exclusioncriteria.
Qualitativeresearch:AfilterfortheCINAHLdatabaseisavailablefromtheEdwardMinerLibrary
http://www.urmc.rochester.edu/hslt/miner/digital_library/tip_sheets/Cinahl_eb_filters.pdf
28
Wheretolocatestudies
a)Electronicdatabasesofrelevancetohealthpromotionandpublichealth
Reviewersshouldensurethatthesearchstrategy(subjectheadingsandtextwords)isdevelopedfora
numberofdatabasesthatcoverthevarietyofdomainswheretheliteraturemaybelocated.
A full list of free public health databases and subscriptiononly databases is available at
http://library.umassmed.edu/ebpph/dblist.cfm. This website contains a number of databases that
havenotbeenincludedinthefollowinglist.
Some examples of electronic databases that may be useful to identify public health or health
promotionstudiesinclude(websiteslistedfordatabasesavailablefreelyviatheinternet):
Psychology:
PsycINFO/PscyLIT
Biomedical:
CINAHL, LILACS (Latin American Caribbean Health Sciences Literature)
http://www.bireme.br/bvs/I/ibd.htm, Web of Science, Medline, EMBASE,
CENTRAL
(http://www.updatesoftware.com/clibng/cliblogon.htm),
Combined Health Information Database (CHID) http://chid.nih.gov/,
ChronicDiseasePreventionDatabase(CDP)http://www.cdc.gov/cdp/
Sociology:
Sociofile,SociologicalAbstracts,SocialScienceCitationIndex
Education:
ERIC (Educational Resources Information Center), C2SPECTR (Campbell
Collaboration Social, Psychological, Educational and Criminological Trials
Register) http://www.campbellcollaboration.org, REEL (Research Evidence
inEducationLibrary,EPPICentre)http://eppi.ioe.ac.uk
Transport:
NTIS (National Technical Information Service), TRIS (Transport Research
Information Service) http://ntl.bts.gov/tris, IRRD (International Road
ResearchDocumentation),TRANSDOC(fromECMT(EuropeanConference
ofMinistersofTransport)
Physicalactivity:
SportsDiscus
HP/PH:
BiblioMap (EPPICentre) http://eppi.ioe.ac.uk, HealthPromis (HDA, UK)
http://www.hdaonline.org.uk/evidence/,GlobalHealth
Other:
Popline
(population
health,
family
planning)
http://db.jhuccp.org/popinform/basic.html,
Enviroline
(environmental
health) available on Dialog, Toxfile (toxicology) available on Dialog,
Econlit(economics)
Qualitative:
ESRC Qualitative Data Archival Resource Centre (QUALIDATA)
(http://www.qualidata.essex.ac.uk), Database of Interviews on Patient
Experience(DIPEX)(http://www.dipex.org).
29
b)Handsearchinghealthpromotionandpublichealthjournals
Itmaybeusefultohandsearchspecialistjournalsrelevanttothereviewtopicareatoidentifyfurther
primaryresearchstudies.Alsoconsidernonhealthpromotionandpublichealthjournalswhichmay
coverthetopicofinterest,i.e.,marketingjournals,etc.
Two lists of health promotion and public health journals have been produced which may help to
determinewhichjournalstosearch.
1)TheLamarSoutterLibrarylistofpublichealthjournals,http://library.umassmed.edu/ebpph/,
(alistoffreelyavailablejournalsisalsoincluded)
2)TheCorePublicHealthJournalsListcompiledbyYaleUniversity,
http://www.med.yale.edu/eph/library/phjournals/,
TheEffectivePublicHealthPracticeProject(Canada)hasfoundthatthemostproductivejournalsto
handsearch to locate public health and health promotion articles are: American Journal of Health
Promotion,AmericanJournalofPreventiveMedicine,AmericanJournalofPublicHealth,Canadian
JournalofPublicHealth,BMJ.OtherusefuljournalsincludeAnnualReviewofPublicHealth,Health
EducationandBehavior(formerlyHealthEducationQuarterly),HealthEducationResearch,JAMA,
PreventiveMedicine,PublicHealthReports,SocialScienceandMedicine.
c)Greyliterature
Methodstolocateunpublished,difficulttofindliteratureinclude:
Scanningreferencelistsofrelevantstudies
Contactingauthors/academicinstitutionsofkeystudies
Searchingfortheses,dissertations,conferenceproceedings(onesourceofdissertationsandtheses
is the Networked Digital Library of Theses and Dissertations (NDLTD) which can be accessed
fromhttp://www.theses.org/)
Searching the internet for national public health reports, local public health reports, reviews
serving as background documentation for legislation, quality assurance reports, etc. A useful
internetsearchengineforlocatingacademicworkisGoogleScholar(http://scholar.google.com).
Save,documentandexportthesearch
Always save and print out the search strategy for safe recordkeeping. It is essential to have
bibliographic software (Endnote, Reference Manager, GetARef) to export the retrieved citations to
apply the inclusion/exclusion criteria. Citations from unpublished literature cannot usually be
exported,sowillrequireindividualentrybyhandintothereferencemanagingsystem.Bibliographic
softwarewillalsoassistwiththereferencingwhenwritingthefinalreview.
REFERENCES
1. EggerM,JuniP,BartlettC,HolensteinF,SterneJ.Howimportantarecomprehensiveliterature
searches and the assessment of trial quality in systematic reviews? Empirical study. Health
TechnolAssess2003;7(1).
http://www.cochrane.dk/cochrane/handbook/handbook.htm
30
ADDITIONALREADING
HardenA,PeersmanG,OliverS,OakleyA.Identifyingprimaryresearchonelectronicdatabasesto
informdecisionmakinginhealthpromotion:thecaseofsexualhealthpromotion.HealthEducation
Journal1999;58:290301.
EXERCISE
1.Gothroughtheworkedexamplesearchingexercise.
2.GobacktoPICOquestiondevelopedinUnitFive.
(a)findMedicalSubjectHeadings(MeSH)/descriptorsandtextwordsthatwouldhelpdescribeeach
ofthePICOcomponentsofthereviewquestion.
MeSH/descriptors
Textwords
eg.Adolescent(Medline)
student,highschool,teenage
egHighSchoolStudents(PsycINFO)
P=
I=
C=Maynotberequired
O=
(b) Whichdatabases would be mostuseful to locate studies on this topic? Do the descriptors differ
betweenthedatabases?
31
Examplesofsearchingstrategies
CampbellK,WatersE,OMearaS,KellyS,SummerbellC.Interventionsforpreventingobesityin
children(CochraneReview).In:TheCochraneLibrary,Issue3,2004.Chichester,UK:JohnWiley
&Sons,Ltd.
MEDLINE,1997
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
explodeObesity/allsubheadings
WeightGain/allsubheadings
WeightLoss/allsubheadings
obesityorobese
weightgainorweightloss
overweightoroverweightorovereat*orovereat*
weightchange*
(bmiorbodymassindex)near2(gainorlossorchange)
#1or#2or#3or#4or#5or#6or#7or#8
ChildinMIME,MJME
Adolescence/allsubheadings
ChildPreschool/allsubheadings
InfantinMIME,MJME
child*oradolescen*orinfant*
teenage*oryoungpeopleoryoungpersonoryoungadult*
schoolchildrenorschoolchildren
p?ediatr*inti,ab
boysorgirlsoryouthoryouths
#10or#11or#12or#13or#14or#15or#16or#17or#18
explodeBehaviorTherapy/allsubheadings
SocialSupportinMIME,MJME
FamilyTherapy/allsubheadings
explodePsychotherapyGroup/allsubheadings
(psychologicalorbehavio?r*)adj(therapyormodif*orstrateg*orintervention*)
grouptherapyorfamilytherapyorcognitivetherapy
(lifestyleorlifestyle)adj(chang*orintervention*)
counsel?ing
socialsupport
peernear2support
(childrennear3parent?)neartherapy
#20or#21or#22or#23or#24or#25or#26or#27or#28or#29or#30
explodeObesity/drugtherapy
explodeAntiObesityAgents/allsubheadings
lipaseinhibitor*
orlistatorxenicalortetrahydrolipstatin
appetiteadj(suppressant*ordepressant*)
sibutramineor(meridiainti,ab)
dexfenfluramineorfenfluramineorphentermine
bulkingagent*
methylcelluloseorcelevac
(antiobesityorantiobesity)adj(drug*oragent*)
guargum
explodeObesity/diettherapy
32
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.
80.
81.
82.
83.
84.
85.
86.
87.
88.
89.
90.
91.
92.
93.
94.
95.
DietFatRestricted/allsubheadings
DietReducing/allsubheadings
DietTherapy/allsubheadings
Fasting/allsubheadings
dietordietsordieting
diet*adj(modif*ortherapyorintervention*orstrateg*)
lowcalorieorcaloriecontrol*orhealthyeating
fastingormodifiedfast*
explodeDietaryFats/allsubheadings
fruitorvegetable*
highfat*orlowfat*orfattyfood*
formuladiet*
#44or#45or#46or#47or#48or#49or#50or#51or#52or#53or#54or#55or#56
Exercise/allsubheadings
ExerciseTherapy/allsubheadings
exercis*
aerobicsorphysicaltherapyorphysicalactivityorphysicalinactivity
fitnessadj(class*orregime*orprogram*)
aerobicsorphysicaltherapyorphysicaltrainingorphysicaleducation
dancetherapy
sedentarybehavio?rreduction
#58or#59or#60or#61or#62or#63or#64or#65
explodeObesity/surgery
SurgicalStaplers/allsubheadings
SurgicalStapling/allsubheadings
Lipectomy/allsubheadings
GastricBypass/allsubheadings
Gastroplasty/allsubheadings
dentalsplintingorjawwiring
gastroplastyorgastricband*orgastricbypass
intragastricballoon*orverticalband*
stomachadj(stapl*orband*orbypass)
liposuction
explodeAlternativeMedicine/allsubheadings
alternativemedicineorcomplementarytherap*orcomplementarymedicine
hypnotismorhypnosisorhypnotherapy
acupunctureorhomeopathyorhomoeopathy
chinesemedicineorindianmedicineorherbalmedicineorayurvedic
#79or#80or#81or#82or#83
(dietordietingorslim*)adj(club*ororgani?ation*)
weightwatcher*orweightwatcher*
correspondenceadj(course*orprogram*)
fatcamp*ordiet*camp*
#85or#86or#87or#88
HealthPromotion/allsubheadings
HealthEducation/allsubheadings
healthpromotionorhealtheducation
mediaintervention*orcommunityintervention*
healthpromotingschool*
(school*near2program*)or(communitynear2program*)
33
96. familyintervention*orparent*intervention*
97. parent*near2(behavio?rorinvolve*orcontrol*orattitude*oreducat*)
98. #90or#91or#92or#93or#94or#95or#96or#97
99. HealthPolicy/allsubheadings
100. NutritionPolicy/allsubheadings
101. healthpolic*orschoolpolic*orfoodpolic*ornutritionpolic*
102. #99or#100or#101
103. explodeObesity/preventionandcontrol
104. PrimaryPrevention/allsubheadings
105. primarypreventionorsecondaryprevention
106. preventivemeasure*orpreventativemeasure*
107. preventivecareorpreventativecare
108. obesitynear2(prevent*ortreat*)
109. #103or#104or#105or#106or#107or#108
110. explodeControlledClinicalTrials/allsubheadings
111. RandomAllocationinMIME,MJME
112. DoubleBlindMethodinMIME,MJME
113. SingleBlindMethodinMIME,MJME
114. Placebos/allsubheadings
115. explodeResearchDesign/allsubheadings
116. (singl*ordoubl*ortrebl*ortripl*)near5(blind*ormask*)
117. exact{CONTROLLEDCLINICALTRIAL}inPT
118. placebo*
119. matchedcommunitiesormatchedschoolsormatchedpopulations
120. control*near(trial*orstud*orevaluation*orexperiment*)
121. comparisongroup*orcontrolgroup*
122. matchedpairs
123. outcomestudyoroutcomestudies
124. quasiexperimentalorquasiexperimentalorpseudoexperimental
125. nonrandomi?edornonrandomi?edorpseudorandomi?ed
126.
#110or#111or#112or#113or#114or#115or#116or#117or#118or#119or#120or#121or#122or
#123or#124or#125
127. #9and#19
128. #31or#43or#57or#66or#78or#84or#89or#98or#102or#109
129. #126and#127and#128
130. animalintg
131. humanintg
132. #130not(#130and#131)
133. #129not#132
134. #133and(PY>=1997)
34
Brunton G, Harden A, Rees R, Kavanagh J, Oliver S, Oakley A (2003). Children and Physical
Activity: A systematic Review of Barriers and Facilitators. London: EPPICentre, Social Science
ResearchUnit,InstituteofEducation,UniversityofLondon.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
Expchild/
Exp adolescence/ or exp child/ hospitalized/ or exp child institutionalized/ or exp disabled children/ or
infant
1not2
expchildpreschool/
expstudents/
((universityorcollegeormedicalorgraduateorpostgraduate)adj2student$).ti.ab.
5not6
(schooladj3(child$orpupil$orstudent$orkidofkidsofprimaryornurseryorinfant$)).ti.ab.
or/34,78
exphealthpromotion/
exphealtheducation/
exppreventivemedicine/
(prevent$orreduc$orpromot$orincrease$orprogram$orcurricul$oreducat$orproject$orcampaign$
or impact$ or risk$ or vulnerab$ or resilien$ or factor$ or correlate$ or predict$ or determine$ or
behavio?r$).ti.ab.
(health$ or ill or illness or ills or well or wellbeing or wellness or poorly or unwell or sick$ or
disease$).ti.ab.
((prevent$orreduc$orpromot$orincrease$orprogram$orcurricul$ireducat$orproject$orcampaign$
or impact$ or risk$ or vulnerab$ or resilien$ or factor$ or correlate$ or predict$ or determine$ or
behavio?r$) adj3 (health$ or ill or illness or ills or well or wellbeing or wellness or poorly or unwell or
sick$ordisease$).ti.ab.
or/1012,15
(determine$orfacilitate$orbarrier$).ti
Riskfactors/
Culture/
Family/ or Internalexternal control/ or Life style/ or Prejudice/ or Psychology, social/ or Psychosocial
deprivation/
Childbehavior/
Habits/
Poverty/
Socialclass/
Socialconditions/
Socioeconomicfactors/
Familycharacteristics/
Ethnicity.ti,ab.
Attitudetohealth/
Or/1729
Expsports/
Expphysicalfitness/
Expexertion/
Physicaleducationandtraining/
expleisureactivities/
Recreation/
((sedentaryorinactive$)adj3child$).ti,ab.
((physical$orsport$orexercise$orgame$1)adj3(activit$orexercise$orexert$orfitorfitnessorgame$1
orenduranceorendure$oreducat$ortrain$1ortraining)).ti,ab.
Or/3138
Or/16,30
And/9,3940
35
WORKEDEXAMPLE
We will work through the process of finding primary studies for a systematic review, using the
reviewbelowasanexample:**Thissearchhasbeenmodifiedfromtheoriginalversion**
Sowden A, Arblaster L, Stead L. Community interventions for preventing smoking in young

people(CochraneReview).In:TheCochraneLibrary,Issue3,2004.Chichester,UK:JohnWiley&
Sons,Ltd.
1adolescent/
2child/
Allthesubjectheadingsand
3Minors/
textwordsrelatingtoP
4youngpeople.tw.
population
5(child$orjuvenile$orgirl$orboy$orteen$oradolescen$).tw.
6minor$.tw
7or/16
8expsmoking/
9tobacco/
Allthesubjectheadingsand
10tobaccousedisorder/
textwordsrelatingtoO
11(smok$ortobaccoorcigarette$).tw.
outcome
12or/811
13(communityorcommunities).tw.
14(nationwideorstatewideorcountrywideorcitywide).tw.
15(nationadjwide).tw.
16(stateadjwide).tw.
Allthesubjectheadings
17((countryorcity)adjwide).tw.
(nonefound)andtextwords
18outreach.tw.
relatingto
19(multiadj(componentorfacetorfacetedordisciplinary)).tw.
Iintervention
20(interadjdisciplinary).tw.
21(fieldadjbased).tw.
22local.tw.
23citizen$.tw.
24(multiadjcommunity).tw.
25or/1324
26massmedia/
27audiovisualaids/
Thisreviewwantstoexclude
28exptelevision/
massmediainterventionsas
29motionpictures/
acommunitybased
30radio/
intervention(areviewhas
31exptelecommunications/
alreadybeencompletedon
32videotaperecording/
thistopic)
33newspapers/
seesearchline42
34advertising/
35(tvortelevis$).tw.
36(advertis$adj4(preventorprevention)).tw.
37(massadjmedia).tw.
38(radioormotionpicturesornewspaper$orvideo$oraudiovisual).tw.
39or/2638
40youngpeopleandsmokingandcommunitybasedintervention
407and12and25
41youngpeopleandsmokingandmassmediainterventions
417and12and39
42communityinterventionsnotincludingmassmediainterventions
4240not41
36
1.
Startwiththeprimaryconcept,i.e.youngpeople.
2. The Ovid search interface allows plain language to be mapped to related subject headings,
terms from a controlled indexing list (called controlled vocabulary) or thesaurus (eg. MeSH in
MEDLINE).Mapthetermyoungpeople
3.
Theresultshouldlooklikethis:
Scope note to see

related terms
Link to tree
37
4.
ClickonthescopenotefortheAdolescentterm(isymbol)tofindthedefinitionofadolescent
andtermsrelatedtoadolescentthatcanalsobeusedinthesearchstrategy.NotethatMinorscan
alsobeusedforthetermadolescent.
Related subject
headings
Related
textwords
4.
ClickonPreviouspageandthenAdolescenttoviewthetree(thenumberswillbedifferent).
Explode box to
include
narrower terms
No narrower terms for

adolescent
Broader term
Child
Narrower term
Child, Preschool
38
5. Because adolescent has no narrower terms click continue at the top of the screen. This will
producealistofallsubheadings.
(Ifadolescenthadnarrowertermsthatareimportanttoincludetheexplodeboxwouldbechecked).
6.Presscontinue(itisnotrecommendedtoselectanyofthesubheadingsforpublichealthreviews).
7.ThescreenwillnowshowallcitationsthathaveadolescentasaMeSHheading.
8.Repeatthisstrategyusingthetermschildandminors.
39
9.Usingfreetextortextwordstoidentifyarticles.
Truncation $ Unlimited truncation is used to retrieve all possible suffix variations of a root
word. Type the desired root word or phrase followed by either of the truncation characters $
(dollar sign). Another wild card character is ? (question mark). It can be used within or at the
endofaquerywordtosubstituteforoneornocharacters.Thiswildcardisusefulforretrieving
documentswithBritishandAmericanwordvariants.
10.Freetextwordsforsearchingtypeinyoungpeople.tw.
YoucanalsocombinealltextwordsinonelinebyusingtheoperatorORthiscombinestwoormore
queryterms,creatingasetthatcontainsallthedocumentscontaininganyofthequeryterms(with
duplicates eliminated). For example, type in (child$ or juvenile$ or girl$ or boy$ or teen$ or
adolescen$).tw.
11.Combineallyoungpeoplerelatedtermsbytypingor/16
40
12.Completesearches812and1325intheworkedexample.Combinethethreesearches(7,12,25)
byusingthecommandAND.
13.Welldone!NowtryasearchusingthePICOquestiondevelopedearlierinUnitFive.Agoodstart
istolookatcitationsthatareknowntoberelevantandseewhattermshavebeenusedtoindexthe
article,orwhatrelevantwordsappearintheabstractthatcanbeusedastextwords.
Goodluck!
41
42
Unit Seven: Data Abstraction
LearningObjectives
Tounderstandtheimportanceofawelldesigned,unambiguousdataabstractionform
Toidentifythenecessarydatatoabstract/extractfromtheprimarystudies
Oncedatahasbeenabstractedfromprimarystudiesthesynthesisoffindingsbecomesmucheasier.
The data abstraction form becomes a record to refer back to during the latter stages of the review
process.Inaddition,theformsmaybeofusetofuturereviewerswhowishtoupdatethereview.
Differentstudydesignswillrequiredifferentdataabstractionforms,tomatchthequalitycriteriaand
reportingofthestudy.Thedataabstractionformshouldmirrortheformatforwhichtheresultswill
bepresented.
Detailstocollect:
Sometimes,thedatarequiredforsynthesisisnotreportedintheprimarystudies,orisreportedina
way that isnt useful for synthesis. Studies vary in the statistics they use to summarise the results
(medians rather than means) and variation (standard errors, confidence intervals, rangesinstead of
standarddeviations).1Itisthereforeimportantthatauthorsarecontactedforanyadditionaldetailsof
thestudy.
** It is possible that one study is reported in more than one journal (duplication of publication). In
addition,differentaspectsofthestudy(processoutcomes,interventiondetails,outcomeevaluations)
maybereportedindifferentpublications.Allofthepapersfromthestudycanbeusedtoassistwith
dataabstraction.Howevereachpapershouldhaveauniqueidentifierinthedataabstractionformto
recordwheretheinformationwaslocated.
Thedataabstractionformshouldbepilotedonasmallgroupofstudiestoensuretheformcaptures
alloftheinformationrequired.Inaddition,ifthereismorethanonerevieweraselectionofstudies
shouldbetestedtoseeifthereviewersdifferintheinterpretationofthedetailsofthestudyanddata
abstraction form. If reviewers do not reach a consensus they should try to determine why their
accountsdiffer.
Thedataabstractionformshouldcontainthecriteriausedforqualityappraisal.Ifthestudydoesnot
meet the predetermined criteria for quality there is no point continuing with the data abstraction
process.
Usefuldatatocollect:
Publicationdetails
Theoreticalframework
Studydetails(date,followup)
Provider
Studydesign
Setting
Populationdetails(n,
Targetgroup
characteristics)
Consumerinvolvement
Interventiondetails
43
Processmeasuresadherence,
exposure,training,etc
Contextdetails
Outcomesandfindings
Examplesofdataabstractionforms:
Anumberofdataabstractionformsareavailableinthefollowingpublication:HedinA,andKallestal
C. Knowledgebased public health work. Part 2: Handbook for compilation of reviews on
interventionsinthefieldofpublichealth.NationalInstituteofPublicHealth.2004.
Otherdataabstractionformscanbefoundat:
The Effective Public Health Practice Project reviews (appendices in reviews)
http://www.city.hamilton.on.ca/phcs/EPHPP/default.asp
TheCommunityGuidehttp://www.thecommunityguide.org/methods/abstractionform.pdf
EffectivePracticeandOrganisationofCareReviewGrouphttp://www.epoc.uottawa.ca/tools.htm
NHSCRDReportNumber4.http://www.york.ac.uk/inst/crd/crd4_app3.pdf
Pleasenote:Nosingledataabstractionformisabsolutelysuitableforeveryreview.Formswillneed
tobeadaptedtomakethemrelevanttotheinformationrequiredforthereview.
REFERENCES
44
Unit Eight: Principles of Critical Appraisal

LearningObjectives
Tounderstandthecomponentsthatrelatetoqualityofaquantitativeandqualitativeprimarystudy
Tounderstandthetermbiasandtypesofbias
Togainexperienceintheassessmentofthequalityofahealthpromotionorpublichealthprimarystudy
(qualitativeandquantitative)
1) QUANTITATIVESTUDIES
Validity
Validityisthedegreetowhicharesultfromastudyislikelytobetrueandfreefrombias.1Interpretation
offindingsfromastudydependsonbothinternalandexternalvalidity.
Internalvalidity
Theextent towhichtheobservedeffectsaretrueforpeopleinastudy.1Commontypesofbias
that affect internal validity include; allocation bias, confounding, blinding, data collection
methods, withdrawals and dropouts, statistical analysis, and intervention integrity (including
contamination).Unbiasedresultsareinternallyvalid.
Externalvalidity(generalisabilityorapplicability)
Theextenttowhichtheeffectsinastudytrulyreflectwhatcanbeexpectedinatargetpopulation
beyondthepeopleincludedinthestudy.1Note:Onlyresultsfrominternallyvalidstudiesshould
beconsideredforgeneralisability.
Criticalappraisaltools
1)RCTs,nonrandomisedcontrolledstudies,uncontrolledstudies
TheQualityAssessmentToolforQuantitativeStudies
(http://www.city.hamilton.on.ca/phcs/EPHPP/).
Developed by the Effective Public Health Practice Project, Canada. This tool assesses both
internal and external validity. Content and construct validity have been established.2 Rates the
followingcriteriarelevanttopublichealthstudies:
1) selectionbias(externalvalidity)
6) withdrawalsanddropouts(attrition
2) allocationbias
bias)
3) confounding
7) statisticalanalysis
4) blinding(detectionbias)
8) interventionintegrity
5) datacollectionmethods
2)Interruptedtimeseriesdesigns
MethodsfortheappraisalandsynthesisofITSdesignsareincludedontheEffectivePractice
andOrganisationofCarewebsite(www.epoc.uottawa.ca).
45
Introductionofbiasintotheconductofaprimarystudy
Recruitparticipants
Allocatetointervention
andcontrolgroups
SELECTION
BIAS
ALLOCATION
BIAS
Interventiongroup
CONFOUNDING
(dissimilargroups)
Controlgroup
Implementintervention
INTEGRITYOF
INTERVENTION
Implementintervention
Followupparticipants
INTENTION
TOTREAT
Followupparticipants
WITHDRAWALS/
DROPOUTS
Measureoutcomes
BLINDING
OUTCOME
ASSESSORS
Measureoutcomes
DATA
COLLECTION
METHODS
Analysedata
STATISTICAL
ANALYSIS
Analysedata
46
Typesofbiasinhealthpromotionandpublichealthstudies
Bias
A systematic error or deviation in results. Common types of bias in health promotion and public
healthstudiesarisefromsystematicdifferencesinthegroupsthatarecompared(allocationbias),the
exposure to other factors apart from the intervention of interest (eg. contamination), withdrawals
from the study (attrition bias), assessment of outcomes (detection bias), including data collection
methods,andinadequateimplementationoftheintervention.
The following sections of this unit describe the types of bias to be assessed using The Quality
Assessment Tool for Quantitative Studies (http://www.city.hamilton.on.ca/phcs/EPHPP/) developed
by the Effective Public Health PracticeProject, Canada. Further informationis also provided in the
QualityAssessmentDictionaryprovidedinthefollowingpages.
1)Selectionbias
Selection bias is used to describe a systematic difference in characteristics between those who are
selectedforstudyandthosewhoarenot.AsnotedintheQualityAssessmentDictionary,itoccurs
when the study sample (communities, schools, organisations, etc) does not represent the target
populationforwhomtheinterventionwasintended.Examples:
Resultsfromateachinghospitalmaynotbegeneralisabletothoseinnonteachinghospitals
Resultswhichrecruitedvolunteersmaynotbegeneralisabletothegeneralpopulation
ResultsfromlowSESschoolsorinnercityschoolsmaynotbegeneralisabletoallschools
Examplesfromwww.reaim.org
Example:Eakinandherassociates(1998)illustrateselectionbiasinasmokingcessationstudyoffered
toparticipantsinaplannedparenthoodprogram.Theybeginbyexplicitlyreportingtheirinclusion
criteriafemalesmokersbetween15and35yearsofagewhoarepatientsataplannedparenthood
clinic.Duringaroutinevisittotheclinicthepatientservicesstaffdescribedthestudyandsolicited
participants.Thosewomenwhodeclined(n=185)wereaskedtocompleteashortquestionnairethat
included questions to assess demographics, smoking rate, and reasons for nonparticipation.
Participants (n=518) also completed baseline demographic and smoking rate assessments. They
tracked recruitment efforts and reported that 74% percent of the women approached agreed to
participate in the study. To determine the representativeness of the sample two procedures were
completed.First,basedoninformationfrompatientmedicalcharts,thosewhowerecontactedwere
compared on personal demographics to those who were not contacted. Second, participants were
compared to nonparticipants on personal demographics and smoking rate. The study found that
thosecontacteddidnotdifferfromthosenotcontactedonanyofthetestvariables.Also,theresults
suggested that participants were slightly younger than nonparticipants, but there were no other
differencesbetweenthesegroups.ThissuggeststhatEakinandherassociateswerefairlysuccessful
incontactingandrecruitingafairlyrepresentativesampleoftheirtargetpopulation.
Example: The Language for Health (Elder et al., 2000) nutrition education intervention provides a
good example of determining the representativeness of study participants to a given target
population. The behaviour change intervention was developed to target Latino participants in
Englishasasecondlanguage(ESL)classesatsevenschools.Toexaminerepresentativeness,the710
participantsinthestudywerecomparedtotheoverallLatinoESLstudentpopulationinthecity.This
comparison revealed that the intervention participants did not differ from the general ESL student
47
population on gender, age, or education level. As such, the authors concluded that the study had
stronggeneralisabilitytothegreatertargetpopulation(Elderetal.,2000).
Example: All the participating schools were state primary schools sited outside the inner city area.
Sociodemographic measures suggested that the schools populations generally reflected the Leeds
school aged population, although there was a slight bias towards more advantaged children. The
schoolshad142%childrenfromethnicminoritiesand729%entitledtofreeschoolmealscompared
with11%and25%respectivelyforLeedschildrenasawhole.
2)Allocationbias
Biascanresultfromthewaythattheinterventionandcontrolgroupsareassembled.3 Unlessgroups
areequivalentorbalancedatbaseline,differencesinoutcomescannotconfidentlybeattributedtothe
effectsoftheintervention.4Studieswhichshowthatcomparisongroupsarenotequivalentatbaseline
havehighallocationbias.
Randomallocationisthebestmethodtoproducecomparisongroupsthatarebalancedatbaselinefor
known and unknown confounding factors, and therefore reduce allocation bias. This is usually
achievedbytoincossingordevelopingcomputergeneratedrandomnumbertables.Thisensuresthat
everyparticipantinthestudyhasanequalchance(50%/50%)ofbeingintheinterventionorcontrol
group.
Ideally,thecointossingorcomputergeneratedrandomisationshouldbecarriedoutbyindividuals
external to the study. Once the allocation scheme is developed, the allocation of participants to
intervention and control group should be carried out by someone who is not responsible for the
studytopreventmanipulationbyresearchersandparticipants.Therefore,oncetheallocationscheme
has been developed it is important that allocation to intervention and control group is concealed.
Concealmentofallocationistheprocesstopreventforeknowledgeofgroupassignment.1Methodsto
conceal allocation include allocation by persons external to the study and sequentially numbered,
sealed opaque envelopes. Unfortunately, information on concealment of allocation is very rarely
reportedinprimarystudies.
Example:Worksiteswererandomisedwithinblocks:unionisedversusnonunionised;singleversus
multiple buildings; and three worksites that were part of a single large company. Worksites were
randomly assigned by the study biostatistician using a process conducted independently from the
interventionteam.
Example:Subjectswererandomisedtooneofthreearms:(1)DirectAdvice,(2)BriefNegotiationor
(3)Controlbyhouseholdwitheachmonthlybatchformingasinglepermutedblock.Randomisation
of intervention arms were sent to CF (the investigator) in sealed opaque envelopes. At the health
check participants wereasked to consent to a randomised trial of the effect ofhealth professionals
communication style on patients health behaviour, namely physical activity. If consent was given,
theenvelopewasopenedandtheappropriateinterventioncarriedout.
There are also quasirandomised methods of allocating participants into intervention and control
groups.Theseincludealternation(eg.firstpersonintervention,secondpersoncontrol),allocationby
dateofbirth,dayofweek,etc.Thesemethodsarenotabletoconcealallocation,donotguaranteethat
everyparticipanthasanequalchanceofbeingineithercomparisongroup,andconsequentiallydo
notguaranteethatgroupswillbesimilaratbaseline.
48
Example:Familiesthenwererandomlyassignedtoanintervention(n=65)orcontrolgroup(n=70).
An alternateday randomisation system was used to simplify intervention procedures and more
importantlytoavoidwaitingroomcontaminationofcontrolfamiliesbyinterventionfamiliesexiting
theroomswithbooksandhandouts.
Nonrandomised studies often involve the investigators choosing which individuals or groups are
allocatedtointerventionandcontrolgroups.Therefore,thesestudydesignshavehighallocationbias
andarelikelytoproduceunevengroupsatbaseline.Evenifeveryattempthasbeenmadetomatch
the intervention and control groups it is impossible to match for unknown confounding factors.
Furthermore,thereareinherentproblemsinassessingknownconfoundingfactors,asmeasurement
toolsforcollectingtheinformationmaynotbevalid.
3)Confounding
Confounding is a situation where there are factors (other than the intervention) present which
influence the outcome under investigation. A confounding factor has to be related to both the
intervention and the outcome. For example, Body Mass Index at baseline would be a confounding
factor when investigatingthe effect ofschool basednutritionintervention onpreventing obesity.A
factorcanonlyconfoundanassociationifitdiffersbetweentheinterventionandcontrolgroups.
Theassessmentofconfoundingisthenextstageinthecriticalappraisalprocessafterdeterminingthe
method of allocation. Remember, randomisation of participants or groups to intervention/control
group is the best way to distribute known and unknown confounding factors evenly. Differences
between groups in baseline characteristics that relate to the outcome may distort the effect of the
interventionunderinvestigation.
Beforebeginningtoanswerthiscriticalappraisalquestionitisimportanttodeterminethepotential
confounding factors relating to the particular intervention under question. Good knowledge of the
subjectareaisessentialwhendeterminingpotentialconfounders.
Example:
Presenceofconfounders:Interventionandcontrolsubjectsweresimilaronbaselinevariables.
Adjustment for confounders: We assessed the effect of the intervention after adjusting for sex, age,
baselineBMIandtypeofschool.
4)Blindingofoutcomeassessors(detectionbias)
Outcomeassessorswhoareblindtotheinterventionorcontrolstatusofparticipantsshouldlogically
belessbiasedthanoutcomeassessorswhoareawareofthestatusoftheparticipants.
Detectionbiasisimportantinhealthpromotionstudieswhereoutcomesaregenerallysubjective.For
example,ifoutcomeassessorswererequiredtointerviewchildrenregardingtheirfoodconsumption
in the past 24 hours, they may be more likely to prompt the intervention group to respond
favourably.
Example: Questionnaires were developed based on a review of other STD/HIV risk questionnaires
andourfindingsfromfocusgroupsandindepthinterviews.Whenadministeringthe3and9month
followupquestionnaires,interviewerswereblindtothestudygroupassignmentofadolescents.
49
5)Datacollectionmethods
As highlighted, a number of outcomes measured in health promotion are subjectively reported.
Although a number of outcomes can be measured objectively, such as Body Mass Index or
pregnancy,generallyhealthpromotioninterventionsaretryingtochangebehaviour,whichusually
requires subjective selfreporting (unless behaviour is directly observed). Subjective outcome data
mustbecollectedwithvalidandreliableinstruments.
Criticalappraisalthereforerequiresthereadertoassesswhethertheoutcomeshavebeenmeasured
withvalidandreliableinstruments.
Example: We used three validated tools to evaluate the effect of the intervention on psychological
wellbeing; the selfperception profile for children; a measure of dietary restraint; and the adapted
bodyshapeperceptionscale.
6)Withdrawals(attritionbias)
Attritionbiasrelatestothedifferencesbetweentheinterventionandcontrolgroupsinthenumberof
withdrawalsfromthestudy.Itarisesbecauseofinadequaciesinaccountingforlossesofparticipants
duetodropouts,leadingtomissingdataonfollowup.4
Iftherearesystematicdifferencesinlossestofollowupthecharacteristicsoftheparticipantsinthe
interventionandcontrolgroupsmaynotbeassimilarastheywereatthebeginningofthestudy.For
randomisedcontrolledtrials,theeffectofrandomisationislostifparticipantsarelosttofollowup.
An intentiontotreat analysis, where participants are analysed according to the group they were
initiallyallocated,protectsagainstattritionbias.
For clusterlevel interventions all members of the cluster should be included in the evaluation,
regardlessoftheirexposuretotheintervention.5Thus,asampleofeligiblemembersoftheclusteris
generallyassessed,notonlythosewhoweresufficientlymotivatedtoparticipateintheintervention.5
Therefore,itissaidthatstudiestrackingchangeinentirecommunitiesarelikelytoobservesmaller
effectsizesthanotherstudiestrackingchangeininterventionparticipantsalone.5
Example:Twentyone(14%)ofthe148patientswhoenteredthetrialdroppedout,aratecomparable
tothatinsimilartrials.Ofthese,19wereintheinterventiongroupanddroppedoutduringtreatment
(eightformedicalreasons,sevenforpsychiatricreasons,fourgavenoreason,oneemigrated,andone
wasdissatisfiedwithtreatment).
Example:Completedfollowupresponseswereobtainedfrom87%ofsurvivinginterventionpatients
and 79% of surviving control patients. There were no significant differences between respondents
andnonrespondentsinage,sex,educationalachievement,maritalstatus,orbaselinehealthstatus.
7)Statisticalanalysis
A trial/study must have a sufficient sample size to have the ability (or power) to detect significant
differences between comparison groups. A lack of a significant effect could be due to the study
havinginsufficientnumbers,ratherthantheinterventionbeingineffective.
The publication of the study should report whether a sample size calculation was carried out. For
group/clusterstudiesthestudyshouldreportthatittooktheclusteringintoaccountwhencalculating
sample size. These types of study designs should also analyse the data appropriately; if
schools/classroomswereallocatedtointerventionandcontrolgroupsthentheymustbeanalysedat
50
thislevel.Oftenthisisnotthecase,astheinterventionisallocatedtoschools(forpracticalreasons)
and individual outcomes (eg. behaviour change) are analysed. In these instances, a cluster analysis
(takingintoaccountthedifferentlevelsofallocationandanalysis)shouldbereported.
Example: A power calculation indicated that with five schools in each arm, the study would have
80%powertodetectandunderlyingdifferenceinmeansofanormallydistributedoutcomemeasure
of1.8standarddeviationsatthe5%significanceleveland65%todetectadifferenceof1.5SD.This
tookintoaccounttheclusterrandomisationdesign.
Example:Thestatisticalmodeltookintoaccountthelackofindependenceamongsubjectswithinthe
school,knownastheclusteringeffect.
8)Integrityofintervention
Criticalappraisalshoulddetermineifresultsofineffectivenesswithinprimarystudiesissimplydue
to incomplete delivery of the intervention (failure of implementation) or a poorly conceptualised
intervention (failure of intervention concept or theory)6,7. Evaluating a program that has not been
adequatelyimplementedisalsocalledaTypeIIIerror8.Assessingthedegreetowhichinterventions
areimplementedasplannedisimportantinpreventiveinterventionswhichareoftenimplementedin
conditionsthatpresentnumerousobstaclestocompletedelivery.6 Areviewofsmokingcessationin
pregnancy9 found that in studies which measured the implementation of the intervention the
implementationwaslessthanideal.
In order to provide a comprehensive picture of intervention integrity five dimensions of the
intervention should be measured. These five factors are adherence, exposure, quality of delivery,
participantresponsiveness,andprogramdifferentiation(topreventcontamination).6
Adherence:theextenttowhichspecifiedprogramcomponentsweredeliveredasprescribedin
programmanuals.
Exposure: an index that may include any of the following: (a) the number of sessions
implemented; (b) the length of each session; or (c) the frequency with which program
techniqueswereimplemented.
Qualityofdelivery:ameasureofqualitativeaspectsofprogramdeliverythatarenotdirectly
related to the implementation of prescribed content, such as implementer enthusiasm, leader
preparedness and training, global estimates of session effectiveness, and leader attitude
towardstheprogram.
Participantresponsiveness:ameasureofparticipantresponsetoprogramsessions,whichmay
includeindicatorssuchaslevelsofparticipationandenthusiasm.
Program differentiation: a manipulation check that is performed to safeguard against the

diffusion of treatments, that is, to ensure that the subjects in each experimental condition
receivedonlytheplannedinterventions.Contaminationmaybeaproblemwithinmanypublic
healthandhealthpromotionstudieswhereinterventionandcontrolgroupscomeintocontact
witheachother.ThisbiasisminimisedthroughtheuseofclusterRCTs.
Thesedataprovideimportantinformationthatenhancestheabilitytointerpretoutcomeassessments,
identify competing explanations for observed effects and measure exposure to the intervention.5
However,veryfewstudiesdisentanglethefactorsthatensuresuccessfuloutcomes,characterisethe
51
failure to achieve success, or attempt to document the steps involved in achieving successful
implementationofcomplexinterventions.10,11
In relation to the appraisal of process evaluations the EPPICentre has developed a 12question
checklist,availableat:
http://eppi.ioe.ac.uk/EPPIWeb/home.aspx?page=/hp/reports/phase/phase_process.htm.
Doesthestudyfocusonthedeliveryofahealthpromotionintervention?
Screeningquestions
1. Doesthestudyfocusonahealthpromotionintervention?
2. Doestheinterventionhaveclearlystatedaims?
3. Doesthestudydescribethekeyprocessesinvolvedindeliveringtheintervention?
Detailedquestions
4. Doesthestudytellyouenoughaboutplanningandconsultation?
5. Doesthestudytellyouenoughaboutthecollaborativeeffortrequiredfortheintervention?
6. Does the study tell you enough about how the target population was identified and
recruited?
7. Doesthestudytellyouenoughabouteducationandtraining?
B)Whataretheresults?
8. Werealltheprocessesdescribedandadequatelymonitored?
9. Wastheinterventionacceptable?
C)Willtheresultshelpme?
10. Cantheresultsbeappliedtothelocalpopulation?
11. Wereallimportantprocessesconsidered?
12. Ifyouwantedtoknowwhetherthisinterventionpromoteshealthwhatoutcomeswouldyou
wanttomeasure?
Examplesofassessmentoftheinterventionimplementation
Example: This study evaluated a 19lesson, comprehensive schoolbased AIDS education program
lastingoneyearinruralsouthwesternUganda.Quantitativedatacollection(viaquestionnaire)found
thattheprogramhadverylittleeffectonoverallknowledge,overallattitude,intendedcondomuse,
andintendedassertivebehaviour.Datafromthefocusgroupdiscussionssuggestedthattheprogram
wasincompletelyimplemented,andthatkeyactivitiessuchascondomsandtheroleplayexercises
were only completed superficially. Themainreasons for this were ashortage of classroom time, as
wellasteachersfearofcontroversy(condomsareanunwelcomeintrusionintoAfricantraditionand
maybeassociatedwithpromiscuity).Teacherstendedtoteachonlytheactivitiesthattheypreferred,
leavingouttheactivitiestheywerereluctanttoteach.Oneproblemwiththeinterventionwasthatthe
program was additional to the standard curriculum, so teaching time was restricted. It was also
found that neither teachers nor students were familiar with roleplay. Furthermore, a number of
teachersalsolefttheinterventionschools(ordied).
Therefore, it is suggested that AIDS education programs in subSaharan Africa may be more fully
implementediftheyarefullyincorporatedintonationalcurricula(seeinterpretationorresultsunit)
andexaminedaspartofschooleducation.
References:
Kinsman J, Nakiyingi J, Kamali A, Carpenter L, Quigley M, Pool R, Whitworth J. Evaluation of a
comprehensiveschoolbasedAIDSeducationprogrammeinruralMasaka,Uganda.
HealthEducRes.2001Feb;16(1):85100.
KinsmanJ,HarrisonS,KengeyaKayondoJ,KanyesigyeE,MusokeS,WhitworthJ.Implementation
ofacomprehensiveAIDSeducationprogrammeforschoolsinMasakaDistrict,Uganda.AIDSCare.
1999Oct;11(5):591601.
52
Example:Gimme5Fruit,JuiceandVegetableintervention.Thisschoolbasedinterventionincluded
componentstobedeliveredattheschoolandnewsletterswithfamilyactivitiesandinstructionsfor
intervention at home. Overall, there were small changes in fruit, juice, and vegetable consumption.
Teacherselfreporteddeliveryoftheinterventionwas90%.However,allteacherswereobservedat
leastonceduringthe6weekinterventionanditwasfoundthatonly51%and46%ofthecurriculum
activities were completed in the 4th and 5th grade years. Reference: Davis M, Baranowski T,
Resnicow K, Baranowski J, Doyle C, Smith M, Wang DT, Yaroch A, Hebert D. Gimme 5 fruit and
vegetablesforfunandhealth:processevaluation.HealthEducBehav.2000Apr;27(2):16776.
REFERENCES
1.
2.
3.
4.
5.
6.
7.
8.
9.
Cochrane
Reviewers
Handbook
Glossary,
Version
4.1.5.
www.cochrane.org/resources/handbook/glossary.pdf,6 December2004[datelastaccessed]
ThomasH,MicucciS,ThompsonOBrienMA,BrissP.Towardsareliableandvalidinstrument
forqualityassessmentofprimarystudiesinpublichealth.Unpublishedwork.2001.
Reviews
and
Dissemination,
University
of
York.
March
2001.
Sorensen G, Emmons K, Hunt MK, Johnston D. Implications of the results of community
interventiontrials.AnnuRevPublicHealth.1998;19:379416.
Dane AV, Schneider BH. Program integrity in primary and early secondary prevention: are
implementationeffectsoutofcontrol?ClinPsycholRev1998;18:2345.
Rychetnik L, Frommer M, Hawe P, Shiell A. Criteria for evaluating evidence on public health
interventions.JEpidemiolCommunityHealth2002;56:11927.
Basch CE, Sliepcevich EM, Gold RS, Duncan DF, Kolbe LJ. Avoiding type III errors in health
educationprogramevaluations:acasestudy.HealthEducQ.1985Winter;12(4):31531.
LumleyJ,OliverS,WatersE.Interventionsforpromotingsmokingcessationduringpregnancy.
In:TheCochraneLibrary,Issue3,2004.Chichester,UK:JohnWiley&Sons,Ltd.
10. Steckler A, Linnan L (eds). Process Evaluation for Public Health Interventions and Research.
JosseyBass,USA,2002.
11. GreenJ,TonesK.Towardsasecureevidencebaseforhealthpromotion.JournalofPublicHealth
Medicine1999;21(2):1339.
53
ADDITIONALREADING
Rychetnik L, Frommer M, Hawe P, Shiell A. Criteria for evaluating evidence on public health
interventions.JEpidemiolCommunityHealth2000;56:11927.
KahanB,GoodstadtM.TheIDMManualforUsingtheInteractiveDomainModalApproachtoBest
PracticesinHealthPromotion.
www.utoronto.ca/chp/bestp.html#Outputs/Products
GuyattGH,SackettDL,CookDJ,fortheEvidenceBasedMedicineWorkingGroup.UsersGuidesto
theMedicalLiterature.II.HowtoUseanArticleAboutTherapyorPrevention.A.AretheResultsof
theStudyValid?EvidenceBasedMedicineWorkingGroup.JAMA1993;270(21):25982601.
Notes on terms/statistics used in primary studies: Adapted from the Cochrane Reviewers
HandbookGlossary,Version4.1.5.Availableatwww.cochrane.org/resources/handbook/glossary.pdf
Bias
A systematic error or deviation in results. Common types of bias in health promotion and public
healthstudiesarisefromsystematicdifferencesinthegroupsthatarecompared(allocationbias),the
exposure to other factors apart from the intervention of interest (eg. contamination), withdrawals
from the study (attrition bias), assessment of outcomes (detection bias), including data collection
methods,andinadequateimplementationoftheintervention.
Blinding
Keeping secret group assignment (intervention or control) from the study participants or
investigators. Blinding is used to protect against the possibility that knowledge of assignment may
affect subject response to the intervention, provider behaviours, or outcome assessment. The
importance of blinding depends on how objective the outcome measure is; blinding is more
importantforlessobjectivemeasures.
ConfidenceInterval(CI)
Therangewithinwiththetruevalue(eg.sizeofeffectoftheintervention)isexpectedtoliewithina
givendegreeofcertainty(eg.95%).Itisabouttheprecisionoftheeffect.CIsthereforeindicatethe
spreadorrangeofvalueswhichcanbeconsideredprobable.ThenarrowertheCIthemoreprecise
wecantakethefindingstobe.
Confounding
Asituationinwhichthemeasureoftheeffectofaninterventionorexposureisdistortedbecauseof
theassociationofexposurewithotherfactorsthatinfluencetheoutcomeunderinvestigation.
Intentiontotreat
Anintentiontotreatanalysisisoneinwhichalltheparticipantsinthetrialareanalysedaccordingto
theinterventiontowhichtheyareallocated,whethertheyreceiveditornot.
Oddsratios
The ratio of the odds of an event (eg. prevention of smoking, unintended pregnancy) in the
interventiongrouptotheoddsofaneventinthecontrolgroup.
54
pvalue
Theprobability(from0to1)thattheresultsobservedinastudycouldhaveoccurredbychance.They
areusedabenchmarkofhowconfidentwecanbeinaparticularresult.Youwilloftenseestatements
like this result was significant at p<0.05. This means that we could expect this result to occur by
chancenomorethan5timesper100(oneintwenty).Thelevelofp<0.05isconventionallyregarded
asthelowestlevelatwhichwecanclaimstatisticalsignificance.
Relativerisk
The ratio of the risks of an event (eg. prevention of smoking, unintended pregnancy) in the
intervention group to the odds of an event in the control group. eg. RR=0.80 for unintended
pregnancytheinterventiongrouphada20%reducedriskofunintendedpregnancycomparedto
thoseinthecontrolgroup.Note:aRRof<1isgoodifyouwantlessofsomething(pregnancy,death,
obesity), a RR>1 is good if you want more of something (people stopping smoking, using birth
control).
55
56
QUALITY ASSESSMENT TOOL FOR QUANTITATIVE STUDIES
Ref ID:
Author:
Year:
Reviewer:
COMPONENT RATINGS
A)
SELECTION BIAS
(Q1)
Are the individuals selected to participate in the study likely to be representative of

the target population?
Very Likely
(Q2)
Somewhat Likely
Not Likely
What percentage of selected individuals agreed to participate?

80 - 100%
Agreement
60 - 79%
Agreement
Rate this section (see dictionary)
Less than 60%

Agreement
Strong
Not Reported
Moderate
Not Applicable
Weak
B)
ALLOCATION BIAS
Indicate the study design
RCT
(go to i)
Quasi-Experimental
(go to C)
Case-control, Before/After study,

No control group,
or Other:
(go to C)
(i)
Is the method of random allocation stated?
Yes
No
(ii)
If the method of random allocation is stated

is it appropriate?
Yes
No
Was the method of random allocation

reported as concealed?
Yes
No
(iii)
Strong
Moderate
Weak
C) CONFOUNDERS
(Q1)
Prior to the intervention were there between group differences for important
confounders reported in the paper?
Yes
No
Cant Tell
Please refer to your Review Group list of confounders. See the dictionary for some examples.
Relevant Confounders reported in the study:
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
57
(Q2)
If there were differences between groups for important confounders, were they
adequately managed in the analysis?
Yes
(Q3)
No
Not Applicable
Were there important confounders not reported in the paper?

Yes
No
Relevant Confounders NOT reported in the study:

_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
Strong
Moderate
Weak
D) BLINDING
(Q1)
Was (were) the outcome assessor(s) blinded to the intervention or exposure status of
participants?
Yes
No
Strong
Not reported
Not applicable
Moderate
Weak
E) DATA COLLECTION METHODS

(Q1)
Were data collection tools shown or are they known to be valid?

Yes
(Q2)
No
Were data collection tools shown or are they known to be reliable?

Yes
No
Strong
Moderate
Weak
F) WITHDRAWALS AND DROP-OUTS

(Q1)
Indicate the percentage of participants completing the study. (If the percentage differs
by groups, record the lowest).
80 -100%
60 - 79%
Less than
60%
Not Reported
Strong
Moderate
Not Applicable
Weak
58
G)
ANALYSIS
(Q1)
Is there a sample size calculation or power calculation?

Yes
(Q2)
Partially
Is there a statistically significant difference between groups?

Yes
(Q3)
No
No
Not Reported
Are the statistical methods appropriate?

Yes
No
Not Reported
(Q4a) Indicate the unit of allocation (circle one)

Community
Organization/ Group
Provider
Client
Institution
Provider
Client
Institution
(Q4b) Indicate the unit of analysis (circle one)

Community
Organization/ Group
(Q4c) If 4a and 4b are different, was the cluster analysis done?

Yes
(Q5)
Not Applicable
Is the analysis performed by intervention allocation status (i.e. intention to treat)

rather than the actual intervention received?
Yes
H)
No
No
Cant Tell
INTERVENTION INTEGRITY
(Q1) What percentage of participants received the allocated intervention or exposure of

interest?
80 -100%
(Q2)
Less than
60%
Not Reported
Not Applicable
Was the consistency of the intervention measured?

Yes
Q3)
60 - 79%
No
Not reported
Not Applicable
Is it likely that subjects received an unintended intervention (contamination or

cointervention) that may influence the results?
Yes
No
Cant tell
59
SUMMARY OF COMPONENT RATINGS

Please transcribe the information from the grey boxes on pages 1-3 onto this page.
A SELECTION BIAS
Strong
Moderate
Weak
B STUDY DESIGN
Strong
Moderate
Weak
C CONFOUNDER
Strong
Moderate
Weak
D BLINDING
Strong
Moderate
Weak
E DATA COLLECTION METHODS

Strong
Moderate
Weak
F WITHDRAWALS AND DROPOUTS

Strong
Moderate
Weak
ANALYSIS
Comments ____________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
INTERVENTION INTEGRITY
Comments ____________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
WITH BOTH REVIEWERS DISCUSSING THE RATINGS:

Is there a discrepancy between the two reviewers with respect to the component ratings?
No
Yes
If yes, indicate the reason for the discrepancy
Oversight
Differences in
Interpretation of Criteria
Differences in
Interpretation of Study
60
DICTIONARY
for the
Effective Public Health Practice Project
Quality Assessment Tool for Quantitative Studies
INTRODUCTION
The purpose of this tool is to assess the methodological quality of relevant studies since lesser quality
studies may be biased and could over-estimate or under-estimate the effect of an intervention. Each of
two raters will independently assess the quality of each study and complete this form. When each rater is
finished, the individual ratings will be compared. A consensus must be reached on each item. In cases of
disagreement even after discussion, a third person will be asked to assess the study.
When appraising a study, it is helpful to first look at the design then assess other study methods. It is
important to read the methods section since the abstract (if present) may not be accurate. Descriptions of
items and the scoring process are located in the dictionary that accompanies this tool.
The scoring process for each component is located on the last page of the dictionary.
INSTRUCTIONS FOR COMPLETION
Circle the appropriate response in each component section (A-H). Component sections (A-F) are each rated
using the roadmap on the last page of the dictionary. After each individual rater has completed the form,
both reviewers must compare their ratings and arrive at a consensus.
The dictionary is intended to be a guide and includes explanations of terms.
The purpose of this dictionary is to describe items in the tool thereby assisting raters to score study
quality. Due to under-reporting or lack of clarity in the primary study, raters will need to make
judgements about the extent that bias may be present. When making judgements about each component,
raters should form their opinion based upon information contained in the study rather than making inferences
about what the authors intended.
A)
SELECTION BIAS
Selection bias occurs when the study sample does not represent the target population for whom
the intervention is intended. Two important types of biases related to sample selection are referral
filter bias and volunteer bias. For example, the results of a study of participants suffering from asthma
from a teaching hospital are not likely to be generalisable to participants suffering from asthma
from a general practice. In volunteer bias, people who volunteer to be participants may have
outcomes that are different from those of non-volunteers. Volunteers are usually healthier than
non-volunteers.
Q1
Are the individuals selected to participate in the study likely to be representative of

the target population?
The authors have done everything reasonably possible to ensure that the
Very likely
target population is represented.
Participants may not be representative if they are referred from a source within a Somewhat likely
target population even if it is in a systematic manner (eg. patients from a
teaching hospital for adults with asthma, only inner-city schools for
adolescent risk.
Participants are probably not representative if they are self-referred or are Not likely
volunteers (eg. volunteer patients from a teaching hospital for adults with
asthma, inner-city school children with parental consent for adolescent risk)
or if you can not tell.
61
Q2
What percentage of selected individuals agreed to participate?
The % of subjects in the control and intervention groups that
agreed to participate in the study before they were assigned to
intervention or control groups.
B)
There is no mention of how many individuals were

approached to participate.
Not Reported
The study was directed at a group of people in a specific

geographical area, city, province, broadcast audience, where
the denominator is not known, eg. mass media intervention.
Not Applicable
ALLOCATION BIAS
In this section, raters assess the likelihood of bias due to the allocation process in an experimental
study. For observational studies, raters assess the extent that assessments of exposure and
outcome are likely to be independent. Generally, the type of design is a good indicator of the
extent of bias. In stronger designs, an equivalent control group is present and the allocation
process is such that the investigators are unable to predict the sequence.
Q1 Indicate the study design
Investigators randomly allocate eligible people to an
intervention or control group.
RCT
Cohort (two group pre and post)

Groups are assembled according to whether or not exposure
to the intervention has occurred. Exposure to the intervention
may or may not be under the control of the investigators.
Study groups may not be equivalent or comparable on some
feature that affects the outcome.
Two-group
Quasi
Experimental
Before/After Study (one group pre + post)

The same group is pretested, given an intervention, and
tested immediately after the intervention. The intervention
group, by means of the pretest, act as their own control group.
Case-control,
Before/After
Study or No
Control Group
Case control study

A retrospective study design where the investigators gather
cases of people who already have the outcome of interest
and controls that do not. Both groups are then questioned or
their records examined about whether they received the
intervention exposure of interest.
No Control Group
Note: The following questions are not for rating but for additional statistics that can be incorporated in the
writing of the review.
62
(i)
If the study was reported as an RCT was the method of random allocation stated?
The method of allocation was stated.
YES
The method of allocation was not stated.
(ii)
Is the method of random allocation appropriate?

YES
The method of random allocation is appropriate if the
randomization sequence allows each study participant to have
the same chance of receiving each intervention and the
investigators could not predict which intervention was next. eg.
an open list of random numbers of assignments or coin toss
The method of random allocation is not entirely transparent,
eg. the method of randomization is described as alternation,
case record numbers, dates of birth, day of the week.
(iii)
NO
NO
Was the method of random allocation concealed?

The randomization allocation was concealed so that each study YES
participant had the same chance of receiving each intervention
and the investigators could not predict which group assignment
was next. Examples of appropriate approaches include assignment of
subjects by a central office
unaware of subject characteristics, or sequentially numbered, and sealed
in opaque envelopes.
The method of random allocation was not concealed or not
NO
reported as concealed.
C)
CONFOUNDERS
A confounder is a characteristic of study subjects that:
- is a risk factor (determinant) for the outcome to the putative cause, or
- is associated (in a statistical sense) with exposure to the putative cause
Note: Potential confounders should be discussed within the Review Group and decided a
priori.
Q1
Prior to the intervention were there differences for important confounders

reported in the paper
The authors reported that the groups were balanced at baseline
with respect to confounders (either in the text or a table)
The authors reported that the groups were not balanced at
baseline with respect to confounders.
Q2
Q3
Were the confounders adequately managed in the analysis?

Differences between groups for important confounders were
controlled in the design (by stratification or matching) or in the
No attempt was made to control for confounders.
Were there important confounders not reported?
describe
All confounders discussed within the Review Group were
reported.
NO
YES
YES
NO
YES
NO
63
D)
BLINDING
The purpose of blinding the outcome assessors (who might also be the care
providers) is to protect against detection bias.
Q1
Was (were) the outcome assessor(s) blinded to the intervention or exposure status of
participants?
Assessors were described as blinded to which participants were in the YES
control and intervention groups.
Assessors were able to determine what group the participants were in.
NO
The data was self-reported and was collected by way of a survey,

questionnaire or interview.
Not Applicable
It is not possible to determine if the assessors were blinded or not.
Not Reported
E) DATA COLLECTION METHODS

Some sources from which data may be collected are:
Self reported data includes data that is collected from participants in the study (eg.
completing a questionnaire, survey, answering questions during an interview, etc.).
Assessment/Screening includes objective data that is retrieved by the researchers.
(eg. observations by investigators).
Medical Records / Vital Statistics refers to the types of formal records used for the
extraction of the data.
Reliability and validity can be reported in the study or in a separate study. For example,
some standard assessment tools have known reliability and validity.
Q1
Were data collection tools shown or known to be valid for the outcome of
interest?
The tools are known or were shown to measure what they were intended YES
to measure.
There was no attempt to show that the tools measured what they were
intended to measure.
Q2
NO
Were data collection tools shown or known to be reliable for the outcome of
interest?
The tools are known or were shown to be consistent and accurate in
YES
measuring the outcome of interest (eg., test-retest, Cronbacks alpha,
interrater reliability).
There was no attempt to show that the tools were consistent
NO
and accurate in measuring the outcome of interest.
64
F) WITHDRAWALS AND DROP-OUTS

Q1
Indicate the percentage of participants completing the study.
The percentage of participants that completed the study.
Not Applicable
The study was directed at a group of people in a specific geographical
area, city, province, broadcast audience, where the percentage of
participants completing, withdrawing or dropping-out of the study is not
known, eg. mass media intervention.
The authors did not report on how many participants
Not Reported
completed, withdrew or dropped-out of the study.
G) ANALYSIS
If you have questions about analysis, contact your review group leader.
Q1.
The components of a recognized formula are present. Theres a citation for the formula
used.
Q2.
The appropriate statistically significant difference between groups needs to be

determined by the review group before the review begins.
Q3.
The review group leader needs to think about how much the study has violated the
underlying assumptions of parametric analysis?
Q5.
Whether intention to treat or reasonably high response rate (may need to clarify
within the review group).
H) INTERVENTION INTEGRITY
Q1
What percentage of participants received the allocated intervention or exposure of

interest?
The number of participants receiving the intended intervention is noted. Not Applicable
For example, the authors may have reported that at least 80% of the
participants received the complete intervention.
describe
Not Reported
describe
Q2
Q3
Not applicable
Was the consistency of the intervention measured?

The authors should describe a method of measuring if the intervention was
provided to all participants the same way.
describe
Yes
describe
No
describe
Not reported
Is it likely that subjects received an unintended intervention (contamination or

cointervention) that may influence the results?
65
The authors should indicate if subjects received an unintended intervention that may have
influenced the outcomes. For example, co-intervention occurs when the study group
receives an additional intervention (other than that intended). In this case, it is possible that
the effect of the intervention may be over-estimated. Contamination refers to situations
where the control group accidentally receives the study intervention. This could result in an
under-estimation of the impact of the intervention.
describe
Yes
describe
No
describe
Cant tell
Component Ratings for Study

A)
SELECTION BIAS
Strong
Q1 = Very Likely AND Q2 = 80-100% Agreement
OR
Q1 = Very Likely AND Q2 = Not Applicable
Moderate
Q1 = Very Likely AND Q2 = 60 - 79% Agreement
OR
Q1 = Very Likely AND Q2 = Not Reported
OR
Q1 = Somewhat Likely AND Q2 = 80-100%
OR
Q1 = Somewhat Likely AND Q2 = 60 - 79% Agreement
OR
Q1 = Somewhat Likely AND Q2 = Not Applicable
Weak
Q1 = Not Likely
OR
Q2 = Less than 60% agreement
OR
Q1 = Somewhat Likely AND Q2 = Not Reported
B)
ALLOCATION BIAS
Strong
Study Design = RCT
Moderate
Study Design = Two-Group Quasi-Experimental
Weak
Study Design = Case Control, Before/After Study, No Control Group
66
C)
CONFOUNDERS
Strong
Q1 = No
Q1 = Yes
AND Q2 = N/A
AND Q2 = YES
AND Q3 = No
AND Q3 = No
Q1 = Yes
AND Q2 = YES
AND Q3 = Yes
Q1 = Cant tell
Q1 = Yes
Q1 = Yes
Q1 = No
AND Q2 = No
AND Q2 = No
AND Q2 = N/A
AND Q3 = Yes
AND Q3 = No
AND Q3 = Yes
Moderate
Weak
D)
BLINDING
Strong
Q1=Yes
Weak
Q1=No
Q1= Not reported
Not applicable
E)
DATA COLLECTION METHODS

Strong
Q1 = Yes AND Q2 = Yes
Moderate
Q1 = Yes AND Q2 = No
Weak
Q1 = No AND Q2 = Yes
OR
Q1 = No AND Q2 = No
F)
WITHDRAWALS AND DROP-OUTS

Strong
Q1 = 80-100%
Moderate
Q1 = 60-79%
Weak
Q1 = Less than 60%
OR
Q1 = Not Reported Not Applicable
Not applicable
67
68
2)
QUALITATIVE STUDIES
Qualitativeresearchexploresthesubjectiveworld.Itattemptstounderstandwhypeoplebehavethe
waytheydoandwhatmeaningexperienceshaveforpeople.1
Qualitativeresearchmaybeincludedinareviewtoshedlightonwhethertheinterventionissuitable
for a specific target group, whether special circumstances have influenced the intervention, what
factorsmighthavecontributedifaninterventiondidnothavetheexpectedeffects,whatdifficulties
must be overcome if the study is to be generalised to other populations.2 These are all important
questionsoftenaskedbytheusersofsystematicreviews.
Reviewers may choose from a number of checklists available to assess the quality of qualitative
research.Sourcesofinformationonqualityappraisalinclude:
- CASPappraisaltoolforQualitativeResearchincludedinthismanual,
http://www.phru.nhs.uk/casp/qualitat.htm
- Spencer L, Ritchie J, Lewis J, Dillon L. Quality in Qualitative Evaluation: A framework for
assessingresearchevidence.GovernmentChiefSocialResearchersOffice.CrownCopyright,
2003.
www.strategy.gov.uk/files/pdf/Quality_framework.pdf
- HealthCarePracticeResearchandDevelopmentUnit(HCPRDU),UniversityofSalford,UK.
Evaluation
Tool
for
Qualitative
Studies,
http://www.fhsc.salford.ac.uk/hcprdu/tools/qualitative.htm
- Greenhalgh T, Taylor R. Papers that go beyond numbers (qualitative research). BMJ
1997;315:7403.
- Popay J, Rogers A, Williams G. Rationale and standards for the systematic review of
qualitativeliteratureinhealthservicesresearch.QualHealthRes1998;8:34151.
- MaysN,PopeC.Rigourandqualitativeresearch.BMJ1995;311:10912.
In relation to the appraisal of process evaluations the EPPICentre has developed a 12question
checklist,availableat:
http://eppi.ioe.ac.uk/EPPIWeb/home.aspx?page=/hp/reports/phase/phase_process.htm.
REFERENCES
1.
2.

Reviews
and
Dissemination,
University
of
York.
March
2001.
Hedin A, and Kallestal C. Knowledgebased public health work. Part 2: Handbook for
compilationofreviewsoninterventionsinthefieldofpublichealth.NationalInstituteofPublic
Health.2004.
ADDITIONALREADING
JonesR.Whydoqualitativeresearch?BMJ1995;311:2.
69
Pope C, Mays N. Qualitative Research: Reaching the parts other methods cannot reach: an
introductiontoqualitativemethodsinhealthandhealthservicesresearch.BMJ1995;311:4245.
70
Critical Appraisal Skills Programme (CASP)

making sense of evidence
10 questions to help you make sense of qualitative research
This assessment tool has been developed for those unfamiliar with qualitative research and its
theoretical perspectives. This tool presents a number of questions that deal very broadly with some
of the principles or assumptions that characterise qualitative research. It is not a definitive guide
and extensive further reading is recommended.
How to use this appraisal tool
Three broad issues need to be considered when appraising the report of qualitative research:
Rigour: has a thorough and appropriate approach been applied to key research methods
in the study?
Credibility: are the findings well presented and meaningful?
Relevance: how useful are the findings to you and your organisation?
The 10 questions on the following pages are designed to help you think about these issues
systematically.
The first two questions are screening questions and can be answered quickly. If the answer to both
is yes, it is worth proceeding with the remaining questions.
A number of italicised prompts are given after each question. These are designed to remind you
why the question is important. Record your reasons for your answers in the spaces provided.
The 10 questions have been developed by the national CASP collaboration for qualitative
methodologies.
Milton Keynes Primary Care Trust 2002. All rights reserved.
71
Screening Questions
1 Was there a clear statement of the aims
of the research?
Consider:
what the goal of the research was
why it is important
its relevance
Yes
No
2 Is a qualitative methodology appropriate?

Consider:
if the research seeks to interpret or illuminate
the actions and/or subjective experiences of
research participants
Yes
No
Is it worth continuing?
Detailed questions
Appropriate research design

3 Was the research design appropriate to
Write comments here
the aims of the research?
Consider:
if the researcher has justified the research design
(eg. have they discussed how they decided
which methods to use?)
Sampling
4 Was the recruitment strategy appropriate
Write comments here
to the aims of the research?
Consider:
if the researcher has explained how the
participants were selected
if they explained why the participants they
selected were the most appropriate to provide
access to the type of knowledge sought by the study
if there are any discussions around recruitment
(eg. why some people chose not to take part)
72

Data collection
5 Were the data collected in a way that
Write comments here
addressed the research issue?
Consider:
if the setting for data collection was justified
if it is clear how data were collected
(eg. focus group, semi-structured interview etc)
if the researcher has justified the methods chosen
if the researcher has made the methods explicit
(eg. for interview method, is there an indication of
how interviews were conducted, did they used a topic guide?)
if methods were modified during the study.
If so, has the researcher explained how and why?
if the form of data is clear (eg. tape recordings,
video material, notes etc)
if the researcher has discussed saturation of data
Reflexivity (research partnership relations/recognition of researcher bias)

6 Has the relationship between researcher and
Write comments here
participants been adequately considered?
Consider whether it is clear:
if the researcher critically examined their
own role, potential bias and influence during:
formulation of research questions
data collection, including sample
recruitment and choice of location
how the researcher responded to events
during the study and whether they considered
the implications of any changes in the research design
Ethical Issues
7 Have ethical issues been taken into
Write comments here
consideration?
Consider:
if there are sufficient details of how the research
was explained to participants for the reader to
assess whether ethical standards were maintained
if the researcher has discussed issues raised
by the study (e. g. issues around informed consent
or confidentiality or how they have handled the
effects of the study on the participants during
and after the study)
if approval has been sought from the ethics committee
73

Data analysis
8 Was the data analysis sufficiently rigorous?
Write comments here
Consider:
if there is an in-depth description of
the analysis process
if thematic analysis is used. If so, is it
clear how the categories/themes were
derived from the data?
whether the researcher explains how the
data presented were selected from the original
sample to demonstrate the analysis process
if sufficient data are presented to support the findings
to what extent contradictory data are taken into account
whether the researcher critically examined their
own role, potential bias and influence during analysis
and selection of data for presentation
Findings
9 Is there a clear statement of findings?
Write comments here
Consider:
if the findings are explicit
if there is adequate discussion of the evidence
both for and against the researchers arguments
if the researcher has discussed the credibility of
their findings
if the findings are discussed in relation to the original
research questions
Value of the research

10 How valuable is the research?
Write comments here
Consider:
if the researcher discusses the contribution
the study makes to existing knowledge or
understanding eg. do they consider the findings
in relation to current practice or policy, or relevant
research-based literature?
if they identify new areas where research is
necessary
if the researchers have discussed whether or how
the findings can be transferred to other populations
or considered other ways the research may be used
74
TheSchemaforEvaluatingEvidenceonPublicHealthInterventions
The Schema includes questions that encourage reviewers of evidence to consider whether the
evidencedemonstratesthataninterventionwasadequatelyimplementedintheevaluationsetting(s),
whether information is provided about the implementation context, and whether interactions that
occurbetweenpublichealthinterventionsandtheircontextwereassessedandreported.Itisusedto
appraiseindividualpapersandtoformulateasummarystatementaboutthosearticlesandreports.
TheSchemacanbedownloadedfrom:
http://www.nphp.gov.au/publications/phpractice/schemaV4.pdf.
75
AChecklistforEvaluatingEvidenceonPublicHealthInterventions
SECTION1:THESCOPEOFYOURREVIEW
Itemstorecordaboutthescopeofyourreview
1.
Whatisthequestionyouwanttoanswerinthereview?
2.
Howareyou(andpossiblyothers)goingtousethefindingsofthereview?
3.
Whoaskedforthereviewtobedone?
4.
Howhasthereviewbeenfunded?
5.
Whoisactuallycarryingoutthereview?
SECTION2:THEPAPERSINTHEREVIEW
2APublicationdetails
Identify the publication details for each paper or report to be appraised (eg title, authors, date, publication
information, type of article or report). Also note what related papers or reports have been published (eg
processevaluationsorinterimreports).
2BSpecifyingtheintervention
1.
Exactlywhatinterventionwasevaluatedinthestudy?
2.
Whatwastheoriginoftheintervention?
3.
Iftheoriginoftheinterventioninvolvedadegreeofformalplanning,whatwastherationaleforthestrategies
selected?
4.
Whatorganisationsorindividualssponsoredtheintervention(withfundingorinkindcontributions)?Where
relevant,givedetailsofthetypeofsponsorshipprovided.
2CIdentifyingtheinterventioncontext
5.
Whataspectsofthecontextinwhichtheinterventiontookplacewereidentifiedinthearticle?
6.
Wasenoughinformationprovidedinthearticletoenableyoutodescribetheinterventionanditscontextas
requestedabove?(Identifymajordeficiencies)
7.
How relevant to the scope of your review (as recorded in Section 1) are the intervention and the context
describedinthisarticle?
DecisionPoint
Ifyouconcludethatthearticleisrelevant(orpartlyrelevant)tothescopeofyourreview,gotosubsection2D.
Ifthearticleisnotrelevantrecordwhynot,andthenmoveonthenextpaperorreporttobeappraised.
2DTheevaluationcontextbackground,purposeandquestionsasked
8.
Whorequestedorcommissionedtheevaluationandwhy?
9.
Whatresearchquestionswereaskedintheevaluationreportedinthestudy?
10.
Whatmeasuresofeffectorinterventionoutcomeswereexamined?
11.
Whatwastheanticipatedsequenceofeventsbetweentheinterventionstrategiesandthemeasuresofeffector
intendedinterventionoutcomes?
12.
Werethemeasuresofeffectorinterventionoutcomesachievableandcompatiblewiththesequenceofevents
outlinedabove?
13.
Whatwasthetimingoftheevaluationinrelationtotheimplementationoftheintervention?
14.
Wastheinterventionadequatelyimplementedinthesettinginwhichitwasevaluated?
15.
Wastheinterventionreadyforthetypeofevaluationthatwasconducted?
16.
Werethemeasuresofeffectorinterventionoutcomesvalidatedorpilottested?Ifso,how?
17.
Didtheobservationsormeasuresincludetheimportantindividualandgroupleveleffects?
18.
Wasthereacapacitytoidentifyunplannedbenefitsandunanticipatedadverseeffects?
19.
Iftheresearchwasnotprimarilyaneconomicevaluation,wereeconomicfactorsconsidered?
20.
Wasthereasignificantpotentialforconflictofinterest(inthewaytheinterventionand/oritsevaluationwere
fundedandimplemented)thatmightaffectinterpretationofthefindings?
2EThemethodsusedtoevaluatetheintervention
21.
Whattypesofresearchmethodswereusedtoevaluatetheintervention?
22.
Whatstudydesignswereusedintheevaluation?
23.
Howappropriateweretheresearchmethodsandstudydesignsinrelationtothequestionsaskedinthestudy?
24.
Wastheevaluationconductedfromasingleperspectiveormultipleperspectives?Givedetails.
25.
Appraisetherigouroftheresearchmethodsusedinthestudyusingtherelevantcriticalappraisalchecklist(s)
(seeTable1)
26.
What are your conclusions about the adequacy of the design and conduct of the research methods used to
evaluatetheintervention?
27.
Arethereportedfindingsoftheevaluationlikelytobecredible?
DecisionPoint
76
IfyouconcludefromSection2thatthereportedfindingsarelikelytobecrediblegotoSection3.Ifthefindingsare
unlikelytobecrediblegotoSection4toanswerquestion2only,andthenmovetothenextpapertobeappraised.
SECTION3:DESCRIBINGTHERESULTSFROMTHEPAPERSSELECTED
Thestudyfindings
1.
Whatfindingswerereportedinthestudy?
2.
Ifthestudyspecifiedmeasurableorquantifiabletargets,didtheinterventionachievetheseobjectives?
3.
Werereportedinterventioneffectsexaminedamongsubgroupsofthetargetpopulation?
4.
Shouldanyotherimportantsubgroupeffectshavebeenconsideredthatwerenotconsidered?
5.
Wastheinfluenceoftheinterventioncontextontheeffectivenessoftheinterventioninvestigatedinthestudy?
6.
Howdependentonthecontextistheinterventiondescribedinthearticle?
7.
Weretheinterventionoutcomessustainable?
8.
Didthestudyexamineandreportonthevalueofthemeasuredeffectstopartiesinterestedinoraffectedby
them?
SECTION4:INTERPRETINGEACHARTICLE
Yourinterpretations
1.
Howwelldidthestudyansweryourreviewquestion(s)?Givedetails.
2.
Arethereotherlessonstobelearnedfromthisstudy(eglessonsforfutureevaluations)
DecisionPoint
Ifyouareconductingthereviewforthepurposeofmakingrecommendationsforaparticularpolicyorpracticesetting,
continueinSection4toanswerquestions38.OtherwisemoveontoSection5.
3.
Aretheessentialcomponentsoftheinterventionanditsimplementationdescribedwithsufficientdetailand
precisiontobereproducible?
4.
Istheinterventioncontext,asdescribedinthearticlebeingexamined,comparabletotheinterventioncontext
thatisbeingconsideredforfutureimplementationoftheintervention?
5.
Arethecharacteristicsofthetargetgroupstudiedinthearticlecomparabletothetargetgroupforwhomthe
interventionisbeingconsidered?
6.
Ifaneconomicevaluationwasconducted,didthepaperorreportincludeandaddressthedetailsrequiredin
order to make an informed assessment about the applicability and transferability of the findings to other
settings?
7.
Ifenoughinformationwasprovided,arethefindingsoftheeconomicevaluationrelevantandtransferableto
yoursetting?
8.
Aretheeffectsoftheinterventionlikelytobeconsideredimportantinyoursetting?
SECTION5:SUMMARISINGTHEBODYOFEVIDENCE
5AGrouping,ratingandweighingupthepapersandreports(seeTable2forexampleofpresentingfindings)
1.
Grouparticleswithsimilarresearchquestionsandsimilarinterventionstrategies.Witheachgroup,complete
thefollowing:
2.
Ratethequalityofeachstudy,from1(weak)to3(strong).
3.
Assesstheconsistencyofthefindingsamongthestrongerstudies,from1(inconsistent)to3(consistent).
4.
Determine the degree to which the stronger studies with consistent findings are applicable to your review
context.
5AFormulatingasummarystatement
5.
Didstudiesthatexaminedsimilarinterventionstrategies,withsimilarresearchquestions,produceconsistent
results?
6.
Didstudieswithdifferentresearchquestionsproducecompatibleresults?
7.
Overall,whatdoesthebodyofevidencetellyouabouttheintervention?
8.
Arethereimportantgapsintheevidence?Ifso,whatarethey?
9.
Towhatdegreearethereviewfindingsusefulforyourpurposes,asidentifiedinSection1?
10.
Whatareyourrecommendationsbasedonthisreview?
77
78
Unit Nine: Synthesising the Evidence

LearningObjectives
Tounderstandthedifferentmethodsavailableforsynthesisingevidence
To understand the terms: metaanalysis, confidence interval, heterogeneity, odds ratio, relative risk,
narrativesynthesis
Generally,therearetwoapproachestosynthesisingthefindingsfromarangeofstudies:
Narrativesynthesisfindingsaresummarisedandexplainedinwords
Quantitative/statisticalsynthesisdatafromindividualstudiesarecombinedstatisticallyandthen
(metaanalysis)
summarised
The Cochrane Reviewers Handbook1 suggests the following framework for synthesis of primary
studies(regardlessofthemethod(narrative/metaanalysis)usedtosynthesisedata):
Whatisthedirectionoftheeffect?
Whatisthesizeoftheeffect?
Istheeffectconsistentacrossstudies?
Whatisthestrengthofevidencefortheeffect?
Before deciding which synthesis approach to use it is important to tabulate the findings from the
studies. This aids the reviewer in assessing whether studies are likely to be homogenous or
heterogenous, and tables greatly assist the reader in eyeballing the types of studies that were
included in the review. Reviewers should determine which information should be tabulated; some
examplesareprovidedbelow:
Authors
Year
Interventiondetails
Comparisondetails
Theoreticalbasis
Studydesign
Qualityassessment
Outcomes
Setting/context(incl.country)
Populationcharacteristics
Example:Anexampleoftabulatingstudiescanbefoundinthefollowingsystematicreview:
DiCensoA,Guyatt G,Willan A, Griffith L. Interventions to reduce unintended pregnancies among
adolescents:systematicreviewofrandomisedcontrolledtrials.BMJ.2002Jun15;324(7351):1426.
Thechoiceofanalysisusuallydependsonthediversityofstudiesincludedinthereview.Diversityof
studiesisoftenreferredtoasheterogeneity.Becausesomereviewsmayincludestudiesthatdifferin
such characteristicsasdesign, methods, or outcomemeasures,aquantitative synthesis of studiesis
notalwaysappropriateormeaningful.
Isthereheterogeneity?
No
Yes
Metaanalysis
NarrativesynthesisDealwithheterogeneity
(eg.subgroupanalyses)
79

Where studies are more homogenous, i.e., we can compare like with like, it may be appropriate to
combinetheindividualresultsusingametaanalysis.Iftheresultsaresimilarfromstudytostudywe
can feel more comfortable that a metaanalysis is warranted. Heterogeneity can be determined by
presenting the results graphically and examining the overlap of confidence intervals (CI) (if CI
overlap studies are more likely to be homogenous) and by calculating a statistical measure of
heterogeneity. Both of these methods are further outlined in Chapter Eight of the Cochrane
ReviewersHandbook(Analysingandpresentingresults).
Metaanalysis produces a weighted summary result (more weight given to larger studies). By
combining results from more than one study it has the advantage of increasing statistical power
(whichis often inadequate instudieswith asmallsample size). The final estimateisusually in the
formofanoddsratio:theratiooftheprobabilityofaneventhappeningtothatofitnothappening.
Theoddsratioisoftenexpressedtogetherwithaconfidenceinterval(CI).Aconfidenceintervalisa
statementoftherangewithinwhichthetrueoddsratiolieswithinagivendegreeofassurance(eg.
usuallyestimatesofeffectlikeoddsratiosarepresentedwitha95%confidenceinterval).
Guidelinesfornarrativesynthesisarenotyetavailable,althoughresearchiscurrentlyunderwayto
developguidelinesforsystematicreviews.Ideally,thereviewershould2:
Describestudies
Assesswhetherqualityisadequateinprimarystudiestotrusttheresults
Demonstrateabsenceofdataforplannedcomparisons
Demonstratedegreeofheterogeneity
Stratify results by populations, interventions, settings, context, outcomes, validity (if
appropriate)
Example: A number of Cochrane systematic reviews of health promotion and public health topics
synthesisetheresultsnarratively.VisitTheCochraneLibrarytoreadexamples.Anotherexamplecan
befoundinthefollowingarticle:RiemsmaRB,PattendenJ,BridleC,SowdenAJ,MatherL,WattIS,
Walker A. Systematic review of the effectiveness of stage based interventions to promote smoking
cessation.BMJ2003;326:117577.
Integratingqualitativeandquantitativedata
TheEvidenceforPolicyandPracticeInformationandCoordinatingCentrehasdevelopedmethods
for synthesising the findings from diverse types of studies within one review3. These methods
involveconductingthreetypesofsynthesesinthesamereview:1)astatisticalmetaanalysistopool
trialsofinterventionstacklingparticularproblems(oranarrativesynthesiswhenmetaanalysisisnot
appropriateorpossible);2)asynthesisofstudiesexaminingpeoplesperspectivesorexperiencesof
thatproblemusingqualitativeanalysis(viewsstudies);and3)amixedmethodssynthesisbringing
the products of 1) and 2) together. These developments have been driven by particular review
questions rather than methodology; users of the reviews want to know about the effects of
interventions, but also want to know which interventions will be most appropriate and relevant to
people. However, they do illustrate how qualitative studies can be integrated into a systematic
reviewasviewsstudiesareoften,butnotalways,qualitativeinnature.Themethodsforeachofthe
threesynthesesaredescribedinbriefbelow:
Synthesis1)Effectivenesssynthesisfortrials
Effectsizesfromgoodqualitytrialsareextractedand,ifappropriate,pooledusingstatisticalmeta
analysis. Heterogeneity is explored statistically by carrying out subgroup analyses on a range of
categoriesspecifiedinadvance(eg.studyquality,studydesign,settingandtypeofintervention).
80
Synthesis2)Qualitativesynthesisforviewsstudies
Thetextualdatadescribingthefindingsfromviewsstudiesarecopiedverbatimandenteredintoa
softwarepackagetoaidqualitativeanalysis.Twoormorereviewersundertakeathematicanalysison
thisdata.Themesaredescriptiveandstayclosetothedata,buildingupapictureoftherangeand
depth of peoples perspectives and experiences in relation to the health issue under study. The
contentofthedescriptivethemesarethenconsideredinthelightoftherelevantreviewquestion(eg.
whathelpsandwhatstopschildreneatingfruitandvegetables?)inordertogenerateimplicationsfor
interventiondevelopment.Theproductsofthiskindofsynthesiscanbeconceptualisedastheories
aboutwhichinterventionsmightwork.Thesetheoriesaregroundedinpeoplesownunderstandings
abouttheirlivesandhealth.Thesesynthesismethodshavemuchincommonwiththeworkofothers
whohaveemphasisedthetheorybuildingpotentialofsynthesis.4
Synthesis3)Amixedmethodssynthesis
Implicationsforinterventionsarejuxtaposedagainsttheinterventionswhichhavebeenevaluatedby
trialsincludedinSynthesis1.Usingthedescriptionsoftheinterventionsprovidedinthereportsof
the trials, matches, missmatches and gaps are identified. Gaps are used for recommending what
kinds of interventions need to be newly developed and tested. The effect sizes from interventions
whichmatchedimplicationsforinterventionsderivedfrompeoplesviewscanbecomparedtothose
which do not, using subgroup analysis. This provides a way to highlight which types of
interventions are both effective and appropriate. Unlike Bayesian methods, another approach to
combiningqualitativeandquantitativestudieswithinsystematicreviewswhichtranslatestextual
dataintonumericaldata,thesemethodsintegratequantitativeestimatesofbenefitandharmwith
qualitativeunderstandingfrompeopleslives,whilstpreservingtheuniquecontributionofeach.3
REFERENCES
1.
2.
3.
Reviews
and
Dissemination,
University
of
York.
March
2001.
ThomasJ,HardenA,OakleyA,OliverS,SutcliffeK,ReesR,BruntonG,KavanaghJ.Integrating
qualitativeresearchwithtrialsinsystematicreviews.BMJ2004;328:10102.
4. Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A. Applying systematic

reviewmethodstostudiesofpeoplesviews:anexamplefrompublichealthresearch.JEpidemiol
CommunityHealth.2004Sep;58(9):794800.
81
82
Unit Ten: Interpretation of Results

LearningObjectives
To be able to interpret the results from studies in order to formulate conclusions and recommendations
fromthebodyofevidence
To understand the factors that impact on the effectiveness of public health and health promotion
interventions
The following issues should be included in the discussion and recommendations section of a
systematicreviewofahealthpromotionorpublichealthintervention:
1) Strengthoftheevidence
2) Integrityofinterventiononhealthrelatedoutcomes
3) Theoreticalexplanationsofeffectiveness
4) Contextasaneffectmodifier
5) Sustainabilityofinterventionsandoutcomes
6) Applicability
7) Tradeoffsbetweenbenefitsandharms
8) Implicationsforpracticeandfuturehealthpromotionandpublichealthresearch
As those who read systematic reviews (eg. policy makers) may not have time to read the whole
review it is important that the conclusions and recommendations are clearly worded and arise
directlyfromthefindingsofthereview.1
1) Strengthoftheevidence
The discussion should describe the overall strength of the evidence, including the quality of the
evidenceandthesizeandconsistencyoftheresults.Thesizeoftheresultsisparticularlyimportant
in populationbased studies, where a small effect at the community level may have a much more
practical significance than the effect of comparable size at the individual level.2 Using statistical
significancealoneasthestandardforinterpretationoftheresultsofcommunityinterventiontrialsis
inappropriateforresearchatthepopulationlevel.3
This section of the review should also describe the biases or limitations of the review process.
Difficultiesinlocatinghealthpromotion/publichealthliteraturemayhaveresultedintheinabilityto
carryoutacomprehensivesearch.Formanyreviewers,afurtherlimitationofthereviewprocessis
the inability to translate nonEnglish articles, or search nonEnglish electronic databases.
Furthermore,interpretationsmaybelimitedduetostudiesmissingimportantinformationrelatingto
such factors as the implementation of the intervention, context, and methodological features (eg.
blinding,datacollectiontools,etc)requiredinordertodeterminestudyquality.
2) Interventionintegrity
Reviewers should discuss whether the studies included in the review illuminated the key process
factorsthatledtoeffectiveinterventions.Inaddition,therelationshipbetweeninterventionintegrity
and effectiveness should be described, i.e., did studies that address integrity thoroughly show a
greaterimpact?
Animportantoutcomeofprocessevaluationistheassessmentofinterventiondose,ortheamount
ofinterventiondeliveredandreceivedbyparticipantsorthetargetgroup.3 Interventiondosevaries
markedly between community level interventions, and may be one of the factors that explain
differencesineffectivenessbetweenstudies.Investigatorshavepostulatedthatthesmalleffectsizes
83
resulting from some community interventions is a result of an insufficient intervention dose or

intensity,orbecauseparticipationratesweretoolow.3Oralternatively,thedoseoftheintervention
may have been inadequate relative to other forces in the environment, such as an information
environment already saturated with sophisticated advertisements and product promotions.3
Mittlemark and colleagues4 have suggested that intervention effectiveness has been limited by the
length of the intervention, recommending that for communitybased interventions the intervention
periodbeatleastfiveyears,giventhetimeittypicallytakesforthecommunitytobemobilisedinto
action.Thisisbecauseitmaynotberealistictoexpectlargeindividualchangesinlifetimehabitsto
occur with complex behaviours, such as eating patterns, within the timeframe of most community
studies.4 Mittlemark et al4 further suggest that at the organisational or community level, additional
time must be built in for institutionalisation; that is, the continuing process of building local,
regional,andnationalcapacitytomountpermanenthealthpromotionprograms.
Informationisalsoneededinreviewsonwhetheritismoreeffectivetospreadagivendoseoutover
an extended period of time, rather than to compress it into a shorter time frame to maximise the
populationsfocusontheinterventionmessages.
3) Theoreticalexplanationsofeffectiveness
Althoughmanypublichealthinterventionsareplannedandimplementedwithoutexplicitreference
to theory, there is substantial evidence from the literature to suggest that the use of theory will
significantlyimprovethechancesofeffectiveness.5
Typesoftheories:
Theoriesthatexplainhealthbehaviourandhealthbehaviourchangeattheindividuallevel(eg.
Healthbeliefmodel,StagesofChange)
Theories that explain change in communities and communal action for health (eg. Diffusion of
Innovation)
Theoriesthatguidetheuseofcommunicationstrategiesforchangetopromotehealth(eg.social
marketing,communicationbehaviourchangemodel)
Models that explain changes in organisations and the creation of healthsupportive
organisationalpractices(eg.theoriesoforganisationalchange)
Modelsthatexplainthedevelopmentandimplementationofhealthpublicpolicy(eg.evidence
basedpolicymakingtopromotehealth)
Depending on the level of intervention (individual, group, or organisation) or the type of change
(simple, oneoff behaviour, complex behaviour, organisational or policy change), different theories
willhavegreaterrelevance.5
Reviewersshouldseektoexaminetheimpactofthetheoreticalframeworkontheeffectivenessofthe
intervention.Theassessmentoftheorywithinsystematicreviews5:
- helps to explain success or failure in different interventions, by highlighting the possible
impact of differences between what was planned and what actually happened in the
implementationoftheprogram
- assists in identifying the key elements or components of an intervention, aiding the
disseminationofsuccessfulinterventions.
Theory may also provide a valuable framework within which to explore the relationship between
findings from different studies. For example, when combining the findings from different studies,
reviewers can group interventions by their theoretical basis. Alternatively, reviewers may consider
grouping interventions depending of whether they seek to influence individual behaviour,
interpersonalrelationships,orcommunityorstructuralfactorsorwhethertheyusedaProgramLogic
orProgramTheoryapproach.
84

Systematicreviewswouldalsobegreatlyenhancedifinthediscussionattentionwaspaidtothegaps
intheoreticalcoverageofinterventions.Forexample,manyinterventionsseektofocusonsinglelevel
changesratherthanseekingtochangetheenvironmentwithinwhichpeoplemaketheirchoices.
4) Contextasaneffectmodifier
Interventionswhichareeffectivemaybeeffectiveduetopreexistingfactorsofthecontextintowhich
theinterventionwasintroduced.
Where information is available, reviewers should report on the presence of contextrelated

information6:
social and political factors surrounding the intervention, eg. local/national policy
environment,concurrentsocialchanges
timeandplaceofintervention
structural,organisational,physicalenvironment
aspectsofthehostorganisationandstaff,eg,number,experience/training,morale,expertise
ofstaff,competingprioritiestothestaffsattention,theorganisationshistoryofinnovation,
size of the organisation, the status of the program in the organisation, the resources made
availabletotheprogram;
aspectsofthesystem,eg,paymentandfeestructuresforservices,rewardstructures,degrees
ofspecialisationinservicedelivery;and
characteristicsofthetargetpopulation(eg.cultural,socioeconomic,placeofresidence).
Theboundarybetweentheparticularinterventionanditscontextisnotalwayseasytoidentify,and
seemingly similar interventions can have a different effect depending on the context in which it is
implemented.
5) Sustainabilityofinterventionsandoutcomes
The extent to which the intended outcomes or interventions are sustained should be an important
considerationinsystematicreviews,asdecisionmakersandfundersbecomeincreasinglyconcerned
withallocatingscarceresourceseffectivelyandefficiently.7
It is believed that interventions which isolate individual action from its social context would be
unlikely to produce sustainable health gain in the absence of change to the organisational,
communityandinstitutionalconditionsthatmakeupthesocialcontext.7
Reviewers may choose from a number of frameworks which describe the factors that determine
sustainability810
- Bossert8 suggests that both contextual (eg. political, social, economic and organisational)
factors and project characteristics (eg. institution management, content, community
participation)arerelatedtosustainability.
- Swerissen and Crisp9 propose that the relationship between the intervention level
(individual,organisational,community,institutional)andstrategies(eg.education,policies,
social planning, social advocacy) indicates the likely sustainability of programmes and
effects.
- A framework outlining the four integrated components of sustainability has also been
produced.10
6) Applicability
Applicability is a key part of the process of summarising evidence, since the goal of systematic
reviewsistorecommendinterventionsthatarelikelytobeeffectiveindifferentsettings.
85

Reviewers should use the REAIM model11 (Reach, Efficacy, Adoption, Implementation, and
Maintenance) for conceptualising the potential for translation and the public health impact of an
intervention.TheusercanthencomparetheirsituationtotheREAIMprofileoftheincludedstudies
orthebodyofevidence.
REAIM:
Reachtheabsolutenumber,proportion,andrepresentativenessofindividuals(characteristicsthat
reflect the target populations characteristics) who are willing to participate in a given initiative,
intervention,orprogram.Individuallevelsofimpact.
Efficacy/Effectiveness the impact of the intervention on important outcomes, including potential

negativeeffects,qualityoflife,andeconomicoutcomes.Individuallevelsofimpact.
Adoption the absolute number, proportion, and representativeness of settings and intervention
agents(peoplewhodelivertheprogram)whoarewillingtoinitiateaprogram.Comparisonsshould
be made on basic information such as resource availability, setting size and location, and
interventionistexpertise.Organisationallevelsofimpact.
Implementation at the setting level, implementation refers to the intervention agents integrity to
thevariouselementsofaninterventionsprotocol,includingconsistencyofdeliveryasintendedand
thetimeandcostoftheintervention.Attheindividuallevel,implementationreferstoclientsuseof
theinterventionstrategies.Organisationallevelsofimpact.
Maintenance The extent to which a program or policy becomes institutionalised or part of the
routineorganisationalpracticesandpolicies.Attheindividuallevel,itreferstothelongtermeffects
of a program on outcomes after 6 or more months after the most recent intervention contact. Both
individualandorganisationallevelsofimpact.
Exampletakenfromwww.reaim.org
Aschoolbasedinterventionthathasalargeimpactintermsofreachandefficacyattheindividual
level but is only adopted, implemented and maintained at a small number of organisations (with
specificresourcesthatarenotavailableintypicalrealworldschools)couldpotentiallybedescribed
asaninterventionthathasalargepotentialforimpact(iftheREAIMmodelwasnotused).Inreality,
when considering organisationallevel impact, in addition to individual level impact, this
interventionwouldhavelittlehopeofresultinginalargepublichealthimpactbecauseitcouldnot
beadopted,implementedandmaintainedinrealworldsettings.
Thisisalsotrueoftheconversesituationwhereaninterventionhassystemicorganisationaladoption,
implementation,andmaintenance,butlittlereach,efficacyormaintenanceattheindividuallevel.So
ifonlyonelevelwasassessed(i.e.theorganisationallevel)theimpactoftheinterventionwouldbe
consideredlargeeventhoughthereisnoindividuallevelreach,efficacyormaintenance.
CasestudyTheVictoriaCouncilonFitnessandGeneralHealthInc.(VICFIT)
VICFITwasestablishedthroughtheMinistersforSportandRecreationandHealthtoprovideadvice
togovernmentandtocoordinatethepromotionoffitnessinVictoria.OneofVICFITsinitiatives,the
Active Script Program (ASP), was designed to enable all general practitioners in Victoria to give
consistent, effective and appropriate physical activity advice in their particular communities. The
evaluationoftheinitiativeutilisedtheREAIMframework,whichisavailableat
http://www.vicfit.com.au/activescript/DocLib/Pub/DocLibAll.asp.
86
Reviewers should describe the body of evidence with respect to the main domains relevant to the
applicabilityofpublichealthandhealthpromotioninterventionstotheusersneedsseeTableTwo.
TableTwo.Evaluationoftheapplicabilityofanindividualstudyorabodyofevidence
REAIM
evaluationfactor
Reach
Domain
Characteristic
Sample
Samplingframe
Samplingmethod
Population
Age
Efficacy
Internal
validity
Outcomes
Datatobecollectedfrom
thestudy*
Howwellthestudy
populationresemblesthe
targetpopulationthe
authorsindicatetheywould
liketoexamine
Inclusionandexclusion
criteria
Participationrate
Therepresentativenessof
thestudypopulationtothe
targetpopulation,eg.,
volunteers,
provider/researcher
selected,randomsample
Characteristicsofthenon
participants
Ageofthepopulation
Applicabilitytotheusers
needs*
Doesthestudypopulation
resemblethatoftheuserswith
respecttorelevant
characteristics,eg.,diseaserisk
factors?
Ifthestudypopulationwas
selected(i.e.notarandom
samplewithahighparticipation
rate),howmighttheusers
populationdiffer?Mightthey
belessreceptivetothe
intervention?
Whatageofpopulationdothe
datalikelyapplyto,andhow
doesthisrelatetotheusers
needs?
Sex
Percentageofeachsexinthe Whatsexdothedatalikely
population
applyto,andhowdoesthis
relatetotheusersneeds?
Race/ethnicity
Race/ethnicitiesare
Arethedatalikelyspecifictoa
specificracial/ethnicgroup,or
representedinthestudy
population
aretheyapplicabletoother
groups?
Healthstatusand Percentageofthe
Howdoesthebaselinehealth
statusoftheuserspopulation
baselinerisk
populationaffectedat
baselinebydiseasesorrisk comparetothatofthestudy
factors
population?
Other
Otherpopulation
Arethereotherpopulation
characteristicsthatare
characteristicsthatarerelevant
relevanttooutcomesofthis tooutcomesofthisintervention?
intervention
Internalvalidity
Assessinternalvalidityfor Canthestudyresultsbe
thestudy
attributedtotheinterventionor
arethereimportantpotential
confounders?
Processand
Process(eg.,numberof
Aretheoutcomesexaminedin
intermediate
telephonecallstoclients)
thestudyrelevanttoyour
outcomes
andintermediateoutcomes population?Arethelinkages
(eg.,dietarychange)
betweenmoreproximal
examinedinthestudy
(intermediateandprocess)
outcomesbasedonsufficient
evidencetobeusefulinthe
currentsituation?
Distalhealthand Healthandqualityoflife
Aretheoutcomesexaminedin
qualityoflife
outcomesexaminedinthe thestudyrelevanttousers
outcomes
study
population?
87
Adoption
Implementation
Maintenance
Economic
efficiency
Economicoutcomes:cost,
costeffectiveness,cost
benefit,orcostutility
Iseconomicefficiencypartofthe
decisionmakingprocess?Ifso,
arethedataoncostoreconomic
efficiencyrelevanttotheusers
situation?
Harms
Anyharmsfromthe
Aretheseharmsrelevanttothe
interventionthatare
userspopulation?
presentedinthedata
Arethereotherpotentialharms?
Howistheuserbalancing
potentialbenefitswithpotential
harms?
Intervention Provider
Whodeliveredthe
Arethedescribedinterventions
intervention
reproducibleinthesituation
Trainingandexperienceof underconsideration?Isthe
theinterventionists
providerexpertiseandtraining
Iftheinterventionis
available?
deliveredbyateam,
indicateitsmembersand
theirspecifictasks
Contacts
Numberofcontactsmade Isthefrequencyofcontactsin
betweentheprovidersand thestudyfeasibleinthecurrent
eachparticipant
situation?
Durationofeachcontact
Medium
Mediumbywhichthe
Isthismediumfeasibleinthe
interventionwasdelivered: userssituation?
inperson,telephone,
electronic,mail
Presentation
Toindividualsorgroups
Isthisformatfeasibleinthe
currentsituation?
format
Withfamilyorfriends
present
Content
Basedonexistingtoolsand Isthisfeasibleinthecurrent
materialsordevelopedde situation?
novo
Tailoringoftheintervention
toindividualsorsubgroups
Setting
Infrastructureof Organisationalorlocal
Istheneededinfrastructure
thehealthcare
infrastructurefor
presentinthecurrentsituation?
deliverysystemor implementingthe
thecommunity
intervention
Accesstothe
Accesstotheintervention Doesthecurrentsituation
intervention
amongthetarget
providetheresourcestoensure
population
accesstotheintervention?
Individual
Adherence
Individualrateofadherence Aretherebarrierstoadherence
totheintervention
inthecurrentsituation?Are
level
Attritionratefromthe
theirlocalfactorsthatmight
program
influencetheattritionrate?
Programlevel Integrity
Theextenttowhichthe
Aretherebarriersto
interventiondeliveredas
implementationinthecurrent
planned
situation?
Whatistherelativeimportance
Individual
Sustainabilityof
Changeinbehaviouror
level
outcomes
otherimportantoutcomesin ofshortversuslongterm
thelongterm
outcomestotheuser?
Programlevel Sustainabilityof
Facetsoftheintervention
Istheinterventionfeasibleinthe
theintervention
thatweresustainableinthe longtermintheuserssetting?
longterm
Doesthenecessary
Infrastructurethat
infrastructureexist?Arethere
supportedasustained
availableresources?What
88
intervention
barrierstosustainabilitymight
Barrierstolongtermuseof beanticipated?
theintervention
*Datatobecollectedandapplicabilitycanbeappliedtotheindividualstudyortothebodyof
evidence
7) Tradeoffsbetweenbenefitsandharms
Reviewersshoulddiscusswhethertherewereanyadverseeffectsoftheinterventions,ordescribeif
there were certain groups that received more/less benefit from the interventions (differential
effectiveness).Ifcostdataisprovidedfortheinterventionsstudiesthisshouldalsobereported.
8) Implicationsforpracticeandfuturehealthpromotionandpublichealthresearch
Publichealthandhealthpromotionreviewersareinanidealpositiontodeterminetheimplications
for practice and future research to be conducted to address any gaps in the evidence base. For
example, where evidence is shown to be lacking, reviewers should clearly describe the type of
research required, including the study design, participants, intervention details and contexts and
settings.Ifthereviewedevidencebaseisflawedduetoparticularmethodologicalissues(eg.outcome
assessmenttools,allocationbias,etc)thesequalityissuescanbeaddressedinfuturestudies.
REFERENCES
1.
2.
3.
4.
5.
6.
7.
8.

Reviews
and
Dissemination,
University
of
York.
March
2001.
Donner A, Klar N. Pitfalls of and controversies in cluster randomization trials. Am J Public
Health.2004Mar;94(3):41622.
Sorensen G, Emmons K, Hunt MK, Johnston D. Implications of the results of community
interventiontrials.AnnuRevPublicHealth.1998;19:379416.
MittelmarkMB,HuntMK,HeathGW,SchmidTL.Realisticoutcomes:lessonsfromcommunity
based research and demonstration programs for the prevention of cardiovascular diseases. J
PublicHealthPolicy.1993Winter;14(4):43762.
Nutbeam D, Harris E. Theory in a Nutshell. A practical guide to health promotion theories.
McGrawHillAustraliaPtyLtd,2004.
Hawe P, Shiell A, Riley T, Gold L. Methods for exploring implementation variation and local
context within a cluster randomised community intervention trial. J Epidemiol Community
Health.2004Sep;58(9):78893.
ShediacRizkallah MC, Bone LR. Planning for the sustainability of communitybased health
programs:conceptualframeworksandfuturedirectionsforresearch,practiceandpolicy.Health
EducRes1998;13:87108.
Bossert TJ. Can they get along without us? Sustainability of donorsupported health projects in
CentralAmericaandAfrica.SocSciMed1990;30:101523.
89
9.
SwerrissenH,CrispBR.Thesustainabilityofhealthpromotioninterventionsfordifferentlevels
ofsocialorganization.HealthPromotInt2004;19:12330.
10. The Health Communication Unit. Overview of Sustainability, University of Toronto, Centre for
Health
Promotion,
2001.
Available
from:
http://www.thcu.ca/infoandresources/publications/SUS%20Master%20Wkbk%20and%20Wkshts
%20v8.2%2004.31.01_formatAug03.pdf
11. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion
interventions:theREAIMframework.AmJPublicHealth1999;89:13227.
ADDITIONALREADING
RychetnikL,FrommerMS.SchemaforEvaluatingEvidenceonPublicHealthInterventions;Version
4.NationalPublicHealthPartnership,Melbourne2002.
Visithttp://www.reaim.orgforinformationrelatingtogeneralisingtheresultsfromprimarystudies.
Glasgow RE, Lichtenstein E, Marcus AC. Why dont we see more translation of health promotion
research to practice? Rethinking the efficacytoeffectiveness transition. Am J Public Health. 2003
Aug;93(8):12617.
Dzewaltowski DA, Estabrooks PA, Klesges LM, Bull S, Glasgow RE. Behavior change intervention
research in community settings: how generalizable are the results? Health Promot Int. 2004
Jun;19(2):23545.
90
Unit Eleven: Writing the Systematic

Review
LearningObjectives
Tounderstandtherequirementstopublishasystematicreview
Tobefamiliarwiththecriteriathatwillbeusedtojudgedthequalityofasystematicreview
Whenothersreadyourreviewtheywillbeassessingitforthesystematicmannerinwhichbiaswas
reduced. A useful tool to assess the quality of a systematic review is produced by the Critical
Appraisal
Skills
Programme
(CASP)
and
can
be
found
at
http://www.phru.nhs.uk/~casp/appraisa.htm(providedoverleaf).Itisusefultokeepthistoolinmind
whenwritingthefinalreview.
Reviewersmayconsidersubmittingtheirreviewto:
1) TheCochraneCollaborationmustgothroughtheCochraneeditorialprocess
2) The Database of Abstracts of Reviews of Effects (DARE) this database is held by the
UniversityofYorkhttp://www.york.ac.uk/inst/crd/crddatabases.htm
3) TheEvidenceforPolicyandPracticeInformationandCoordinatingCentre(EPPICentre)to
be included in The Database of Promoting Health Effectiveness Reviews (DoPHER)
http://eppi.ioe.ac.uk
4) Apublishedjournalrelevanttothetopicofthereview.
Two sets of guidelines are available for reviewers wishing to submit the review to a published
journal.Reviewersshouldreadtheguidelinesrelevanttothestudydesignsincludedinthereview:
1)SystematicreviewsofRCTs:
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of
metaanalyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of
Metaanalyses.Lancet.1999Nov27;354(9193):1896900.
2)Systematicreviewsofobservationalstudies:
StroupDF,BerlinJA,MortonSC,OlkinI,WilliamsonGD,RennieD,MoherD,BeckerBJ,SipeTA,
ThackerSB.Metaanalysisofobservationalstudiesinepidemiology:aproposalforreporting.Meta
analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000 Apr
19;283(15):200812.
ADDITIONALREADING
OxmanAD,CookDJ,GuyattGHfortheEvidenceBasedMedicineWorkingGroup.Usersguideto
themedicalliterature.VI.Howtouseanoverview.EvidencebasedMedicineWorkingGroup.JAMA
1994;272:136771.
91
Critical Appraisal Skills Programme (CASP)

making sense of evidence
10 questions to help you make sense of reviews

How to use this appraisal tool
Three broad issues need to be considered when appraising the report of a systematic
review:
Is the study valid?
What are the results?
Will the results help locally?
The 10 questions on the following pages are designed to help you think about these issues
systematically.
The first two questions are screening questions and can be answered quickly. If the answer
to both is yes, it is worth proceeding with the remaining questions.
You are asked to record a yes, no or cant tell to most of the questions. A number of
italicised prompts are given after each question. These are designed to remind you why
the question is important. Record your reasons for your answers in the spaces provided.
The 10 questions are adapted from Oxman AD, Cook DJ, Guyatt GH, Users guides to the
medical literature. VI. How to use an overview. JAMA 1994; 272 (17): 1367-1371
Milton Keynes Primary Care Trust 2002. All rights reserved.
92
Screening Questions
1 Did the review ask a clearly-focused question?
Consider if the question is focused in terms of:
the population studied
the intervention given or exposure
the outcomes considered
Yes
Cant tell No
2 Did the review include the right type of study?

Consider if the included studies:
address the reviews question
have an appropriate study design
Yes
Cant tell No
Is it worth continuing?
Detailed questions
3 Did the reviewers try to identify all

Yes
the relevant studies?
Consider:
which bibliographic databases were used
if there was follow-up from reference lists
if there was personal contact with experts
if the reviewers searched for unpublished studies
if the reviewers searched for non-English language studies
Cant tell No
4 Did the reviewers assess the quality of

the included studies?
Consider:
if a clear, pre-determined strategy was used to determine
which studies were included. Look for:
a scoring system
more than one assessor
Yes
Cant tell No
5 If the results of the studies have been combined,

Yes Cant tell No
was it reasonable to do so?
Consider whether:
the results of each study are clearly displayed
the results were similar from study to study (look for tests of heterogeneity )
the reasons for any variations in results are discussed
93

6 How are the results presented and
Yes Cant tell No
what is the main result?
Consider:
how the results are expressed (eg. odds ratio, relative risk, etc.)
how large this size of result is and how meaningful it is
how you would sum up the bottom-line result of the review in one sentence
7 How precise are these results?

Yes
Consider:
if a confidence interval were reported.
Would your decision about whether or not to use
this intervention be the same at the upper confidence limit
as at the lower confidence limit?
if a p-value is reported where confidence intervals
are unavailable
Cant tell No
8 Can the results be applied to the local population?

Yes Cant tell No
Consider whether:
the population sample covered by the review could be
different from your population in ways that would produce different results
your local setting differs much from that of the review
you can provide the same intervention in your setting
9 Were all important outcomes considered?

Consider outcomes from the point of view of the:
individual
policy makers and professionals
family/carers
wider community
Yes
Cant tell No
10 Should policy or practice change as a result

of the evidence contained in this review?
Consider:
whether any benefit reported outweighs
any harm and/or cost. If this information is
not reported can it be filled in from elsewhere?
Yes
Cant tell No
94

HPPH Systematic Review Handbook

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

HPPH Systematic Review Handbook

Uploaded by

Copyright:

Available Formats

CochraneHealthPromotion

Unit One: Background to Systematic

Undertaking Systematic Reviews of Research on Effectiveness. CRDs Guidance for those

Unit Two: International Systematic

Unit Three: Resources Required

Initiate search strategy

Apply inclusion and

Cite reasons for

Obtain full reports and

Unit Four: Developing a Protocol

reviews of approaches to health promotion (eg. community development or peerdelivered

Unit Five: Asking an Answerable Question

Oliver S. 1997. Exploring lay perspectives on questions of effectiveness; IN: Maynard A,

Nutbeam D, Harris E. (2004). Theory in a Nutshell. A practical guide to health promotion

Donner A, Klar N. Pitfalls of and controversies in cluster randomization trials. Am J Public

Spencer L, Ritchie J, Lewis J, Dillon L. Quality in Qualitative Evaluation: A framework for

Centre for Reviews and Dissemination (Undertaking Systematic Reviews of Research on

Unit Six: Finding The Evidence

A lumped review (review of a number of different interventions) is simply a review comprising a

Sowden A, Arblaster L, Stead L. Community interventions for preventing smoking in young

Scope note to see

No narrower terms for

Unit Seven: Data Abstraction

Unit Eight: Principles of Critical Appraisal

Program differentiation: a manipulation check that is performed to safeguard against the

QUALITY ASSESSMENT TOOL FOR QUANTITATIVE STUDIES

Are the individuals selected to participate in the study likely to be representative of

What percentage of selected individuals agreed to participate?

Rate this section (see dictionary)

Less than 60%

Case-control, Before/After study,

Is the method of random allocation stated?

If the method of random allocation is stated

Was the method of random allocation

Rate this section (see dictionary)

Were there important confounders not reported in the paper?

Relevant Confounders NOT reported in the study:

Rate this section (see dictionary)

E) DATA COLLECTION METHODS

Were data collection tools shown or are they known to be valid?

Were data collection tools shown or are they known to be reliable?

Rate this section (see dictionary)

F) WITHDRAWALS AND DROP-OUTS

Rate this section (see dictionary)

Is there a sample size calculation or power calculation?

Is there a statistically significant difference between groups?

Are the statistical methods appropriate?

(Q4a) Indicate the unit of allocation (circle one)

(Q4b) Indicate the unit of analysis (circle one)

(Q4c) If 4a and 4b are different, was the cluster analysis done?

Is the analysis performed by intervention allocation status (i.e. intention to treat)

(Q1) What percentage of participants received the allocated intervention or exposure of

Was the consistency of the intervention measured?

Is it likely that subjects received an unintended intervention (contamination or

SUMMARY OF COMPONENT RATINGS

E DATA COLLECTION METHODS

F WITHDRAWALS AND DROPOUTS

WITH BOTH REVIEWERS DISCUSSING THE RATINGS:

Are the individuals selected to participate in the study likely to be representative of

There is no mention of how many individuals were

The study was directed at a group of people in a specific

Cohort (two group pre and post)

Before/After Study (one group pre + post)

Case control study