CAT

(Critical Appraisal of the Topics)

Warko Karnadihardja, Reno Rudiman
Dept. of Surgery, Hasan Sadikin Hospital
Universitas Padjadjaran
Bandung

Critical Appraisal of the Topics

Requires

certain basic skills : To define the

clinical question as clearly as possible and
translate the question into a statement
This statement can be used to search
literature, conduct the search, and select the
best articles before applying appropriate rules
of evidence to evaluate the articles

Mc Kibbon, ACP Journal Club, 1994. 120 (Suppl 2): A 10-12

Types of Studies

Cross
Sectional

Cohort study
Case Control
Case
T i m e
l i n e
Report
Case `Series
Clinical Trial

Evidence Based Medicine

Can now be looked at as :
The process of life-long, self directed learning
in which caring of patient leads to search for,
critical appraisal and implementation of the
best available evidence in all clinical and
health care related issues with the
incorporation of personal expertise and
patients values
Iqbal Mustafa : EBM Workshop, PERDICI, Jakarta, June 13, 2004

The Users Guide To The Medical Literature

Three fundamental questions
1.
Are the results of the study valid ?
2.
What are the results ?
3.
Will the results help me in caring for my patients?

Oxman AD et al : JAMA 1993;270: 2093-5

Users Guides For Selecting Articles Most Likely To

Provide Valid Results
Subject

Screening Criteria
Original Studies
Therapy
_____ ?
Diagnosis
_____ ?
Harm
_____ ?
Prognosis
_____ ?
Integrative Studies
Overview
_____ ?
Practice Guidelines
_____ ?
Decision Analysis
_____ ?
Modified from American Medical Association. JAMA 1993; 270(17):2094

Users Guides for Selecting Articles Most Likely to Provide

Valid Results
Subject

Screening Criteria
Original Studies

Therapy

Was the assignment of patients to treatments randomized ?

Were all of the patients who entered the trial properly accounted for and
attributed at its conclusion ?

Diagnosis

Was there an independent, blind comparison with a reference standard ?

Did the patient sample include an appropriate spectrum of the sort of patients
to whom the diagnostic test will be applied in clinical practice ?

Harm

Were there clearly identified comparison groups that were similar with respect
to important determinants of outcome (other than the ones of interest) ?
Were outcomes and exposures measured in the same way in the groups being
compared ?

Prognosis

Was there a representative patient sample at a well defined point in the course
of disease ?
Was follow-up sufficiently long and complete ?

Users Guides for Selecting Articles Most Likely to Provide

Valid Results
Subject

Screening Criteria
Integrative Studies

Overview

Did the review address a clearly focused question ?

Were appropriate criteria used to select articles for inclusion ?

Practice guidelines

Were options and outcomes clearly specified ?

Did the guideline use an explicit process to identify, select and combine
evidence ?

Decision analysis

Did the analysis faithfully model a clinically important decision ?

Was valid evidence used to develop the baseline probabilities and
utilities ?

Modified from American Medical Association. JAMA 1993; 270(17):2094

Users Guides for an Article about Therapy

Are the results of the study valid ?
1.
Primary Guides
________ ?
2.
Secondary Guides ________ ?
3.
What were the results ________ ?
4.
Will the results help me in caring for my patients ?

Users Guides for an Article about Therapy

Are the results of the study valid ?
Primary guides
Was assignment of patients to treatments randomized ?
Were all patients who entered the trial properly accounted for and
attributed at its conclusion ?
Was follow-up complete ?
Were patients analyzed in the groups to which they were
assigned ?
Secondary guides
Were patients, health workers, and study personnel blind to
treatment ?
Were the groups similar at the start of the trial ?
Aside from experimental intervention, were the groups treated
equally ?

Users Guides for an Article about Therapy

What were the results ?
How large was the treatment effect ?
How precise was the estimate of the treatment effect ?
Will the results help me in caring for my patients ?
Can the results be applied to my patient care ?
Were all clinically important outcomes considered ?
Are the likely treatment benefits worth the potential harm and
costs ?

Modified from American Medical Asociation. JAMA 1994;271(1):60

Statistics for interpreting the importance and precision

of therapeutic results

Yes

No

Exposed

Not Exposed

Control event rate (CER) = c/c+d

Experimental event rate (EER) = a/a+b
(a) Relative Risk (RR) = EER/CER=(a/a+b)/(c/c+d)
(b) Relative Risk Reduction (RRR) = CER-EER/CER
(commonest reported measure of dichotomous treatment effect)
(c) Absolute Risk Reduction (ARR) = CER-EER
(d) Number Needed to Treat (NNT) = 1/ARR

Users Guides for Evaluating and Applying Results of

Studies of Diagnostic Tests
Are the results of the study valid ?
Primary guides
Secondary guides
What were the results ?
Will the results help me in caring for my patients ?

Users Guides for Evaluating and Applying Results of

Studies of Diagnostic Tests
Are the results of the study valid ?
Primary guides
Was there an independent, blind comparison with a reference
standard ?
Did the patient sample include an appropriate spectrum of
patients to whom the diagnostic test will be applied in clinical
practice ?
Secondary guides
Did the results of the test being evaluated influence the
decision to perform the reference standard ?
Were the methods for performing the test described in
sufficient detail to permit replication ?

Users Guides for Evaluating and Applying Results of

Studies of Diagnostic Tests
What were the results ?
Are the likelihood ratios for the test results presented or are
data necessary for calculation of likelihood ratios provided ?
Will the results help me in caring for my patients ?
Will the reproducibility of the test result and its interpretation be
satisfactory in my setting ?
Are the results applicable to my patients ?
Will the results change my management ?
Will patients be better off as a result of the test ?
Modified from American Medical Asociation. JAMA 1994;271(5):390

Calculations for Diagnostic Tests

Gold standard
Positive (condition
present)

Gold standard
Negative (condition
not present )

Test result
Positive

True Positive

False Positive

Test result
Negative

False Negative

True Negative

Stable Properties:
Sensitivity = True Positives/(True Positives + False Negatives)
Specificity = True Negatives/(False Positive + True Negative)
Frequency Dependent Properties:
Positive Predictive Value = True Positive/(True Positive + False
Positive)
Negative Predictive Value = True Negative/(True Negative + False
Negative)

Likelihood Ratios
The likelihood ratio for a test result compares the likelihood of that result in
patients with disease to the likelihood of that result in patients without disease:
Condition
Present

Condition Absent

Test Positive

Test Negative

Positive LR = (a/a+c)/(b/b+d)
Negative LR = (c/a+c)/(d/b+d)

Causation/Harm/Etiology
Are the results of this study valid ?
Were there clearly defined groups of patients,
similar in all important ways other than exposure
to the treatment or other cause?

Were treatments/exposures and clinical

outcomes measured in the same ways in both
groups (was the assessment of outcomes either
objective or blinded to exposure)?
Was the follow-up of study patients complete and
long enough?

Causation/Harm/Etiology
Is it clear that the exposure preceded the onset of the
outcome ?
Is there a dose-response gradient ?

Is there positive evidence from

dechallenge-rechallenge study ?
Is the association consistent from study to
study ?
Does the association make biological
sense ?

Calculations for Causation/Harm/Etiology

Odds Ratios and Relative Risk
Outcome
Positive

Outcome
Negative

Exposure

No Exposure

Appropriate use Formulae

Relative
Risk

Odds Ratio

Randomize
d controlled
trials
Cohort
studies
Casecontrol
studies

RR=
(a/a+b)/(c/c+d)

OR=
(a/c)/
(b/d)=ad/b
c

Interpreting strength of
association
For randomized trial, strong
association if RR or OR > 1
For cohort study, strong association if
RR > 3 or OR > 4

Prognosis
Are the results of this prognosis study valid ?
Was a defined, representative sample of patients
assembled at a common (usually early) point in
the course of their disease?
Was patient follow-up sufficiently long and
complete?
Were objective outcome criteria applied in a
"blind" fashion?
If subgroups with different prognoses are
identified, was there adjustment for important
prognostic factors?
Was there validation in an independent group
("test set") of patients?

Prognosis
Are the results of this prognosis study important?
How likely are the outcomes over
time ?
How precise are the prognostic
estimates ?

Calculating a confidence interval around the

measure of prognosis
Clinical Measure

Standard Error (SE)

Proportion (as in the rate

of some prognostic
event, etc.) where: the
number of patients
where p is proportion
= n the proportion of
and n is number of
these patients who
patients
experience the event
=P

Typical Calculation of
CI
If p = 24/60 = 0.4 (or
40%) and n = 60

= 0.063 (or 6.3%)

95% CI is
40% 1.96 6.3% or
27.6% to 52.4%

Limitations of CATs
1.
2.
3.

Individual CATs can be wrong. CAT can appear as

drafts, without peer review.
Individual CATs contain a single element of the
relevant literature.
Individual CATs may have a short shelf life. They
become obsolete as soon as newer, better evidence
becomes available.

Level 1 of Evidence
Level
1a

Therapy/Prevention,
Aetiology/Harm

Prognosis

Diagnosis

SR (with homogeneity*) SR (with homogeneity*)

of RCTs
of inception cohort
studies; CDR
validated in different
populations

SR (with homogeneity*) of
Level 1 diagnostic
studies; CDR with 1b
studies from different
clinical centres

1b

Individual RCT (with

narrow
Confidence Interval)

1c

All or none

Individual inception
cohort study with >
80% follow-up;
CDR validated in a
single population

Validating** cohort study

with good reference
standards; or CDR
tested within one
clinical centre

All or none case-series Absolute SpPins and SnNo

uts

Level 2 of Evidence
Level

Therapy/Prevention,
Aetiology/Harm

Prognosis

Diagnosis

2a

SR (with homogeneity* ) of
cohort studies

SR (with homogeneity*) of
either retrospective cohort
studies or untreated
control groups in RCTs

SR (with homogeneity*) of
Level >2 diagnostic studies

2b

Individual cohort study

(including low quality RCT;
e.g., <80% follow-up)

Retrospective cohort
study or follow-up of
untreated control patients
in an RCT; Derivation of
CDR or validated on
split-sample only

Exploratory** cohort study with

goodreference standards;
CDR after derivation, or
validated only on splitsample or databases

2c

Outcomes" Research;"
Ecological studies

Outcomes" Research"

Level 3,4,5 of Evidence

Level

Therapy/Prevention,
Aetiology/Harm

Prognosis

Diagnosis

3a

SR (with homogeneity*) of
case-control studies

SR (with homogeneity*) of 3b
and better studies

3b

Individual Case-Control
Study

Non-consecutive study; or
without consistently applied
reference standards

Case-series (and
Case-series (and
poor quality cohort and case- poor quality prognostic co
control studies
hort studies***
)
)
5
Expert opinion without
Expert opinion without
explicit critical appraisal, or
explicit critical appraisal,
based on physiology, bench
or based on physiology,
"research or "first principles
bench research or "first
"principles

Case-control study, poor or

non-independent reference
standard

Expert opinion without explicit

critical appraisal, or based on
physiology, bench research or
""first principles

Grades of Recommendation
A

consistent level 1 studies

consistent level 2 or 3 studies or

extrapolations from level 1 studies

level 4 studies or extrapolations from

level 2 or 3 studies

level 5 evidence or troublingly

inconsistent or inconclusive studies of
any level

Old world

EBM world

Source of knowledge

Expert opinion

analysis of the evidence

Essential skills

Clinical

Clinical plus ability to appraise evidence

Essential information sources

Experts
Textbooks
Selected journals

Electronic access to all research

evidence
Cochrane Library

Importance of statisticians, epidemiologist, Low

economists, etc

High

Importance of gathering new evidence on

patients

Low

High

Consultant to Juniors

Dictatorship

Democratic

Importance of keeping up to date

Optional

Essential

Importance of access to research

evidence

Low

High

Relationship to patients

Expert to pupil

Potentially much more equal

BMA/ScHARR

CRITICAL
APPRAISAL
SKILLS
WORKSHOP

Critical Appraisal - What is it and Why is it

important?
Andrew Booth
Senior Lecturer in Evidence Based
Healthcare Information

Why Critical Appraisal?

Theory

Practice

What is critical appraisal?

How is it done?
When is it used?
What are some of its
uses?

Why should you get

involved?
How can you get
involved?
What resources are there
to help you?

What is critical appraisal?

To weigh up the evidence
critically to assess its
validity (closeness to the
truth) and usefulness
(clinical applicability).
[Adapted from Sackett &
Haynes EBM 1995; 1 : 45].

Background to critical appraisal

McMaster University
Clinical Epidemiology
Problem Based Learning
Rather than fill students heads with facts - cultivate skills
for lifelong learning
To include searching, filtering, critical appraisal and
digesting
Developed User Guides to the Medical Literature

Critical Appraisal Skills Programme

Critical

appraisal skills
training for the NHS
(Anglia and Oxford)
Appraises:

Reliability Validity;
Applicability

http://www.phru.org.uk/
~casp/index.htm

Why is it important?

Clinicians would have to read

17 articles/day; 365 days per
year to keep up-to-date
(internal medicine)
Research is of variable
quality
Only an estimated 1% is
judged clinically relevant
Which is the 1%?

Why is it important?
A requirement for the EBM component of Membership
Exams (e.g. GP, Psych, Family Planning etcetera)
Royal College of Ophthalmologists:
Professional Attitudes and Conduct:
Through encouragement of personal development, to have
developed a style of care which is :
Scientific (e.g. critical appraisal of the scientific literature,
evidence-based practice and use of information technology and
statistics.)

How is appraisal done?

Problem or scenario
Determine appropriate
source
Identify relevant article(s)
Use appropriate checklist

Assess relative
merits/demerits
Make overall assessment
(strength of evidence)
Apply findings (strength
of recommendations)

Principles of appraisal
Intrinsic

not extrinsic factors

Focussed question [Patient Intervention Outcome
Comparison]
Structured agenda
Explicit judgements
an ivory tower exercise - How can I apply
these findings?

Not

When is appraisal used?

Undergraduate and postgraduate education
Health technology assessment
Systematic Reviews (judgements of methodological
quality)
Production of national or local guidelines
In value-added databases (eg. DARE & NEED)
Current awareness bulletins and journals

...and Teaching EBP

Workshops

Learning outcomes
Skills learned can be used in 3 important ways:

Critical appraisal skills can be used to improve clinical decisionmaking by the implementation of evidence-based healthcare.
Evidence can be used to inform decisions about health policy,
for example, in making choices
about the prioritisation
of services.
The approach can be used to empower consumers of the
health services.

Why should you get involved?

Makes

your practice more scientific

Extends existing skills.
Reduces uncertainty
Improved profile/prestige
Its fun!

Getting started
Scenario

- from real life or invented

Article - primary or secondary study addressing
the problem in hand
Checklist - for assessing the study design of the
article [User Guides]
And optionally, a Crib sheet, digest or
commentary

Appraising a paper
Three

broad issues need to be considered when

appraising a paper:

A/ Are the results valid?

B/ What are the results?
C/ Will the results help locally?

Systematic reviews - definitions

Review
A synthesis of results and conclusions of two
or more publications.
Systematic review
Comprehensive identification and synthesis
of all literature on a given topic.
Meta-analysis
Statistical technique for combining results of
several studies into a single numerical
estimate.

Characteristics
Summaries
Comprehensive
Systematic
Explicit
Reproducible

Systematic reviews provide

Good

quality evidence, more

reliable results
A useful basis for decision making
Information of greater statistical
significance
Control over the volume of
available literature

What to look for?

Review versus systematic
review
Comprehensiveness of search

focused question
databases covered
time period, language
search strategy
inclusion, exclusion criteria

Selection bias addressed

Assessment of study quality
Homogeneity/ heterogeneity

What to look for?

Are

the overall results valid?

How

precise are the results?

Are

the conclusions supported

by the data?
If so,

how do they help my patient care?

Will the results help locally?

Can

the results be applied to my

local population?

Were

all important outcomes

considered?
Should I change my practice as a
result of this evidence?

Sources of reviews

Cochrane Collaboration

NHS Centre for Reviews and

Dissemination (NHS CRD)

Database of Abstracts of Reviews of

Effectiveness (DARE)

NHS Health Technology Assessment

Cochrane Library

HTA reports

InterTASC/NICE

Trent, South & West, West Midlands

Summary
CATs are a tactic for helping clinical learners teach themselves
how to formulate clinical questions; search for the best
evidence; appraise, organise and summarise this evidence;
integrate it with clinical expertise; and practice evidencebased medicine. When generated by clinical teams, journal
clubs, or in academic half-days, their educational value is
multiplied. Existing CATs can be used as starting points for
seeking and appraising updates in the relevant evidence.
The CAT-maker assists this process by:
1.
carrying out the important clinical calculations;
2.
storing appraisals (as well as the search strategies that led
to them); and
3.
generating files that can be formatted with word-processors,
stored and printed for other team members.

Thank You