Systema(c

review and meta-analysis of Diagnos(c Test Studies

Kuan-Fu Chen CGMH EM SR/MA workshop January 8, 2013

Objec(ves
Dierences of diagnos(c SR/MA? Steps of diagnos(c SR/MA Formula(ng a focused ques(on Reviewing guidelines: QUADAS Literature searching Evalua(on of studies Data extrac(on Data synthesis
KFC CGMH EM Dx SR/MR talk 2

1/8/13

Dierence of Dx SR/MA
ID studies, assessing bias Method to combine results Paired of summary sta(s(cs to pool Design of diagnos(c studies

1/8/13

KFC CGMH EM Dx SR/MR talk

Diagnos(c studies
Deni(on
Dia + gnosis = apart/separated + knowledge Dia: Greek through Gnosis: Greek knowledge To reduce uncertainty

Purposes
Screening, triage, add-on or replacement tests, predict outcomes or monitor dz process Purpose must be considered during evalua(on
1/8/13 KFC CGMH EM Dx SR/MR talk 4

Phases of Dx test evalua(on

Prelim evalua(on (case-control)
I: whether results dierent for pts dz II: whether dz more likely to have + results

Accuracy & probability of condi(on (cross-sec(onal/cohort study)

III: how well dis(nguishes btw pts dz with suspicion IV: how informa(ve as add-on

Health outcomes (Randomized or before-ader study)

V: whether leads to beeer outcomes VI: acceptable costs
1/8/13 KFC CGMH EM Dx SR/MR talk 5

Phases of biomarker evalua(on

1/8/13

KFC CGMH EM Dx SR/MR talk

Objec(ves
Dierences of diagnos(c SR/MA? Steps of diagnos(c SR/MA Formula(ng a focused ques(on Reviewing guidelines: QUADAS Literature searching Evalua(on of studies Data extrac(on Data synthesis
KFC CGMH EM Dx SR/MR talk 7

1/8/13

Steps of (diagnos(c) SR/MA

Formula(ng a focused ques(on Reviewing guidelines ID databases/sources of studies Run searches and save cita(ons First screen by two reviewers Second screen by two reviewers Data extrac(on and quality assessment Data analyses
KFC CGMH EM Dx SR/MR talk 8

1/8/13

1/8/13

KFC CGMH EM Dx SR/MR talk

1/8/13

KFC CGMH EM Dx SR/MR talk

10

1/8/13

KFC CGMH EM Dx SR/MR talk

11

Objec(ves
Dierences of diagnos(c SR/MA? Steps of diagnos(c SR/MA Formula(ng a focused ques(on Reviewing guidelines: QUADAS Literature searching Evalua(on of studies Data extrac(on Data synthesis
KFC CGMH EM Dx SR/MR talk 12

1/8/13

PICO(S) for of diagnos(c studies

Popula(on
What are the characteris(cs of the pa(ents? What is the condi(on that may be present? Which diagnos(c test am I considering? What is the diagnos(c gold standard? How likely is the test to predict/rule out this condi(on? What study design would provide the best level of evidence? What clinical sekng would this study apply to?
KFC CGMH EM Dx SR/MR talk 13

Interven(on (diagnos(c test) Comparison Outcome

Study design/sekng

1/8/13

Objec(ves
Dierences of diagnos(c SR/MA? Steps of diagnos(c SR/MA Formula(ng a focused ques(on Reviewing guidelines: QUADAS Literature searching Evalua(on of studies Data extrac(on Data synthesis
KFC CGMH EM Dx SR/MR talk 14

1/8/13

Guidelines for (SR/MA) studies

QUA D A S
QUAlity of Diagnos(c Accuracy research Studies in SR

STA R D
STAndard for Repor(ng of Diagnos(c accuracy

RE MARK
REpor(ng recommenda(ons for tumor MARKer prognos(c studies

P R I S M
Preferred Repor(ng Items for SR/MA
1/8/13 KFC CGMH EM Dx SR/MR talk 15

QUADAS (2)
Four phases to establish tool for each review Four key domains to review Three sec(ons in each domain for risk of bias

1/8/13

KFC CGMH EM Dx SR/MR talk

16

QUADAS phases
Review ques(ons Review specic tailoring Flow diagram Judgments on bias and applicability

1/8/13

KFC CGMH EM Dx SR/MR talk

17

QUADAS phases
Review ques(ons
Four-part ques(on:
(P) Pa(ents, (I) Index test(s), (C) Reference standard, and (O) Target condi(on

Diagnos(c pathway:
Sekng, inten(on, pa(ent presenta(on, and/or prior tes(ng
1/8/13 KFC CGMH EM Dx SR/MR talk 18

QUADAS phases
Review specic tailoring

1/8/13

KFC CGMH EM Dx SR/MR talk

19

QUADAS phases
Flow diagram

1/8/13

KFC CGMH EM Dx SR/MR talk

20

QUADAS phases
Judgments on bias and applicability
Three sec(ons in each domain for risk of bias
Informa(on to support Signaling ques(ons Judgment of risk of bias

Applicability
Similar but not including signaling ques(ons Refer to rst phase (review ques(on)

1/8/13

KFC CGMH EM Dx SR/MR talk

21

QUADAS domains
Pa(ent selec(on Index test Reference standard Flow and (ming

1/8/13

KFC CGMH EM Dx SR/MR talk

22

QUADAS domains
Pa(ent selec(on
Signaling ques(ons
1. Was a consecu(ve or random sample of pa(ents enrolled? 2. Was a case-control design avoided? 3. Did the study avoid inappropriate exclusions? Spectrum bias (one of selec(on biases) (lab)? Excluding dicult to diagnose; liele diagnosis uncertainty:
overop(mis(c

Risk of bias

Exclusion of red ags: underes(ma(on Selec(on towards more severe manifesta(ons

Increase prevalence Bias in any direc(on

1/8/13

KFC CGMH EM Dx SR/MR talk

23

QUADAS domains
population

Study&

population

Study&

Pa(ent selec(on
Applicability concerns

Target' Population

Target' Population

Target vs. Study popula(on: generalizability Any discrepancy re: severity, demographics, presence of dieren(al diagnosis, comorbidity, sekng and previous tes(ng protocol
Target' Population

population

Study&

Supplemental: prospec(ve vs. Retrospec(ve?

Retrospec(ve:
No verica(on of Dz status Same reference standard for all? Same assessors for reference standard for all?
1/8/13 KFC CGMH EM Dx SR/MR talk 24

QUADAS domains
Index test
Signaling ques(ons
1. Were the index test results interpreted without knowledge of the results of the reference standard? 2. If a threshold was used, was it pre-specied?

Risk of bias
Ques(on 1: blinding, subjec(vity Ques(on 2: overop(mis(c

1/8/13

KFC CGMH EM Dx SR/MR talk

25

QUADAS domains
Index test
Applicability concerns
Same index tests?

Supplemental: (QUADAS 1)
Reproducibility: for all tests, dierent phases

1/8/13

KFC CGMH EM Dx SR/MR talk

26

QUADAS domains
Reference standard
Signaling ques(ons
1. Is the reference standard likely to correctly classify the target condi(on? 2. Were the reference standard results interpreted without knowledge of the results of the index test?

Risk of bias
Ques(on 1: direc(on? Misclassica(on bias (textbook?)
Bias confounding and role of chance slide

Ques(on 2: similar to index test, Incorpora(on bias

1/8/13 KFC CGMH EM Dx SR/MR talk 27

QUADAS domains
Reference standard
Applicability concerns:
target condi(on dened by the reference standard does not match the ques(on? Example: U/C for UTI with dierent cutos

1/8/13

KFC CGMH EM Dx SR/MR talk

28

QUADAS domains
Flow and (ming
Signaling ques(ons
1. Was there an appropriate interval between index test and reference standard? 2. Did all pa(ents receive the same reference standard? 3. Were all pa(ents included in the analysis?

Risk of bias
Ques(on 1: misclassica(on bias (if delay or treatment started). Direc(on? Acute vs. Chronic Ques(on 2: Verica(on bias: overes(mate Ques(on 3: lost to follow-up: Direc(on? Could be either direc(on (selec(on bias)
1/8/13 KFC CGMH EM Dx SR/MR talk 29

Objec(ves
Dierences of diagnos(c SR/MA? Steps of diagnos(c SR/MA Formula(ng a focused ques(on Reviewing guidelines: QUADAS Literature searching Evalua(on of studies Data extrac(on Data synthesis
KFC CGMH EM Dx SR/MR talk 30

1/8/13

Literature searching
Electronic database
Less produc(ve and more dicult

Dierent strategies
Dierent indexing terms/text words MeSH: diagnosis Textwords: specicity, sensi(vity

1/8/13

KFC CGMH EM Dx SR/MR talk

31

Objec(ves
Dierences of diagnos(c SR/MA? Steps of diagnos(c SR/MA Formula(ng a focused ques(on Reviewing guidelines: QUADAS Literature searching Evalua(on of studies Data extrac(on Data synthesis
KFC CGMH EM Dx SR/MR talk 32

1/8/13

Evalua(on of studies
Selec(on of sample.
Ideal: consecu(ve/randomly selected

Reference test Blinding Quality repor(ng Evidence of bias:

table of rela(ve DOR in dierent study design

Incorpora(ng QA in SR: Checklist

1/8/13 KFC CGMH EM Dx SR/MR talk 33

Objec(ves
Dierences of diagnos(c SR/MA? Steps of diagnos(c SR/MA Formula(ng a focused ques(on Reviewing guidelines: QUADAS Literature searching Evalua(on of studies Data extrac(on Data synthesis
KFC CGMH EM Dx SR/MR talk 34

1/8/13

Data extrac(on
Accuracy expression
Sensi(vity/specicity: Based on cut-o ROC curve PPV inuence of prevalence May consider DOR instead of Sensi(vity/specicity, LR
DOR (Diagnos(c Odds Ra(o) also = LR(+)/LR(-) However, discarded informa(on
DOR 25: LR(+) 5, LR(-) 0.2, DOR 100: LR(+) 10, LR(-) 0.1

Choice of parameters:

Predic(ve value: high heterogeneityprevalence varia(on Examples of abstract/ar(cle review form

Post-test probability can be used instead

1/8/13

KFC CGMH EM Dx SR/MR talk

35

Objec(ves
Dierences of diagnos(c SR/MA? Steps of diagnos(c SR/MA Formula(ng a focused ques(on Reviewing guidelines: QUADAS Literature searching Evalua(on of studies Data extrac(on Data synthesis
KFC CGMH EM Dx SR/MR talk 40

1/8/13

Data synthesis
Test of heterogeneity
heterogenous

Bayesian adaptation SROC asym

homogenous

Test of cut-off effect no

yes

LM method test of asymmetry, SROC or pool ROC curves

sym

Meta-regression DOR on characteristics, investigate source of heterogeneity

Pool sensitivity & specicity

SROC: summary ROC LM method: Lieenburg Moses regression method DOR: diagnos(c odds ra(o
KFC CGMH EM Dx SR/MR talk 41

1/8/13

Systema(c review and meta-analysis of Diagnos(c Test Studies

WORKSHOP TIME!

1/8/13

KFC CGMH EM Dx SR/MR talk

42

Objec(ves
Understand and workout poten(al bias during diagnos(c SR/MA
Spectrum bias Misclassica(on bias Incorpora(on bias Lost of follow up Selec(ng threshold verica(on bias

Use QUADAS-2 form to evaluate one study

1/8/13 KFC CGMH EM Dx SR/MR talk 43

Selec(on of pa(ent (spectrum bias)

Excluding dicult to diagnose
Overop(mis(c or underes(ma(ng, why?

Excluding red ag
Overop(mis(c or underes(ma(ng, why?

Evaluate and discuss the reasons in examples

1/8/13 KFC CGMH EM Dx SR/MR talk 44

Example 1: Centors CPR

1/8/13

KFC CGMH EM Dx SR/MR talk

45

Example 2: BNP

1/8/13

KFC CGMH EM Dx SR/MR talk

46

Example 2: BNP

1/8/13

KFC CGMH EM Dx SR/MR talk

47

Misclassica(on bias
Use those tables to discuss the reasons
PCR vs. B/C for sepsis diagnosis PCT vs. B/C for sepsis diagnosis

Use blood culture as example

Discuss interval and reference test issue

1/8/13

KFC CGMH EM Dx SR/MR talk

48

Misclassica(on bias
Cases: B/C + PCR + PCR - False 100 50 150 Sensi(vity=66% Control: B/C - 40 60 100 Specicity=60% Total 150 100 250

DierenAal misclassicaAon bias Say Sensi(vity of B/C = 80%, specicity = 90%, and related to PCR results Cases: real sepsis PCR + PCR - True
1/8/13

Control: real ctrl 40 60 100 Specicity=60%

Total 150 100 250

110 40 150 Sensi(vity=73%

KFC CGMH EM Dx SR/MR talk

49

Misclassica(on bias
Cases: B/C + PCT + PCT - False 100 50 150 Sensi(vity=66% Control: B/C - 40 60 100 Specicity=60% Total 150 100 250

Non-DierenAal misclassicaAon bias Say Sensi(vity of B/C = 80%, specicity = 75%, and not related to PCT results Cases: real sepsis PCT + PCT - True
1/8/13

Control: real ctrl 25 37 62 Specicity=60%

Total 150 100 250

125 63 188 Sensi(vity=66%

KFC CGMH EM Dx SR/MR talk

50

Incorpora(on bias
Example: SIRS vs. WBC for sepsis diagnosis
Overop(mis(c or underes(ma(ng, why?

1/8/13

KFC CGMH EM Dx SR/MR talk

51

Lost of follow up
Example: transferred to other hospital
Overop(mis(c or underes(ma(ng, why?

1/8/13

KFC CGMH EM Dx SR/MR talk

52

Selec(ng threshold
Example:
Post-hoc determina(on of BUN/Cre > 15 as risk factor in acute stroke Overop(mis(c or underes(ma(ng, why?

1/8/13

KFC CGMH EM Dx SR/MR talk

53

Verica(on bias
Example: PSA
Verica(on and incorpora(on biases in studies assessing screening tests: prostate-specic an(gen as an example (Gupta & Roehrborn) Overop(mis(c or underes(ma(ng, why?

1/8/13

KFC CGMH EM Dx SR/MR talk

54

Verica(on bias
Disease + PSA + PSA - Total 231 27 258 Disease - 32 54 86 Total 263 8 344 % Veried 61% 37%

Sensi(vity=90% Specicity=63%

Corrected PSA + PSA - Total

Disease +

Disease -

Total

Sensi(vity=__% Specicity=__%

1/8/13

KFC CGMH EM Dx SR/MR talk

55

Verica(on bias
Disease + PSA + PSA - Total 231 27 258 Disease - 32 54 86 Total 263 8 344 % Veried 61% 37%

Sensi(vity=90% Specicity=63%

Disease + PSA + PSA - Total 377 74 451

Disease - 52 147 199

Total 429 221 650

Sensi(vity=84% Specicity=74%

1/8/13

KFC CGMH EM Dx SR/MR talk

56

Verica(on bias

1/8/13

KFC CGMH EM Dx SR/MR talk

57

Verica(on bias

1/8/13

KFC CGMH EM Dx SR/MR talk

58