Chapter 4: Reliability & Validity Reliability: To what extent can we say that the data are consistent?

t? About consistency Different statistical techniques available, each produces a reliability coefficient ran in fro! "#"" to $%#"" &totally inconsistent or totally consistent'

Different techniques to !easure reliability

Approach/Method Test(Retest &coefficient of stability' *arallel(+or!s &coefficient of equivalence' -nternal .onsistency %# /plit(half Description )easures across ti!e )easures across for!s Remarks ( Article need to start ti!e lapse &hi her reliability if lon er ti!e' ( two for!s of the sa!e instru!ent supposedly focusin on the sa!e ob,ect to !easure ( 9ill be expectedly hi h &unusual if low'

( 5xa!inin perfor!ance on odd and even(nu!bered ite!s separately and !easure their correlation ( 6rder of ite!s does not !atter &all possible co!binations co!puted' ( /a!e as 1R 0" if dichoto!ous7 !ore versatile if ite!s have 8 0 possible values ( +or ran:ed data &ordinal' ( +or no!inal data &cate orical' ( Reliability of ratin s &for raw score' ( +or raw score

0# 1uder(Richardson 20" &1(R 0"' 3# .ronbach4s alpha

-nterrater Reliability %# 1endall4s coefficient of concordance 0# .ohen4s :appa 3# -ntraclass .orrelation &-..' ;# *earson4s product( !o!ent correlation

Standard Error Meas rement: Ran e within which a score would li:ely to fall if a iven !easured ob,ect were to be re!easured

Some !arnin"s abo t reliability %# Different !ethods of assessin reliability consider the issue of consistency fro! different perspectives# 5# # a hi h coefficient of stability does not necessarily !ean hi h internal consistency Reliability coefficients really apply to data, not to the !easurin instru!ents# They are characteristics of data rather than the instru!ents that produce the data# *lace !ore faith in ood results for lar e roups rather than for s!all roups# -f a test is ad!inistered under ti!e pressure, various esti!ates of internal consistency &split(half, 1R 0", alpha' will be hi h# /o don4t be overly i!pressed# Reliability not only criterion used to assess quality of data#

0#

3# ;#

<#

Validity ( ( ( ( concerned with accuracy i#e# whether the !easurin instru!ent !easures what it purports to !easure reliability is a necessary but not sufficient condition for validity7 valid data are reliable, but not all reliable data are valid three :inds of validity: %# .ontent =alidity o .ontent experts evaluates instru!ent7 we should as:: 9ho did the evaluations? 9hat did they chec:>do? ?ow was the outco!e? 0# .riterion(Related =alidity o .o!parin scores with a relevant, established criterion variable o .orrelatin these two set of scores to produce the #alidity coefficient o Two :inds: Conc rrent #alidity &two tests ad!inistered at sa!e or near sa!e ti!e' $redicti#e #alidity &test before the criterion' 3# .onstruct =alidity o 3 ways *rovidin correlation with conver ent and discri!inative variables7 score should indicate hi h correlation for

conver ent &con#er"ent #alidity' variables and low correlation for discri!inant &discriminant #alidity' variables# /how certain roups obtain hi her !ean scores on new instru!ent than other roups, with the two roups deter!ined on lo ical rounds prior to the test .onduct a factor analysis

%arnin"s abo t Validity %# 0# =alidity &li:e reliability' is a characteristic of the data, not the instru!ent# -!portance of correlation as correlation plays a central role in assessin construct validity &the first two ways'7 hence re!e!ber warnin s about correlation

&inal Comments %# 0# 3# ?ow hi h should reliability and validity coefficients be? Answer: -t should be ,ud ed in relative to other available instru!ents# Researchers should use !ultiple !ethods to assess reliability and validity# Reliability and validity related to data quality, which by itself, does not deter!ine the de ree to which the study4s results can be trusted# *ossible for conclusions to be worthless because of the wron use of statistical procedure, or desi n of study deficient# -n other words, reliability and validity are i!portant, but other i!portant concerns !ust be attended to as well#