28/07/2015

ItemAnalysisexplained

MakingtheMostofExams:ProceduresforItemAnalysis
RaymondM.Zurawski,Ph.D.
AssociateProfessorandCoordinatorofPsychology
St.NorbertCollege

Oneofthemostimportant(ifleastappealing)tasksconfrontingfacultymembersistheevaluationof
studentperformance.Thistaskrequiresconsiderableskill,inpartbecauseitpresentssomanychoices.
Decisionsmustbemadeconcerningthemethod,format,timing,anddurationoftheevaluative
procedures.Oncedesigned,theevaluativeproceduremustbeadministeredandthenscored,
interpreted,andgraded.Afterwards,feedbackmustbepresentedtostudents.Accomplishingthese
tasksdemandsabroadrangeofcognitive,technical,andinterpersonalresourcesonthepartoffaculty.
Butanevenmorecriticaltaskremains,onethatperhapstoofewfacultyundertakewithsufficientskill
andtenacity:investigatingthequalityoftheevaluativeprocedure.
Evenafteranexam,howdoweknowwhetherthatexamwasagoodone?Itisobviousthatanyexam
canonlybeasgoodastheitemsitcomprises,butthenwhatconstitutesagoodexamitem?Our
studentsseemtoknow,oratleastbelievetheyknow.Butaretheycorrectwhentheyclaimthatan
itemwastoodifficult,tootricky,ortoounfair?
LewisAiken(1997),theauthorofaleadingtextbookonthesubjectofpsychologicalandeducational
assessment,contendsthatapostmortemevaluationisjustasnecessaryinclassroomtestingasitis
inmedicine.Indeed,justsuchapostmortemprocedureforexamsexistsitemanalysis,agroupof
proceduresforassessingthequalityofexamitems.Thepurposeofanitemanalysisistoimprovethe
qualityofanexambyidentifyingitemsthatarecandidatesforretention,revision,orremoval.More
specifically,notonlycantheitemanalysisidentifybothgoodanddeficientitems,itcanalsoclarify
whatconceptstheexamineeshaveandhavenotmastered.
So,whatproceduresareinvolvedinanitemanalysis?Thespecificproceduresinvolvedvary,but
generally,theyfallintooneoftwobroadcategories:qualitativeandquantitative.

QualitativeItemAnalysis
Qualitativeitemanalysisproceduresincludecarefulproofreadingoftheexampriortoits
administrationfortypographicalerrors,forgrammaticalcuesthatmightinadvertentlytipoff
examineestothecorrectanswer,andfortheappropriatenessofthereadinglevelofthematerial.Such
procedurescanalsoincludesmallgroupdiscussionsofthequalityoftheexamanditsitemswith
examineeswhohavealreadytakenthetest,orwithdepartmentalstudentassistants,orevenexpertsin
thefield.Somefacultyuseathinkaloudtestadministration(cf.Cohen,Swerdlik,&Smith,1992)
inwhichexamineesareaskedtoexpressverballywhattheyarethinkingastheyrespondtoeachof
theitemsonanexam.Thisprocedurecanassisttheinstructorindeterminingwhethercertainstudents
(suchasthosewhoperformedwellorthosewhoperformedpoorlyonapreviousexam)
misinterpretedparticularitems,anditcanhelpindeterminingwhystudentsmayhavemisinterpreteda
particularitem.

QuantitativeItemAnalysis
Inadditiontotheseandotherqualitativeprocedures,athoroughitemanalysisalsoincludesanumber
ofquantitativeprocedures.Specifically,threenumericalindicatorsareoftenderivedduringanitem
analysis:itemdifficulty,itemdiscrimination,anddistractorpowerstatistics.

ItemDifficultyIndex(p)
http://faculty.mansfield.edu/lfeil/201/itemanalysisexplained.htm

1/3

28/07/2015

ItemAnalysisexplained

Theitemdifficultystatisticisanappropriatechoiceforachievementoraptitudetestswhentheitems
arescoreddichotomously(i.e.,correctvs.incorrect).Thus,itcanbederivedfortruefalse,multiple
choice,andmatchingitems,andevenforessayitems,wheretheinstructorcanconverttherangeof
possiblepointvaluesintothecategoriespassingandfailing.
Theitemdifficultyindex,symbolizedp,canbecomputedsimplybydividingthenumberoftest
takerswhoansweredtheitemcorrectlybythetotalnumberofstudentswhoansweredtheitem.Asa
proportion,pcanrangebetween0.00,obtainedwhennoexamineesansweredtheitemcorrectly,and
1.00,obtainedwhenallexamineesansweredtheitemcorrectly.Noticethatnotestitemneedhave
onlyonepvalue.Notonlymaythepvaluevarywitheachclassgroupthattakesthetest,aninstructor
maygaininsightbycomputingtheitemdifficultylevelforanumberofdifferentsubgroupswithina
class,suchasthosewhodidwellontheexamoverallandthosewhoperformedmorepoorly.
Althoughthecomputationoftheitemdifficultyindexpisquitestraightforward,theinterpretationof
thisstatisticisnot.Toillustrate,consideranitemwithadifficultylevelof0.20.Wedoknowthat
20%oftheexamineesansweredtheitemcorrectly,butwecannotbecertainwhytheydidso.Does
thisitemdifficultylevelmeanthattheitemwaschallengingforallbutthebestpreparedofthe
examinees?Doesitmeanthattheinstructorfailedinhisorherattempttoteachtheconceptassessed
bytheitem?Doesitmeanthatthestudentsfailedtolearnthematerial?Doesitmeanthattheitemwas
poorlywritten?Toanswerthesequestions,wemustrelyonotheritemanalysisprocedures,both
qualitativeandquantitativeones.

ItemDiscriminationIndex(D)
Itemdiscriminationanalysisdealswiththefactthatoftendifferenttesttakerswillansweratestitem
indifferentways.Assuch,itaddressesquestionsofconsiderableinteresttomostfaculty,suchas,
doesthetestitemdifferentiatethosewhodidwellontheexamoverallfromthosewhodidnot?or
doesthetestitemdifferentiatethosewhoknowthematerialfromthosewhodonot?Inamore
technicalsensethen,itemdiscriminationanalysisaddressesthevalidityoftheitemsonatest,thatis,
theextenttowhichtheitemstaptheattributestheywereintendedtoassess.Aswithitemdifficulty,
itemdiscriminationanalysisinvolvesafamilyoftechniques.Whichonetousedependsonthetypeof
testingsituationandthenatureoftheitems.Imgoingtolookatonlyoneofthose,theitem
discriminationindex,symbolizedD.Theindexparallelsthedifficultyindexinthatitcanbeused
wheneveritemscanbescoreddichotomously,ascorrectorincorrect,andhenceitismostappropriate
fortruefalse,multiplechoice,andmatchingitems,andforthoseessayitemswhichtheinstructorcan
scoreaspassorfail.
Wetestbecausewewanttofindoutifstudentsknowthematerial,butallwelearnforcertainishow
theydidontheexamwegavethem.Theitemdiscriminationindexteststhetestinthehopeof
keepingthecorrelationbetweenknowledgeandexamperformanceascloseasitcanbeinan
admittedlyimperfectsystem.
Theitemdiscriminationindexiscalculatedinthefollowingway:
1. Dividethegroupoftesttakersintotwogroups,highscoringandlowscoring.Ordinarily,thisis
donebydividingtheexamineesintothosescoringaboveandthosescoringbelowthemedian.
(Alternatively,onecouldcreategroupsmadeupofthetopandbottomquintilesorquartilesor
evendeciles.)
2. Computetheitemdifficultylevelsseparatelyfortheupper(pupper)andlower(plower)scoring
groups.
3. SubtractthetwodifficultylevelssuchthatD=pupperplower.
Howistheitemdiscriminationindexinterpreted?Unliketheitemdifficultylevelp,theitem
discriminationindexcantakeonnegativevaluesandcanrangebetween1.00and1.00.Considerthe
http://faculty.mansfield.edu/lfeil/201/itemanalysisexplained.htm

2/3

28/07/2015

ItemAnalysisexplained

followingsituation:supposethatoverall,halfoftheexamineesansweredaparticularitemcorrectly,
andthatalloftheexamineeswhoscoredabovethemedianontheexamansweredtheitemcorrectly
andalloftheexamineeswhoscoredbelowthemedianansweredincorrectly.Insuchasituation
pupper=1.00andplower=0.00.Assuch,thevalueoftheitemdiscriminationindexDis1.00andthe
itemissaidtobeaperfectpositivediscriminator.Manywouldregardthisoutcomeasideal.It
suggeststhatthosewhoknewthematerialandwerewellpreparedpassedtheitemwhileallothers
failedit.
Thoughitsnotasunlikelyaswinningamilliondollarlottery,findingaperfectpositivediscriminator
onanexamisrelativelyrare.Mostpsychometricianswouldsaythatitemsyieldingpositive
discriminationindexvaluesof0.30andabovearequitegooddiscriminatorsandworthyofretention
forfutureexams.
Finally,noticethatthedifficultyanddiscriminationarenotindependent.Ifallthestudentsinboththe
upperandlowerlevelseitherpassorfailanitem,theresnothinginthedatatoindicatewhetherthe
itemitselfwasgoodornot.Indeed,thevalueoftheitemdiscriminationindexwillbemaximized
whenonlyhalfofthetesttakersoverallansweranitemcorrectlythatis,whenp=0.50.Onceagain,
theidealsituationisoneinwhichthehalfwhopassedtheitemwerestudentswhoalldidwellonthe
examoverall.
Doesthismeanthatitisneverappropriatetoretainitemsonanexamthatarepassedbyall
examinees,orbynoneoftheexaminees?Notatall.Therearemanyreasonstoincludeatleastsome
suchitems.Veryeasyitemscanreflectthefactthatsomerelativelystraightforwardconceptswere
taughtwellandmasteredbyallstudents.Similarly,aninstructormaychoosetoincludesomevery
difficultitemsonanexamtochallengeeventhebestpreparedstudents.Theinstructorshouldsimply
beawarethatneitherofthesetypesofitemsfunctionswelltomakediscriminationsamongthose
takingthetest.
[materialomitted...]

Conclusion
Tothoseconcernedabouttheprospectofextraworkinvolvedinitemanalysis,takeheart:item
difficultyanddiscriminationanalysisprogramsareoftenincludedinthesoftwareusedinprocessing
examsansweredonScantronorotheropticallyscannableforms.Assuch,theseanalysescanoftenbe
performedforyoubypersonnelinyourcomputerservicesoffice.Youmightconsiderenlistingtheaid
ofyourdepartmentalstudentassistantstohelpwithitemdistractoranalysis,thusprovidingthemwith
anexcellentlearningexperience.Inanycase,anitemanalysiscancertainlyhelpdeterminewhetheror
nottheitemsonyourexamsweregoodonesandtodeterminewhichitemstoretain,revise,or
replace.
Copyright19961999.PublishedbyOryxPressinconjunctionwithJamesRhem&
Associates,Inc.(ISSN10572880)
References:
Aiken,L.R.(1997).Psychologicaltestingandassessment(9thed.).Boston,MA:AllynandBacon,Inc.
Cohen,R.J.,Swerdlik,M.E.,&Smith,D.K.(1992).Psychologicaltestingandassessment:Anintroductiontotestsand
measurement(2nded.).MountainView,CA:MayfieldPublishingCompany.

http://faculty.mansfield.edu/lfeil/201/itemanalysisexplained.htm

3/3