Psychological Science

http://pss.sagepub.com/

Diagnosing Criterion-Level Effects on Memory : What Aspects of Memory Are Enhanced by Repeated
Retrieval?
Kalif E. Vaughn and Katherine A. Rawson
Psychological Science 2011 22: 1127 originally published online 3 August 2011
DOI: 10.1177/0956797611417724
The online version of this article can be found at:
http://pss.sagepub.com/content/22/9/1127

Published by:
http://www.sagepublications.com

On behalf of:

Association for Psychological Science

Additional services and information for Psychological Science can be found at:
Email Alerts: http://pss.sagepub.com/cgi/alerts
Subscriptions: http://pss.sagepub.com/subscriptions
Reprints: http://www.sagepub.com/journalsReprints.nav
Permissions: http://www.sagepub.com/journalsPermissions.nav

>> Version of Record - Sep 9, 2011

OnlineFirst Version of Record - Aug 3, 2011
What is This?

Downloaded from pss.sagepub.com at Isik University on September 4, 2012

Research Report

Diagnosing Criterion-Level Effects on

Memory: What Aspects of Memory Are
Enhanced by Repeated Retrieval?

Psychological Science
22(9) 11271131
The Author(s) 2011
Reprints and permission:
sagepub.com/journalsPermissions.nav
DOI: 10.1177/0956797611417724
http://pss.sagepub.com

Kalif E. Vaughn and Katherine A. Rawson

Kent State University

Abstract
Previous research has shown that increasing the criterion level (i.e., the number of times an item must be correctly retrieved
during practice) improves subsequent memory, but which specific components of memory does increased criterion level enhance?
In two experiments, we examined the extent to which the criterion level affects associative memory, target memory, and cue
memory. Participants studied Lithuanian-English word pairs via cued recall with restudy until items were correctly recalled
one to five times. In Experiment 1, participants took one of four recall tests and one of three recognition tests after a 2-day
delay. In Experiment 2, participants took only recognition tests after a 1-week delay. In both experiments, increasing the
criterion level enhanced associative memory, as indicated by enhanced performance on forward and backward cued-recall tests
and on tests of associative recognition. An increased criterion level also improved target memory, as indicated by enhanced
free recall and recognition of targets, and improved cue memory, as indicated by enhanced free recall and recognition of cues.
Keywords
memory, learning
Received 2/11/11; Revision accepted 4/29/11

Many educators and students believe that testing is merely a

tool to assess current learning. However, a sizable body of literature has shown that testing can enhance learning (e.g.,
Roediger & Butler, 2011). The effects of testing depend on
several key factors, including the format of the practice test
(e.g., recall vs. recognition), the timing of practice (e.g., interval between trials), and the amount of practice (e.g., number of
trials). In the research reported here, the last factoramount
of retrieval practicewas our primary focus.
Traditionally, researchers interested in the effects of
retrieval practice manipulated the number of practice trials
without regard to whether items were correctly recalled on
those trials. More recently, researchers have examined memory as a function of criterion level, or the required number of
correct recalls during practice, to assess the benefits of repeated
retrieval. Karpicke and Roediger (2007) found that final
cued recall improved as criterion level increased. In Pyc and
Rawsons (2009) experiments, participants studied foreignlanguage word pairs via test-restudy practice until items were
correctly recalled 1 to 10 times, depending on condition;
higher criterion levels enhanced final cued recall. In both
studies, correctly recalling items more than once (up to 8 times
in Karpicke & Roediger, 2007, and up to 10 times in Pyc
& Rawson, 2009), as opposed to only once, dramatically

improved final test performance; the absolute improvement in

performance ranged from 12% to 79% across 13 groups in
four experiments.
Although these studies present a consistent pattern of findings, the underlying basis of criterion-level effects remains
unclear. What aspect of memory improves as criterion level
increases? Intuitively, one might expect that criterion level
strongly affects target memory, as targets represent the only
information overtly retrieved during practice. However, it is
also possible that criterion level influences associative memory.
According to the elaborative-retrieval hypothesis (Carpenter,
2009), testing enhances memory by activating semantically
related information and thus elaborating the association between
cue and target during a retrieval attempt. Different semantic
information could be activated with each subsequent retrieval,
forging an increasingly elaborate semantic pathway with additional retrieval routes to the target. Additionally, reactivating
this semantic information on subsequent retrievals could
strengthen pathways between the cue and target. Finally,
Corresponding Author:
Kalif E. Vaughn, Department of Psychology, Kent State University, P.O. Box
5190, Kent, OH 44242-0001
E-mail: kvaughn4@kent.edu

Downloaded from pss.sagepub.com at Isik University on September 4, 2012

1128

Vaughn, Rawson

increasing the criterion level might also enhance cue memory. The associative-symmetry hypothesis (see Kahana, 2002)
assumes that the two constituent items of a paired associate
form one holistic representation in memory, suggesting that any
improvement accruing to target memory will also benefit cue
memory.
Although criterion level may affect target memory, associative memory, or cue memory, prior research has not identified
which of these aspects of memory are in fact affected by criterion level. Prior studies exploring criterion-level effects have
examined only forward cued recall, which potentially reflects
both target and associative memory. Our goal in the present
research was to employ a constellation of measures to diagnose which aspects of memory are affected by criterion level.
Accordingly, we adapted a battery of test formats that
Carpenter, Pashler, and Vul (2006) used to compare the effects
of testing with those of restudy. In their study, participants were
presented with paired associates either for one practice test with
restudy or for restudy only. Participants then completed one of
four final recall tests: free recall of cue words (a measure of cue
memory), free recall of target words (a measure of target memory), forward cued recall (a measure of associative memory), or
backward cued recall (another measure of associative memory).
Across all tests, performance was better following testing and
restudy than following restudy alone. Carpenter et al. concluded
that the advantage of testing over restudy reflects enhanced
memory for targets, cues, and their associations.
Although we attempted to answer a different question than
did Carpenter et al. (2006), we were nonetheless able to fruitfully employ their manipulation of recall-test format to explore
criterion-level effects. We also administered tests of associative
recognition, target recognition, and cue recognition. As we have
noted, gains in forward cued recall (the only measure used in
prior research on criterion-level effects) may reflect gains in
associative memory, target memory, or both. Associative recognition provides a clearer measure of associative memory than
forward cued recall does (see Westerman, 2001) because it minimizes the contribution of target memory. Likewise, target free
recall and target recognition provide clearer measures of target
memory than does forward cued recall alone. Similarly, cue free
recall and cue recognition provide measures of cue memory.
Thus, we aimed to go beyond prior studies by including six
additional measures that would enable us to fully diagnose the
underlying basis of criterion-level effects.
In our experiments, participants were presented with
Lithuanian-English word pairs for initial study followed by
practice via forward cued recall. We assigned each item to one
of five criterion levels (i.e., one, two, three, four, or five correct retrievals of the English target required during the practice
session). In Experiment 1, after a 2-day delay, participants
completed one of four recall tests: free recall of English
words, free recall of Lithuanian words, forward cued recall of
English words, or backward cued recall of Lithuanian words.
Participants then completed one of three recognition tests:

recognition of Lithuanian words, recognition of English

words, or associative recognition. In Experiment 2, only the
three recognition tests were administered.

Experiment 1
Method
Participants and design. One hundred thirty-one undergraduates participated in Experiment 1 for course credit. Criterion
level (one, two, three, four, or five correct recalls) was manipulated within participants, and the format of the recall and recognition tests was manipulated between participants. Participants
were assigned to recall-test groups using a randomized block
design; assignment of participants to recognition-test groups
was counterbalanced within each recall-test group.
Materials. Materials included 50 Lithuanian-English word
pairs (e.g., Lithuanian word: vilkas; English word: wolf),
divided into five lists of 10 words. The range of normative
item difficulty was similar across lists (Grimaldi, Pyc, &
Rawson, 2010). Each word-pair list was assigned to one of the
five criterion levels; the assignment of lists to criterion level
was approximately counterbalanced across participants.
Procedure. Word pairs were presented on a computer one at a
time in random order. Each word pair was displayed for 10 s.
After all items had been studied once, the practice phase
began. During each practice trial, participants were shown a
Lithuanian cue and were allowed 8 s to type its English translation. After an incorrect response, the cue and target were presented for restudy for 4 s, and the item was placed at the end
of the list, to be repeated on a later practice trial. If a response
was correct but the item had not yet reached its assigned criterion, it was also placed at the end of the list, to be repeated on
a later practice trial. Items were not presented again after they
reached criterion. The practice session ended when all items
reached criterion, or after 60 min had elapsed.
Two days later, the test session took place. Participants took
one of four recall tests, with items presented one at a time. On
the test of forward cued recall, Lithuanian words were presented
as cues, and participants were instructed to type the English
translations. On the test of backward cued recall, English words
were presented as cues, and participants responded with
Lithuanian translations. On the test of free recall of Lithuanian
words, participants recalled as many of the Lithuanian words as
possible. On the test of free recall of English words, participants
recalled as many of the English words as possible.
After the recall test, participants completed one of three
recognition tests. Items in the test of recognition of Lithuanian
words (cue recognition) were the 50 Lithuanian cue words and
50 new Lithuanian words, randomly intermixed. Items in the
test of recognition of English words (target recognition) consisted of the 50 English target words and 50 new English

Downloaded from pss.sagepub.com at Isik University on September 4, 2012

1129

Diagnosing Criterion-Level Effects on Memory

words, also randomly intermixed. Items in the test of associative recognition consisted of 50 Lithuanian-English word
pairs; 25 of the Lithuanian words were paired correctly with
their English translations (5 correct pairs from each criterion
level), and the remaining 25 Lithuanian words were paired
incorrectly with an English target from another pair from the
same criterion level. Participants who took the associative recognition test were informed that all pairs contained words they
had studied earlier and that they were to indicate whether each
Lithuanian word was paired with its correct English translation.

Results
We excluded data for participants whose mean number of correct answers across items was less than 75% of the mean criterion level for all items during the practice session (n = 32).1
Final sample sizes in the recall-test groups were similar (forward cued recall of English words: n = 27; backward cued
recall of Lithuanian words: n = 23; free recall of English
words: n = 24; free recall of Lithuanian words: n = 25).
Recall-test performance. Means for recall-test performance
are presented in Figure 1. In the test of forward cued recall of
English words, the same test format used in the practice phase
Cued Recall of English

80

Free Recall of English

70
60
50
40

Final Recall (%)

30
20
10
Cued Recall of Lithuanian

35

Free Recall of Lithuanian

30
25
20
15
10
5
0

Criterion Level

Fig. 1. Results of Experiment 1. The top panel presents mean performance

on the tests of forward cued recall and free recall of English words as a
function of criterion level; the bottom panel presents mean performance on
the tests of backward cued recall and free recall of Lithuanian words as a
function of criterion level. Error bars indicate standard errors of the mean.

(and the final test format used in previous research on

criterion-level effects), performance increased with criterion
level, F(4, 104) = 30.08, MSE = 195.27, p < .001; this finding
replicates the results of previous research. Insofar as cued
recall depends on the association between cues and targets,
this outcome suggests that increasing the criterion level
improved associative memory. As mentioned earlier, gains on
this test may also reflect improvement in target memory.
Indeed, performance on the free-recall test of English targets
also increased with criterion level, F(4, 92) = 3.45, MSE =
205.67, p = .011. Thus, increasing the criterion level clearly
enhanced target memory. Does the gain in forward cued recall
of English words reflect an enhancement only in target memory? If so, performance on tests of cued recall and free recall
of English words would show similar incremental gains as criterion level increased. However, the interaction between criterion level and test format was significant, F(4, 196) = 6.190,
MSE = 200.152, p < .001; forward cued recall of English
words showed greater incremental gains than did free recall of
English words. We attribute these additional gains to an effect
of criterion level on associative memory above and beyond its
effect on target memory.
Backward cued recall of Lithuanian (Fig. 1) also increased
with criterion level, F(4, 88) = 3.61, MSE = 142.91, p = .009,
albeit less than did forward cued recall. This result provides
further evidence for an effect of criterion level on associative
memory apart from any effect on target memory (the overt
presentation of English words in this test minimized demands
on target memory). However, although it is unlikely that performance on this test reflects target memory, it may reflect cue
memory. Free recall of Lithuanian words also increased with
criterion level, F(4, 96) = 4.25, MSE = 93.80, p = .003, but the
interaction between criterion level and test format for recall
of Lithuanian words was not significant, F(4, 184) = 1.033,
MSE = 117.285, p = .392. Thus, it is possible that gains
in backward cued recall did not reflect gains in associative
memory above and beyond gains in cue memory. An interaction may have been masked by qualitative differences in the
functions relating criterion level to performance; that is, in
Figure 1, the curve for backward cued recall of Lithuanian is
decelerating, and the curve for free recall of Lithuanian is
accelerating. The tests of associative recognition, which we
discuss in the next section, provide clearer evidence concerning criterion-level effects on associative memory.
Recognition-test performance. For each participant, we calculated corrected recognition as the percentage of hits minus
the percentage of false alarms. To maintain counterbalanced
assignment of participants from each recall-test group to the
recognition-test groups, we necessarily had to expose some
participants to information on the recall test that would facilitate performance on the recognition test. Accordingly, we
excluded data from 8 participants who completed the forwardcued-recall test (in which Lithuanian words were presented as
cues) followed by the recognition test for Lithuanian and from

Downloaded from pss.sagepub.com at Isik University on September 4, 2012

1130

Vaughn, Rawson

9 participants who completed the backward-cued-recall test

(in which English words were presented as cues) followed by
the recognition test for English. For the remaining participants, recognition-test performance did not differ significantly
as a function of recall group (Fs < 1.91), so we collapsed
across this variable in the analyses we report here. Final
sample sizes were 23 for the Lithuanian recognition-test
group, 26 for the English recognition-test group, and 33 for the
associative-recognition-test group.
As Figure 2 shows, associative recognition increased with
criterion level, F(4, 128) = 10.77, MSE = 249.74, p < .001; this
finding provides further evidence that criterion level strongly
affects associative memory. This evidence is particularly compelling given that associative recognition provides a clearer
indicator of associative memory than does cued recall. Similarly, recognition of English and recognition of Lithuanian
both increased with criterion level (Fig. 2), F(4, 100) = 13.47,
MSE = 35.35, p < .001, and F(4, 88) = 2.97, MSE = 83.70, p =
.024, respectively; these results constitute converging evidence that criterion level affects both target and cue memory.

Experiment 2
Experiment 1 revealed that increasing the criterion level
improved associative memory, as indicated by enhanced performance on tests of forward and backward cued recall and
associative recognition. Increasing the criterion level also
improved target memory, as indicated by enhanced free recall
and recognition of English target words, and improved cue
memory, as indicated by enhanced free recall and recognition
of Lithuanian cue words. Thus, a consistent pattern of results
emerged across the measures of recall and recognition. However, in Experiment 1, recall tests preceded recognition
tests. To ensure that recall did not affect the results for the

recognition tests, we omitted the recall tests and administered

only recognition tests in Experiment 2.

Method
Sixty-nine undergraduates participated in Experiment 2 for
course credit. The overall design was identical to that of
Experiment 1, with the following exceptions: We extended the
total time for Session 1 to 90 min to minimize the number of
participants who did not reach criterion, we omitted the recall
tests, and we increased the retention interval to 7 days to lower
overall performance so that recognition of English targets
would not approach ceiling, as it had in Experiment 1.

Results
Eight participants whose mean number of correct answers was
less than 75% of the mean criterion level for all items during
Session 1 were excluded from subsequent analysis. Final sample sizes in the recognition-test groups were similar (recognition
of Lithuanian: n = 21; recognition of English: n = 19; associative recognition: n = 21). Recognition-test performance for each
participant was computed as in Experiment 1, and means are
reported in Figure 3. The same qualitative pattern emerged
as in Experiment 1: Increasing the criterion level improved performance on the tests of associative recognition, F(4, 80) =
4.26, MSE = 565.52, p = .004; recognition of English words,
F(4, 72) = 3.77, MSE = 85.00, p = .008; and recognition of Lithuanian words, F(4, 80) = 5.96, MSE = 128.29, p < .001.

General Discussion
In two experiments, we investigated which components of
memory improve as a function of increased criterion level.

Recognition of English
Recognition of Lithuanian
Associative Recognition

100

100

Final Recognition (%)

Final Recognition (%)

95
90
85
80
75
70
65
60

Criterion Level

Recognition of English
Recognition of Lithuanian
Associative Recognition

Fig. 2.Results of Experiment 1: mean recognition-test performance as a

function of test format and criterion level. Error bars indicate standard errors
of the mean.

90
80
70
60
50
40

Criterion Level

Fig. 3.Results of Experiment 2: mean recognition-test performance as a

function of test format and criterion level. Error bars indicate standard errors
of the mean.

Downloaded from pss.sagepub.com at Isik University on September 4, 2012

1131

Diagnosing Criterion-Level Effects on Memory

Our results definitively support the conclusion that increasing
the criterion level enhances associative memory, target memory, and cue memory. This is the first investigation of the
underlying basis of criterion-level effects on memory.
Our results suggest theoretically relevant directions for
future research. For example, a perennial issue in the literature
has concerned symmetry versus asymmetry in associative
memory. Kahana (2002) reviewed evidence for the independentassociation hypothesis, which states that pathways between the
two component elements of an association are unidirectional
and independently encoded, and for the associative-symmetry
hypothesis, which states that both constituent items and their
association are encoded as one holistic representation. Kahanas
review of available evidence strongly supported the associativesymmetry hypothesis. By contrast, our results showed some
asymmetry, as increasing the criterion level benefited forward
cued recall more than backward cued recall. However, Kahana
outlined conditions in which associative memory is symmetric
but asymmetries in test performance (e.g., due to a difference in
the strength of memory or the number of preexisting associations for cue and target words) can nonetheless be observed.
Both of these factors likely operated within our paradigm,
assuming our participants memory strength and number of preexisting associations were lower for Lithuanian words than for
English words. Our results suggest that the potential asymmetry
in the effects of criterion level on learning may be an intriguing
direction for future research.
Declaration of Conflicting Interests
The authors declared that they had no conflicts of interest with
respect to their authorship or the publication of this article.

Funding
This research was supported by a James S. McDonnell Foundation
21st Century Science Initiative in Bridging Brain, Mind and Behavior
Collaborative Award.

Note
1.This exclusion caused minimal interpretive difficulty because
criterion level was a within-participants manipulation and because
similar numbers of participants were dropped from the various test
groups; we extended total learning time in Experiment 2 to minimize
exclusion of participants.

References
Carpenter, S. K. (2009). Cue strength as a moderator of the testing
effect: The benefits of elaborative retrieval. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35,
15631569.
Carpenter, S. K., Pashler, H., & Vul, E. (2006). What types of learning are enhanced by a cued recall test? Psychonomic Bulletin &
Review, 13, 826830.
Grimaldi, P. J., Pyc, M. A., & Rawson, K. A. (2010). Normative multitrial recall performance, metacognitive judgments, and retrieval
latencies for LithuanianEnglish paired associates. Behavior
Research Methods, 42, 634642.
Kahana, M. J. (2002). Associative symmetry and memory theory.
Memory & Cognition, 30, 823840.
Karpicke, J. D., & Roediger, H. L., III. (2007). Repeated retrieval
during learning is the key to long-term retention. Journal of
Memory and Language, 57, 151162.
Pyc, M. A., & Rawson, K. A. (2009). Testing the retrieval effort
hypothesis: Does greater difficulty correctly recalling information lead to higher levels of memory? Journal of Memory and
Language, 60, 437444.
Roediger, H. L., III, & Butler, A. C. (2011). The critical role of
retrieval practice in long-term retention. Trends in Cognitive Sciences, 15, 2027.
Westerman, D. L. (2001). The role of familiarity in item recognition,
associative recognition, and plurality recognition on self-paced
and speeded tests. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27, 723732.

Downloaded from pss.sagepub.com at Isik University on September 4, 2012