Test Development

Linear On the Fly Testing (LOFT) for the

Common Admission Test
www.prometric.com
This year, through an important enhancement to the Common Admission Test (CAT), the Indian
Institutes of Management (IIMs) will once again raise the standard for business school admissions.
This change will keep the IIMs at the forefront of innovation and further enhance the global reputation
of the Institutes and their graduates. Building upon the strong foundation established previously
through computerization, this years CAT will introduce Linear On the Fly Testing (LOFT), a proven
delivery model that improves exam validity, security and fairness.
What it is
Unlike less sophisticated test construction techniques that simply randomly assign test questions to
each candidate, LOFT is a dynamic forms generation model that utilizes advanced computer algorithms
and item characteristics to custom-build each exam. A randomly assembled exam can vary dramatically
from one candidate to another, leading to serious concerns about fairness and validity. LOFT, on the
other hand, carefully balances each candidates exam to ensure proper content coverage and overall
difculty, enabling reliable comparisons of performance from one candidate to another.
In the case of the CAT, subject matter experts (SMEs) have been hard at work for the past few years
building a thorough and robust item bank of thousands of tested, psychometrically sound and valid
questions that can be applied to this model of testing. Concurrently, the method adjusts the item
selection routine to account for item exposure, making the memorization of signicant portions of the
overall exam extremely difcult. The LOFT process helps ensure that each candidate receives a virtually
unique and individualized exam.
How it works
LOFT is a proven approach to constructing high-stakes, secure examinations with a large volume of
candidates. With LOFT, a test is constructed by selecting items from the bank each time an examinee is
administered the exam. The number of possible forms is limited only by the size of the item bank and
the requirements imposed on the construction process, such as item difculty and number of content
domains and subdomains. Because of the large number of forms, test security is less of an issue than in
traditional form-based construction, where a small number of xed forms are administered to the entire
population of candidates.
www.prometric.com
In the simplied example above, you can see that each candidate receives a unique exam, with one test
question assigned from each difculty level across all of the available content areas. The system
automatically takes into consideration the number of times each item has been presented previously,
and adjusts the selection routine accordingly.
LOFT is a more dynamic form of testing in that it can produce, on the y, signicantly more exam
forms than traditional xed form tests that must be published in advance by SMEs. LOFT capitalizes
on the fact that forms can be designed to have equivalent cut-scores before they are ever administered to
candidates (pre-equating), given that a sufcient bank of quality items exists. However, rather than build
the test before releasing it to the eld, an entire pool of items is released to the eld, accompanied by
computer code to intelligently design a new form every time a candidate takes the test at the test site.
Because this code depends on pseudo-random components (e.g., randomly selecting one of several
items that have the same difculty), there is no complete guarantee that every candidate will receive a
completely different form. Yet because of the randomness, the vast majority will receive different forms
if the item bank is sufciently robust in terms of number of items available for each content domain/
subdomain and in terms of the range of difculties of these items. Because all the items are pre-calibrated
(have established statistics), they are linked psychometrically in a legally defensible manner.
The ability to select items based not only on content but also on statistical parameters independent of
xed forms on which items appear and independent of a particular candidate group taking pre-specied
(xed) forms is facilitated by use of probabilistic modeling methodologies, generically referred to as Item
Response Theory (IRT), rather than the Classical Test Theory (CTT) methodologies frequently used with
other exam programs.
2
Each candidate gets randomly presented with a pre-set number
of questions from each difficulty level bucket from the item
bank. Imagine this diagram multiplied by thousands of items.
Each candidate gets a random, fair and completely unique exam.
CANDIDATES A, B AND C GET RANDOMLY ASSIGNED
ONE QUESTION FROM EACH DIFFICULTY LEVEL.
CANDIDATE A
Easy Q1
Medium Q2
Difficult Q3
CANDIDATE C
Easy Q2
Medium Q1
Difficult Q2
CANDIDATE B
Easy Q3
Medium Q3
Difficult Q1
MEDIUM-DIFFICULTY
Bank of Items
Q1 Q2 Q3
DIFFICULT
Bank of Items
Q1 Q2 Q3
EASY
Bank of Items
Q1 Q2 Q3
Benefts
While LOFT is not appropriate for every exam program due to requirements on size, diversity and
pre-calibration of the item bank as well as the size of the candidate population, for exam programs
meeting those requirements, the benets of LOFT are numerous.
+ Unique test experience for every candidate: While it is certainly possible that two or three individuals
out of 300,000 would end up with the same exact test, it is a virtual guarantee that no one else in
your test center will. The exam a candidate gets is completely and totally individualized.
+ What does this mean? It means that memorization of any exam would be extremely difcult,
ensuring that legitimate, honest candidates are given a fair test experience (and that potential
cheaters have no advantage).
+ Flexibility in scheduling: Since the inherent nature of LOFT item banks is that they need to be much
larger to support this testing model, there is a substantially reduced risk of any overexposure. Since
the IIMs dont have to worry about how many people see any one exam question, they are able to
sustain a few weeks long testing window without compromising item security.
+ What does this mean? A longer testing window means candidates have a wider choice and more
exibility in scheduling dates, time slots and seats for their exams.
+ The test and scores for legitimate, honest test takers are fair: Since item exposure is controlled and
minimized, concerns on validity of scores that may be due to xed sets of items becoming known in
the community of test takers is also minimized.
+ What does this mean? It is a guarantee that those candidates who achieve a good score on the exam
did so fairly, without prior knowledge of any question or test. This really sets apart those top
performers from the rest.
+ The use of IRT statistics to create dynamically-built forms for each candidate increases the measure-
ment precision for the knowledge and skills required of the successful test taker.
+ What does this mean? A deeper and closer evaluation of skill and knowledge for each individual
test taker, based on the questions they answer as part of their randomly generated exam form.
Again this sets apart the top performers from the rest statistically.
LOFT is considered to be an advanced test delivery methodology. Its use during the next CAT adminis-
tration will add yet another layer of security, validity, reliability, legal defensibility and exam soundness
to this widely important test. LOFT represents the natural evolution of the CAT exam as it continues to
carve out its place in history as the single largest test administration in the shortest period of time.
With each technology advancement and enhancement, the IIMs takes one step further into 21st century
testing, blazing a trail for all other high stakes, high caliber exams to follow.
www.prometric.com
Prometric, a wholly-owned subsidiary
of ETS, is a trusted provider of
technology-enabled testing and
assessment. Its market-leading test
development and delivery solutions
allow clients to develop and launch
global testing programs as well as
accurately measure program results
and data. Prometric reliably delivers
and administers more than 10 million
tests a year on behalf of approximately
400 clients in the academic, financial,
government, healthcare, professional,
corporate and information technology
markets. It delivers tests flexibly via the
Web or by utilizing a robust network of
more than 10,000 test centers in more
than 160 countries.
Copyright 2011 Prometric, Inc. All Rights Reserved.