USABILITY EVALUATION METHODS:

A LITERATURE REVIEW
ANKITA MADAN
CSE Department, Amity University,
Noida, U.P. 201303, India
ankita.mdn@gmail.com
http://www.amity.edu
SANJAY KUMAR DUBEY
CSE Department, Amity University,
Noida, U.P. 201303, India
skdubey1@amity.edu
http://www.amity.edu
Abstract:
Usability is an important factor for all software quality models. It is the key factor in the development of
successful interactive software applications. Usability is the most widely used concept in the software
engineering field and defines the software systems demand and use. Due to such wide importance of this
quality factor various usability evaluation methods are proposed by usability experts and researchers. This paper
presents a comprehensive study of different usability evaluation methods. The objective of this paper is to lay
down the intensive and conceptual study of the usability concepts.
Keywords: Usability; software; system; approach.
1. Introduction
Demand for quality software system is increasing rapidly. But at the same time there is wide range of rejections
of software systems inspite of great expenses on their development. This is due to non-interaction of the system
and the failure of the software system to fulfill their tasks. Usability is a product attribute that influences the
quality of a software system. It is a transient and elusive concept which has various sub attributes related to it for
the explanation of its abstractness. There exists several quality models like McCall (1977), FURPS(1987),
Capability Maturity Models(1989), IEEE(1992), Dromey (1995), ISO(1991,1998,2001), given by researchers
and experts, which accounts usability as an indispensable quality attribute for the development of a quality
software system. Usability is defined as the ease with which a user can learn to operate, prepare inputs for, and
interpret outputs of a system or component (IEEE Std.1061, 1992). Usability correlates with the functionality
of the system and helps in its evaluation. The lack of usability causes failure of the software system that leads to
a substantial monetary loss, user dissatisfaction, staff unproductivity and time wastage. Therefore, usability
evaluation is very important for the process of designing usable software system. But still there are no apt
criteria or models for usability evaluation because of its fuzzy characteristics. Therefore, this paper presents
extensive survey of the usability concepts and evaluation. The tour to the evolution of usability concept and
evaluation methods over the past three decades has been covered in this paper.
2. Usability Models
The usability models are conceptual view which lays down the focus areas to demonstrate the usability of the
existing software. These criteria are helpful in the usability evaluation of the software system. Easons Model
(1984) characterized usability into three sections based on their independency on the platform in which the task
is being performed i.e. Task Characteristics, User Characteristics, System Characteristics and User Reaction
which is variable dependent. Later, Shackel (1991) gave the importance of usability engineering and the
relativity of its concept. He gave the four important characteristics of usability namely effectiveness,
learnability, flexibility, attitude. Nielsen Model (1993) studied and recognized usability as an important attribute
to influence the acceptance of a product. He divided acceptability into practical and social acceptance and
further on gave five sub attributes of usability namely learnability, efficiency, memorability, errors, and
Ankita Madan et al. / International Journal of Engineering Science and Technology (IJEST)
ISSN : 0975-5462 Vol. 4 No.02 February 2012 590
satisfaction. The international organization of standardization gave a model consisting of three basic sub
attributes namely effectiveness, efficiency, and satisfaction (ISO 9214-11, 1998). Moving ahead, ISO 9126
(2001) laid down the following sub attributes of usability namely understandability, learnability, operability,
attractiveness, usability compliance. Usability model and their definitions are mentioned in Table 1.

Table 1. Taxonomy of Usability Models

Model Sub-Attributes Definitions




Eason
Model
(1984)
Task Frequency Number of times a task is performed by a user.
Openness Extent to which a task is modifiable.
User Knowledge The knowledge that the user applies to the task. It may be appropriate or
inappropriate.
Motivation How determined the user is to complete the task.
Discretion The user's ability to choose not to use some part of a system.
System Ease of
learning
The effort required to understand and operate an unfamiliar system.
Ease of use The effort that is required to operate a system once it has been
understood and mastered by the user.
Task match The extent to which information and functions that a system provides
matches the needs of the user.




Shackel
Model
(1991)
Effectiveness It is described as systems performance is better than some required
level, by some required percentage of the specified target range of
users, within some required portion of the range of usage
environments.
Learnability It is the training of users after some specific time from installation of the
system. Also, includes users re-learnability time for training and support
systems.
Flexibility It is the positive changes or variations in the system to the existing ones.
Attitude It is the acceptance of users within their levels of discomfort, tiredness,
frustration and personal effort.




Nielsen
model
(1993)
Learnability The system should be easy to learn and understand. It should be easy for
the user to get their job or task executed using the software system.
Efficiency Efficiency of the system is directly related to its productivity. The more
efficient a system is its throughput is correspondingly high.
Memorability It is best suited for intermittent users. The user can return to the systems
previous state without starting away from the beginning.
Errors The error rate in any system should be less. If any error is occurred, the
system should be able to recover from it.
Satisfaction It is the pleasant feeling that user gets while or after using the system. It
can be observed as likeability for the system and fulfillment of specified
task.

ISO
9241-11
(1998)
Effectiveness It is the performance measure of a system to complete a specified task or
goal successfully within time.
Efficiency It is the successful completion of a task by a system. It relate to accuracy
and completeness of the specified goal.
Satisfaction It is acceptability of a system by the users, in specified context of use.




ISO 9126
(2001)
Understandability The capability of the software product to enable the user to understand
whether the software is suitable, and how it can be used for particular
tasks and conditions of use.
Learnability The capability of the software product to enable the user to learn its
application.
Operability The capability of the software product to enable the user to operate and
control it.
Attractiveness The capability of the software product to be attractive to the user.
Usability compliance The capability of the software product to adhere to standards,
conventions, style guides, or regulations related to usability
Ankita Madan et al. / International Journal of Engineering Science and Technology (IJEST)
ISSN : 0975-5462 Vol. 4 No.02 February 2012 591
3. Literature Survey
Down the time usability has been studied and discovered by great researchers and scholars. It is a multi-
dimensional concept that opens areas for research. It has evolved over time and has got its relevance in many
aspects.
Foley and Van Dam (1982) described it with respect to user interface guidelines as a property of the
syntactic and semantic analysis of a user interface. Smith and Moiser (1984) made the next attempt by
describing it as a product attribute, which defines the concept by naming product or system attributes or
qualities that influence the usability. In the same year Eason supported the view, usability is the question of how
well users can use that functionality. Gould (1985) defined usability as any system designed for people to use
should be built by keeping in mind that it should be easy to learn and remember, it should be useful , it should
contain functions that people really in their work and be easy and enjoyable to use. B. Shackel (1986) gave an
outstanding definition of usability that could be used throughout the system development lifecycle. As per the
description the system must accomplish following criteria: effectiveness, learnability, flexibility and attitude.
Doll and Torkzadeh (1988) presented a model for satisfaction measurement called End User Computing
Satisfaction Instrument (EUCSI). This was used for specific application. Ravden and Johnson (1989) they
presented a usability evaluation mechanism, software inspection and gave a detailed checklist of 122 items
divided into 9 dimensions. Igbaria and Parasuraman (1989) considered fun to be very influential in acceptance
of any software system. Booth (1989) thought it difficult to specify and measure flexibility of a system and
believed that being useful should be fundamental to usability, thus he modified Shackels criteria into
usefulness, effectiveness, learnability (or ease of use), and attitude (or likeability). Polson and Lewis (1990)
suggested for the user interface design solutions. They contributed by giving problem solving strategies for
novice users when they interact with the complex interface.
Holcomb and Tharp (1991) presented a software usability model for the system designers to decide which
usability sub attributes should be included. It provides a consistent metric for usability. ISO 9126(1991) defined
usability as a set of attributes that bear on the effort needed for use, and on the individual assessment of such
use, by a stated or implied set of user. Brian Shackel (1991) elaborated the usability concept, usability of a
system or equipment is the capability in human functional terms to be used easily and effectively by the
specified range of users, given specified training and user support, to fulfill the specified range of tasks, within
the specified range of environmental scenarios. Mayhew (1992) reviewed the guidelines and instruction for
user interface development. It also includes the general usability principles which describe the desirable
properties of the interface. Grudin (1992) proceeded towards practical acceptability of the system within the
various categories like cost, support, system usefulness. System usefulness was linked to system in use. The
study was later characterized into usability which referred how successfully a user can use systems
functionality. The reaction of user to the interface can be related to its usability on the basis of efficiency and
efficacy of the interface (Hix and Hartson, 1993). Their classification of usability depends upon learnability,
retainability, initial performance, long term performance, advanced feature usage, first impression and
prolonged user satisfaction. Nielsen (1993) presented usability heuristics for the inspection method of usability
evaluation to check the usability principles of the software system. The usability principles of the heuristics
were utilized by the evaluators for examining the interface. According to his classification, usability has five sub
attributes, they are, learnability (easy to learn), efficiency (efficient to use), memorability (easy to remember),
errors (the relevance of catastrophic errors for applications, and satisfaction (pleasant to use). Dumas and Redish
(1993) explained their definition of usability on the basis of focus on users, usability means, use of product by
users for productivity, users are busy people trying to accomplish tasks, decision of user about when the product
is easy to use. Preece et al. (1993) categorized usability into sub attributes namely: safety, effectiveness,
efficiency and enjoyableness. There can be a variation in usability as it depends on users prior experience with
the similar software systems. Later they proposed a new classification composed of learnability, throughput,
flexibility and attitude (Preece et al., 1994). Bevan and Macleod (1994), who discusses the ISO 9241 approach
regarded usability as a property of the overall system: it is the quality of use in a context". Nielsens and
Levys (1994), both worked on user satisfaction assessment of product with the aim of usability evaluation.
Logan (1994) cited the most different concept of usability; he divided usability into social and emotional
dimension. In the emotional dimension, it was explained that a usable product will attract users attention,
enable learning and relive computer anxiety. Caplan (1994) was another expert with a new approach of
usability; he defined apparent usability as an important consideration in the design of a software system. He
defined it as the ease of use that is perceived by a customer upon first looking at a product, but not using it
and the actual usability as the ease of use experienced during operation of the product. Lamb (1995) claimed
usability issues are not restricted to interface usability; it is a wider concept which includes content usability,
organizational usability and inters organizational usability. Guillemette (1995) refers usability to the degree to
which an information system can be effectively used by target users in the performance of tasks. Usability was
later divided into:i) Inherent usability (Kurosu and Kashimura 1995) it is defined as the functional or dynamic
Ankita Madan et al. / International Journal of Engineering Science and Technology (IJEST)
ISSN : 0975-5462 Vol. 4 No.02 February 2012 592
part of the interface usability. The sub attributes focus on how to make the product easy to understand, easy to
learn, efficient to use, less erroneous and pleasurable; ii) Apparent usability (Kurosu and Kashimura 1995,
Tractinsky 1997) - is more related to the visual impression of the interface. Nielsen (1995) presented Discount
usability engineering; Botman (1996) Do it yourself usability evaluation. Butler (1996) dealt with usability
engineering which includes software system models, user models, interfaces model, links between these,
development of the standards and prototyping activities. Harrison and Rainer (1996) reviewed a model used for
computing satisfaction EUCSI, which is defined as an affective attitude towards a specific computer
application by someone who interacts with the application directly. It included following sub attributes:
content, accuracy, format, timeliness and ease of use. Products are tools and high degree of usability can be
determined when the error rate of usability is minimum was given by Kanis (1997) and Hollnagel (1997). Gluck
(1997) correlated Usability to usefulness and usableness. Usableness answers to the question Can I invoke this
function? and Usefulness answers to question Did it really help me? or Was it worth the effort? ISO 9241-
11(1998) is Guidance on usability which discusses usability for the purposes of system requirement
specifications and its evaluation. Lecerof and Paterno (1998) provided a definition addressing importance of a
system to users, efficiency of software system, users subjective feelings, learnability, and a systems safety
feature. Thomas (1998) categorized usability sub attributes into three categories: outcome, which includes
effectiveness, efficiency and satisfaction; process, defines ease of use, interface, learnability, memorability and
error recovery; and task which defines functionality and compatibility. Microsoft also regarded usability as
strategies to attract users reactions for a system and utilizing them into the various development stages (Veldof,
Prasse, and Mills, 1999). However, Head (1999) pointed a simple easy to use interface is the main value point
for usability: the core value of usability is rooted in cognitive science - the study of how people perceive and
process information through learning, the use of memory, and attention. Design guidelines contain instructions
and principles that are required in building of an effective user friendly interface. This was conceptualized by
the methods described into five categories: design rules, ergonomic algorithms, style guide, standards and
collection of guidelines (Vanderdonckt, 1999). Usability concept was regarded for pedagogical value as an
important for e-learning systems. Hence, improvement based on the UE results made the systems more usable,
still such a system may not have any pedagogical sense (Squires and Preece, 1999). In technical writing, clear
and accurate definitions are critical (Alred, Brusaw, and Oliu, 2000) and there is a requirement of concise
usability definition as it is potentially affected by technical/system and human factors. Arms (2000) stated that
usability comprised of aspects including interface design, functional design, data and metadata, and the
computer systems and networking.
Whitney Quesenbery (2001, 2003, and 2004) says the five Es of usability which include effectiveness,
efficiency, engagement, error tolerance, and ease of learning. This explained the requirement of an interface
design that must be easy to learn, remember, and use, with few errors for its implied users and the tasks that it is
assigned to use support by (Battleson, Booth, and Weintrop, 2001). The concept of web usability was described
as, users experience in reading or interacting with a Web site (Brophy and Craven, 2007; Hudson, 2001). The
notion of human-computer interaction extends to Web technology. Turner (2002) illustrated a checklist for the
evaluation of usability. For this usability could be characterized into navigation, design of the page, its
consistency, and content, context of use, accessibility and interactivity. Blandford and Buchanan (2002a) cited
the usability concept as technical, cognitive, and social design based. In the context of web usability, Palmer
(2002) defined usability, which explains ease of navigation for task performance, clarity of interaction, ease of
reading, information organization, speed and layout. The combination of analytical and empirical evaluation
method called systematic usability evaluation was devised for usability measurement (Matera et al., 2002).
Oulanov and Pajarillo (2002) stated for successful communication interface effectiveness is one of the most
important aspects of because it is the medium of interaction. Guenther (2003) marked out the difficulties; he
stated defining usability is complicated. Pack (2003) also added on to this by expressing that the term has
been used so often in so many different contexts, it is in danger of losing its precise meaning. Campbell and
Aucoin (2003) explicitly stated that usability refers to the relationships between tools and their user and it is the
quality of a system that makes it easy to learn, easy to use, easy to remember, error tolerant and subjectively
pleasing. Abran et al. (2003) referred usability as a set of multiple concepts, performance of the system,
execution time of a specified task, user satisfaction and ease of learning. Villers (2004), Dringus and Cohen
(2005) all had common expression, the talked about usability evaluation methods should consider pedagogical
factors also. Hence, the evaluators should take into account learning theory, learning cycle, educational test
research and then use it for e-learning evaluation. Krug (2006) studied usability from the users perspective
based on their experience. Similarly, Dee, and Allen (2006) noted that when an end-user interface is easy to use
and intuitive, it conforms to usability principles. 10 usability factors namely, efficiency, effectiveness,
productivity, satisfaction, learnability, safety, trustfulness, accessibility, universality, and usefulness are
associated with twenty-six usability measurement criteria classified by Seffah, Donyaee, Kline and Padda
(2006). Tom Tullis and Bill Albert (2008) presented Tips and Tricks for Measuring the User Experience this
includes certain points, they are: Know your data; Show your confidence (intervals); Deal with binary success
Ankita Madan et al. / International Journal of Engineering Science and Technology (IJEST)
ISSN : 0975-5462 Vol. 4 No.02 February 2012 593
data (appropriately); Compare means; Consider using expectation measures; Use the System Usability Scale
(SUS); Show frequency distributions; Combine different metrics; Use appropriate tools; Present data
appropriately. Thomas S. Tullis, (2009) explained some of the myths regarding usability, regarded them as Top
Ten Myths about Usability. Gardner-Bonneau D. (2010), he talked about the software systems capability to
sustain the changes in the technical prospects without hampering the usability effectiveness. Jennifer C. Romano
Bergstrom et al. (2011) carried out a demonstration and explained the benefits and challenges faced by the
designers while usability testing of website design. Table 2. gives the comprehensive overview of the usability
concepts.
Table 2. Quick review at the development of usability concept from 1982-2011
Researchers Usability Concepts
Foley and Van Dam
(1982)
User interface guidelines.
Smith and Moiser
(1984)
Described usability as products attribute.
Eason (1984) Interrelated usability and functionality.
Gould(1985) Defined usability in terms of learnability, usefulness and ease of use.
Shneiderman (1986)

Guidelines for error prevention, discussed the systems response time, data entry
within HCI.
Shackel(1986) Defined usability with the factors effectiveness, learnability, flexibility and attitude.
Tyldesley (1988) Mentioned 22 factors that could be used to build the metrics and specifications.
Doll & Torkzadeh
(1988)
End User Computing Satisfaction Instrument (EUCSI).
Ravden & Johnson
(1989)
Presented software inspection as usability evaluation mechanism.
Igbaria & Parasuraman
(1989)
Enjoyability is directly proportional to acceptance of a system
Booth (1989) He modified Shackels criteria into usefulness, effectiveness, learnability, and
attitude.
Polson & Lewis (1990) He gave problem solving strategies for novice users to interact with the complex
interface.
Holcomb & Tharp
(1990)
Presented a software usability model for the system designers to decide which
usability sub attributes should be included.
Brian Shackel (1991) Elaborated the usability concept.
Mayhew (1992) Reviewed usability principles to describe the desirable properties of the interface.
Grudin (1992) Practical acceptability of the system within the various categories like cost, support,
system usefulness.
Nielsen (1993) Presented usability heuristics for the inspection method of usability evaluation. He
classified usability to, learnability, efficiency, memorability, errors, and satisfaction.
Dumas & Redish
(1993)

explained their definition of usability on the basis of focus on users, usability
means, use of product by users for productivity, users are busy people trying to
accomplish tasks, decision of user about when the product is easy to use.
Preece et al. (1993) Categorized usability into sub attributes namely: safety, effectiveness, efficiency
and enjoyableness.
Beimal et al. (1994) Principles of acceptance for usability.
Nielsen & Levy (1994) Worked on user satisfaction assessment of product.
Logan (1994) Divided usability into social and emotional dimension.
Caplan(1994)

Defined apparent usability as an important consideration in the design of a software
system.
Preece et al. (1995) Related usability to overall performance of the system and user satisfaction.
Lamb (1995)

Claimed usability as a wider concept which includes content usability,
organizational usability and inter organizational usability.
Guillemette (1995) Reviewed and defined usability with respect to effective use of information system.
Kurosu & Kashimura
(1995)
Divided usability into Inherent usability and Apparent usability.
Nielsen (1995) Presented Discount usability engineering.
Botman (1996) Presented Do it yourself usability evaluation.
Ankita Madan et al. / International Journal of Engineering Science and Technology (IJEST)
ISSN : 0975-5462 Vol. 4 No.02 February 2012 594
Researchers Usability Concepts
Butler (1996) Dealt with usability engineering.
Harrison & Rainer
(1996)
Reviewed a model used for computing satisfaction EUCSI.
Kanis & Hollnagel
(1997)
High degree of usability can be determined when the error rate of usability is
minimum.
Gluck (1997) Correlated Usability to usefulness and usableness.
Tractinsky(1997) Contributed in explaining the concept of Apparent usability.
Lecerof & Paterno
(1998)
Declared functionality being essential to usability.
Thomas (1998) Categorized usability sub attributes into three categories: outcome, process, and
task.
ISO 9241-11(1998) Guidance on usability which discusses usability for the purposes of system
requirement specifications and its evaluation.
Veldof, Prasse, & Mills
(1999)
Related usability, users reaction and system development
Vanderdonckt (1999) Design guidelines and principles to build an effective user friendly interface.
Kengeri et al. (1999) Explained usability using effectiveness, likability, learnability and usefulness.
Squires & Preece
(1999)
Usability concept was regarded for pedagogical value for e-learning systems.
Arms (2000) Aspects of usability that are interface design, functional design, data and metadata,
and the computer systems and networking.
Alred et al. (2000) Related usability to technical/system and human factors.
Battleson et al.(2001) Explained interface design that is easy to learn, remember, and use, with few errors.
Hudson (2001) The concept of web usability was described.
Turner(2002) Illustrated a checklist for the evaluation of usability.
Blandford &
Buchanan (2002)
Explained usability in terms of technical, cognitive, and social design. Also, looked
into the future work on methods for analyzing usability.
Palmer (2002) Explained usability in context of web usability.
Oulanov & Pajarillo
(2002)
Interface effectiveness as one of the most important aspects of interaction.
Matera et al. (2002) Gave Systematic usability evaluation.
Guenther (2003)
Pack (2003)
Illustrated the difficulties in defining usability.

Campbell & Aucoin
(2003)
Explained usability as a relationship between tools and its users.
Abran et al. (2003) Referred usability as a set of multiple concepts, performance of the system,
execution time of a specified task, user satisfaction and ease of learning.
Whitney Quesenbery
(2001,2002, 2003)
Presented the five Es of usability which include effectiveness, efficiency,
engagement, error tolerance, and ease of learning.
Villers (2004), Drigus
& Cohen (2005), Miller
(2005)
Expressed usability evaluation methods should consider pedagogical factors.
Krug(2006) Studied usability from the users perspective based on their experience.
Dee & Allen (2006) End-user interface conforms to usability principles.
Seffah, Donyanee,
Kline & Padda (2006)
Gave 10 usability factors namely, efficiency, effectiveness, productivity,
satisfaction, learnability, safety, trustfulness, accessibility, universality, and
usefulness are associated with twenty-six usability measurement criteria.
Brophy & Craven
(2007)
Explained web usability.
Tom Tullis & Bill
Albert (2008)
Presented Tips and Tricks for Measuring the User Experience.
Thomas S. Tullis
(2009)
Explained Top Ten Myths about Usability.
Gardner-Bonneau
(2010)
Explained the effectiveness sustained by the software system when technical
changes are made to it.
Jennifer C. Romano
Bergstrom et al. (2011)
Conducted iterative usability testing.
Ankita Madan et al. / International Journal of Engineering Science and Technology (IJEST)
ISSN : 0975-5462 Vol. 4 No.02 February 2012 595
4. Usability evaluation
Various methods are available in the literature for usability evaluation like Inspection, DRUM, QUIS, SUMI
MUSIC, Empirical testing.
4.1 I nspection
This method is proposed by Boehm et al. (1976).The users are observer or testing and evaluation of the design
layout of the software system is done by the experts. It provides experts views and opinions which are essential
for development of various aspects of the software system. The two most widely used inspection methods are:
4.1.1. Heuristic Evaluation
It is fast, cheap and easy method to figure out the shortcomings and problems in a user interface design. The
evaluators use the usability principles or the heuristics for its implementation.
4.1.2. Cognitive Walkthrough
This method is based on assessment of the user interface by the experts who consider the opinion and
experience of the users. It is useful in identifying the problems of user interface.

4.2 Empirical testing
This method was proposed by Marciniak, J.J., (2002). It is a lab oriented methodology which accounts user
experience as a requirement for the design and development of the software system. It also examines
performance and attitude of the users involved in testing the system (Lund, A. M., 1997). The naive users are
allowed to interact with the system and the behavior of the user and the systems response is recorded. The
system can either be a prototype or the final product on which the testing is performed. In case of prototype
system, amendments can be made for the successful design of the product whereas in the case of final product
user acceptance can be measured. Accordingly, if required a system can be discarded.

4.3 Metrics for Usability Standards in Computing (MUSiC)
MUSiC (Bevan, 1995; Macleod et al., 1997) was developed at the National Physical Laboratory, UK, for the
purpose of quantitative and qualitative data required to support usability engineering. This method evaluates the
measures of effectiveness and efficiency as follows:

Effectiveness is the capability of a software system to carry out the specified task successfully. It is defined
as a function of two components, the quantity of the task attempted by the users, and the quality of goals
they achieve (Miles Macleod et al., 1998).

Effectiveness = f(Quantity, Quality)

Task Effectiveness (TES), = (Quantity*Quality) %
100
Efficiency and cost of task performance, this can be formulated by calculating the amount of effort put
into that is the basically the input. Hence, two definitions can be generated which are mentioned as:

User Efficiency = Effectiveness where [Task Time= time spent by user to complete the task]
Task Time

Human Efficiency = Effectiveness
Effort

Corporate Efficiency = Effectiveness where [Total cost=cost of labor + cost of resources + cost of
Total Cost training]


Ankita Madan et al. / International Journal of Engineering Science and Technology (IJEST)
ISSN : 0975-5462 Vol. 4 No.02 February 2012 596
4.4 Software Usability Measurement I nventory (SUMI )
SUMI was developed by University College Cork Software Usability Measurement Inventory as part of the
MUSIC project (Kirakowski, Porteous and Corbett, 1992). It measures the quality of the software system from
the end users point of view. It consists of industry standardized questionnaire statements which are answered by
the user according to whether they Agree, Dont Know, or Disagree. This internationally structured 50-item
questionnaire is available in variety of languages for the convenience of the users like in English, German,
Dutch, Spanish and Italian. It is not at all time consuming and takes 10 minutes or so. We can decide to
administer it on paper or on the other hand can decide for the internet, online option.

4.5 Diagnostic Recorder for Usability Measurement (DRUM)

DRUM (Macleod and Rengger, 1993) is a software tool developed at NPL within the MUSIC project, for
usability evaluation. DRUM has a graphical user interface, online context-sensitive help and a comprehensive
user manual (Macleod et al., 1992). It analyses the tests of a product and derives performance-based usability
metrics from the results and send it to the usability engineer. The video session recorded is analyzed in real time
during first pass of recording. The DRUM increases the pace of the analysis greatly and automates the activity
wherever possible. The component of the DRUM, Log Processor provides calculation to its database of
performance measures and performance based usability metrics, which includes task time; snag, Search and
Help Times; Efficiency; Relative Efficiency; Productive Period (Miles Macleod, Rosemary Bowden and Nigel
Bevan, 1998). There is a tabular and graphical representation of the measures and metrics. The report is given to
the products designers who are concerned with the usability defects.

4.6 The Questionnaire for user interaction Satisfaction (QUI S)

QUIS (Chin et al. 1988, Harper and Norman 1993) evaluation is based on the factors mentioned in the User
Evaluation of Interactive Computer Systems given by Ben Shneiderman (1986). QUIS is designed in modular
format to get section wise accessibility as well as to specific aspects. This questionnaire provided is effective in
providing guidance in the design or redesign of systems. It also helps the evaluators find the areas of potential
improvement in the software system. Thereby, it serves as a testing instrument in usability laboratories and
operates on the concrete product features and the user experience.

5. Conclusion
Usability concept has been under focus over the years and has evolved with different definitions by researchers.
Different attributes have been built for a clear view of usability and its aspects. The usability has been
decomposed into several sub attributes which are hypothetical constructs to define the success of a system. User
involvement plays key role in determining the software usability after it has been developed. This paper has
surveyed research papers, published articles and views of usability experts to describe usability models,
usability evaluation methods and has determined the sub attributes of usability which form the basis for the
usability evaluation of the software system. The paper will be beneficial for the both the students and the
researchers who are working in the field of software engineering. There is still a dilemma about appropriate
selection of measurement technique for usability evaluation of a software system. To find such an appropriate
technique is the future scope of this paper.

References:
[1] Alred, G. J.; Brusaw, C.T.; Oliu, W.E. (2000). Handbook of technical writing, 6th edn. New York: Macmillan.
[2] Abran, A.; Khelifi, A.; Suryn; Seffah, Ahmed W. (2003): Consolidating the ISO Usability Models. Proceedings of International
Software Quality Management Conference Springer). Glasgow, Scotland, UK.
[3] Alonso-Ros, D.; Vzquez-Garca, A.; Mosqueira-Rey, E.; Moret-Bonillo, V. (2010): Usability: A Critical Analysis and a Taxonomy
Department of Computer Science, University of Corua, Spain, Int. Journal Of HumanComputer Interaction, 26(1), pp. 5374.
[4] Arms, W. Y. (2000). Digital libraries. Cambridge, Mass.: MIT Pr.
[5] Battleson, B.; Booth, A.; Weintrop, J. (2001): Usability testing of an academic library Web site: A case study. The Journal of
Academic Librarianship, 27(3), pp. 188-198.
[6] Bevan, N. (1995): Measuring usability as quality of use, Software Quality Journal 4, pp. 115130.
[7] Bevan, N.; Macleod, M. (1994): Behaviour and Information Technology, Usability measurement in context Nigel Bevan and Miles
Macleod National Physical Laboratory, Teddington, Middlesex, UK. 13, pp. 132-145.
[8] Blandford, A.; Buchanan, G. (2002a): Usability for digital libraries. Proceedings of the second ACM/IEEE-CS Joint Conference on
Digital Libraries. New York: ACM Press, pp. 424.
[9] Brophy, P.; Craven, J. (2007): Web accessibility. Library Trends, 55(4), pp. 950-972.
Ankita Madan et al. / International Journal of Engineering Science and Technology (IJEST)
ISSN : 0975-5462 Vol. 4 No.02 February 2012 597
[10] Boehm, B.W.; Brown, J.R.; Lipow, M. (1976): Quantitative Evaluation of software quality, international Conference on Software
Engineering Proceedings.
[11] Booth, P. (1989). An introduction to human-computer interaction. Hillsdale, USA: Lawrence Erlbaur Associates Publishers.
[12] Botman, H. (1996): Do-it-yourself usability evaluation: Guiding software developers to usability. Taylor & Francis, London, pp. 59-
66.
[13] Butler, K. A. (1996): Usability engineering turns 10, Interactions, 3, pp. 59-75.
[14] Campbell, K.; Aucoin, R. (2003). Value-based design of learning portals as new academic spaces, In: Jafari, A. and M. Sheehan,
Designing Portals: Opportunities and Challenges, Hershey, PA: IRM Press, pp. 162185.
[15] Caplan, S.H. (1994). Making usability a Kodak product differentiator. In: Wiklund, M.E. (Ed.), Usability in Practice. AP Professional,
NY, pp. 2158.
[16] Chin, J.P.; Diehl, V.A.; Norman, K.L. (1988): Development of an instrument measuring user satisfaction of the human-computer
interface. Proc. ACM CH1'88 Conf. (Washington, DC 15-19 May), pp. 213-218.
[17] De Villers, R. (2004): Proceedings of the annual research conference of the South African institute of computer scientists and
information technologists on IT research in developing countries, Usability evaluation of an e-learning tutorial: criteria, questions and
case study. pp. 284-291.
[18] Doll, W. J.; Torkzadeh, G: (1988): The Measurement of End-User Computing Satisfaction, MIS Quarterly, pp. 259-274.
[19] Dringus, L.P.; Cohen, M.S., (2005): Proceedings 35th Annual Conference Frontiers in Education. FIE'05, an adaptable usability
heuristic checklist for online courses. pp. T2H-6.
[20] Dee, C.; Allen, M. (2006): A Survey of the Usability of Digital Reference Services on Academic Health Science Library Web Sites.
The Journal of Academic Librarianship, 32(1), pp. 69-78.
[21] Dumas, J.S.; Redish, J. A. (1993). Practical Guide to Usability Testing, Ablex Publishing Norwood NJ.
[22] Eason K. D. (1984): Towards the experimental study of usability, Behaviour and Information Technology, 3(2), pp. 133-143.
[23] Foley, J.; Dam, A. (1982): Fundamentals of Interactive Computer Graphics. Rea, Addison-Wesley, USA.
[24] Gardner-Bonneau, D. (2010): Is Technology Becoming More Usable or Less and With What Consequences, Journal of Usability
Studies, 5(2), pp. 46-49.
[25] Gluck, M. (1997): A descriptive study of the usability of geospatial metadata. Annual Review of OCLC Research.
Accessed: October, 2011, www.oclc.org/research/publications/arr/1997/gluck/gluck_frameset.htm.
[26] Gould, J.D.; Lewis, C. (1985): Designing for Usability, Key Principles and What Designers Think, Communications of the ACM,
28(3).
[27] Grudin, J. (1992): Utility and Usability: Research Issues and Development Contexts, Interacting with Computers, 4(2), pp. 209-2 .
[28] Guenther, K. (2003): Assessing Web site usability. Online, 27 (2), pp. 65-68.
[29] Guillemette, Ronald A. (1995). The evaluation of usability in interactive information systems. In Human factors in information
systems: Emerging theoretical bases, Jane M. Carey. Norwood, N.J.: Ablex.
[30] Harrison, A.W.; Rainer Jr, R.K. (1996). A General Measure of User Computing Satisfaction. In Computers in Human Behavior,
12(1), pp. 79-92.
[31] Head, A. (1999): Web redemption and the promise of usability. Online, 23(6), pp. 20-32.
[32] Holcomb, R.; Tharp, A. (1991): Users, a software usability model and product evaluation, Interacting with Computers, Butterworth-
Heinemann, Oxford, UK, 3(2), pp. 155-166.
[33] Hix, D.; Hartson, H.R. (1993). Developing User Interfaces: Ensuring usability through product and process, chap2, Wiley and Sons,
NY.
[34] Hollnagel, E. (1997): Cognitive ergonomics or the mind at work. Proceedings of the 13th Triennial Congress of the International
Ergonomics Association, Tampere Finland 1997, Finnish Institute for Occupational Health, Helsinki, 3, pp. 3-5.
[35] Hudson, L. (2001): From theory to (virtual) reality. Library Journal, 126 (11), pp. 12-15.
[36] IEEE Std. 1061. (1992): IEEE standard for a software quality metrics methodology, New York, IEEE Computer Society Press.
[37] Igbaria, M. and Parasuraman, S. (1989): A path analytic study of individual characteristics, computer anxiety, and attitudes toward
microcomputers. Journal of Management, 15, pp. 373-388.
[38] ISO 9126. (1991): Software Product Evaluations- Quality characteristics and guidelines for their use, ISO DIS 9126.
[39] ISO 9241. (1998): Ergonomics requirements for office work with visual display terminals (VDTs) Part 11: Guidance on usability.
[40] ISO/IEC 9126-1, (2001): Software engineering - Product quality -Part 1: Quality model.
[41] Bergstrom, J. C. R.; Olmsted-Hawala, E. L.; Chen, J. M.; Murphy, E. D. (2011): Conducting Iterative Usability Testing on a Web
Site: Challenges and Benefits Journal of Usability Studies, 7(1), pp. 9-30
[42] Kanis, H. (1997): Usability centered research for everyday product design. Proceedings of the 13th Triennial Congress of the
International Ergonomics Association, Tampere Finland, Finnish Institute for Occupational Health, Helsinki, 2, pp. 153-155.
[43] Krug, S. (2006). Don't make me think: A common sense approach to Web usability. Berkeley, CA: New Riders Publishing.
[44] Kurosu, M.; Kashimura, K. (1995): Apparent usability vs. inherent usability: Experimental analysis on the determinants of the
apparent usability. Conference on Human Factors and Computing Systems. New York: ACM Press, pp. 292-93
[45] Lamb, R. (1995). Using online resources: Reaching for the *.*s. In Digital Libraries95, F. M. Shipman, R. Furuta, and D. M. Levy,
13746. Austin, TX: Department of Computer Science, Texas A&M University.
[46] Leventhal & Barnes, (2009). Defining Usability and Models of Usability Engineering: Process, Products and examples, chapter 3.
[47] Lecerof, A.; Paterno, F. (1998): Automatic Support for Usability Evaluation. IEEE Transaction on Software Engineering, 24(10), pp.
863-888.
[48] Logan, R.J. (1994): Behavioral and emotional usability: Thomson consumer electronics. In: Wiklund, M.E. (Ed.), Usability in Practice.
AP Professional, NY, pp. 59-82.
[49] Lund, A. M. (1997): Expert ratings of usability maxims. Ergonomics in Design. A study of the heuristics design experts consider
important for good design. 5(3), pp. 15-20.
[50] Macleod, M.; Drynan, A.; Blayney, M. (1992): DRUM User Guide. National Physical Laboratory, DITC, Teddington, UK.
[51] Macleod, M.; Bowden, R.; Bevan, N.; Curson, I.; (1997): The MUSiC performance method, Behaviour and Information Technology
16 , pp. 279-293.
[52] Marciniak, J.J. (2002): Encyclopedia of software Engineering, 2, 2nd edn, Chichester: Wiley.
[53] Matera, M.; Costabile, M.F.; Garzotto, F.; Paolini, P.; e Inf, D.E. (2002): SUE inspection: an effective method for systematic usability
evaluation of hypermedia, IEEE Transactions on Systems, Man and Cybernetics, Part A, 32(1), pp. 93-103.
[54] Mayhew, D. J. (1992): Principles and guidelines in software user interface. Prentice Hall, Englewoog Cliffs, NJ.
[55] Macleod, M.; Rengger, R. (1993): The Development of DRUM: A Software Tool for Video-assisted Usability Evaluation National
Physical Laboratory DITC HCI Group Teddington, Middlesex, TW11 0LW, UK.
[56] Macleod, M.; Bowden, R.; Bevan, N. (1998): The MUSiC Performance Measurement Method, NPL, Draft 0..8
Ankita Madan et al. / International Journal of Engineering Science and Technology (IJEST)
ISSN : 0975-5462 Vol. 4 No.02 February 2012 598
[57] Nielsen, J. (1993): Usability Engineering. Academic press, San Diego, CA.
[58] Nielsen, J.; Levy, J. (1994): Measuring usability: Preference vs. Performance, Communications of the ACM, 37 (4), pp. 66-76.
[59] Nielsen, J. (1995): Scenarios in Discount Usability Engineering in Caroll, J.M., (Ed), Scenarios-Based Design: Envisioning Work and
Technology in System Development. John Wiley and Sons.
[60] Oulanov, A.; Edmund, F. Y. Pajarillo, (2002): CUNY + Web: Usability study of the Web-based GUI version of the bibliographic
database of the City University of New York (CUNY). The Electronic Library 20 (6), pp. 48187.
[61] Pack, T. (2003): Fiddling with the Internet dials: Understanding usability. Online, 27 (2), pp. 36-38.
[62] Palmer, J. W. (2002): Web site usability, design, and performance metrics. Information Systems Research, 13(2), pp.151 - 167.
[63] Polson, P.G.; Lewis, C. H. (1990): Theorybased design for easily learned interfaces. Human-Computer Interaction, 5, pp.191-220.
[64] Porteous, M.; Kirakowski, J.; Corbett, M. (1993). SUMI User Handbook. Human Factors Research Group, University College Cork,
Ireland.
[65] Preece, J.; Benyon, D.; Davies, G.; Keller, L.; Rogers, Y. (1993). A guide to usability: Human factors in computing. Reading, MA:
Addison-Wesley.
[66] Preece, J. Y. Rogers; H. Sharp; D. Benyon; S. Holland, T. (1994). Carey, Human-Computer Interaction, Addison Wesley.
[67] Quesenbery, W. (2003): Dimensions of usability. In Albers, M., & Mazur, B., Content and complexity: Information design in technical
communication. Mahwah, NJ: Lawrence Erbaum Associates.
[68] Ravden, S.; Johnson, G. (1989). Evaluating usability of Human computer Interfaces: A practical method. Ellis Hardwood Limited,
New York.
[69] Seffah, A.; Donayaee, M.; Kline, R.B.; Padda, H.K. (2006): Usability measurement and metrics: A consolidated model, Software
Quality Control, 14(2).
[70] Shneiderman, B. (1986): Designing the user interface: Strategies for effective human-computer interaction. Addison-Wesley, Reading
MA.
[71] Shackel, B. (1986): Ergonomics in design for usability. In Harrison, M. D. and A. F. Monk (Ed.) Proceedings of the Second
Conference of the British Computer Society Human Computer Interaction Specialist Group: people and Computers Design for
Usability Cambridge. British Computer Society Human Computer Interaction Specialist Group, New York: Cambridge University
Press, pp. 44-64.
[72] Shackel, B. (1991). UsabilityContext, framework, definition, design and evaluation, in B. Shackel and S. Richardson, Human
Factors for Informatics Usability, Cambridge, MA: University Press, pp. 2138.
[73] Squires, D.; Preece, J. (1999): Predicting quality in educational software: Evaluating for learning, usability and the synergy between
them, interacting with computers, 11(5), pp. 467-83.
[74] Smith, S.; Mosier, J. (1984): Design Guidelines for the User Interface for Computer-Based Information Systems, Bedford, MA: The
MITRE Corporation.
[75] Thomas, R. L. (1998). Elements of performance and satisfaction as indicators of the usability of digital spatial interfaces for
information-seeking: Implications for ISLA. PhD diss., Univ. of Southern California.
[76] Thomas S. Tullis, (2009): Top Ten Myths about Usability, Simmons College.
[77] Tractinsky, N. (1997): Aesthetics and apparent usability: Empirically assessing cultural and methodological issues. CHI97
Conference proceedings, pp. 115-122.
[78] Tullis, T.; Albert, B. (2008): Tips and Tricks for Measuring the User Experience Usability and User Experience, UPA-Boston's
Seventh Annual Mini UPA Conference.
[79] Turner, S. (2002): The HEP test for grading Web site usability. Computers in Libraries 22 (10), pp. 3739.
[80] Tyldesley, D. A. (1988): Employing usability engineering in development of office products. Computer Journal, 31(5), pp. 431-436.
[81] Vanderdonckt, J. (1999): Development Milestones towards a Tool for Working with Guidelines. Interacting with Computers, 11 (4).
[82] Veldof, J. R.; Prasse, M. J.; Mills, V. A. (1999): Chauffeured by the user: Usability in the electronic library. Journal of Library
Administration, 26(34), pp.115-140.



Ankita Madan et al. / International Journal of Engineering Science and Technology (IJEST)
ISSN : 0975-5462 Vol. 4 No.02 February 2012 599