HUMAN COMPUTER INTERACTION AND

COMMUNICATION
USABILITY TEST REPORT - EXAMPLE

EXECUTIVE SUMMARY
In February, we conducted a usability study on the new employee website design to evaluate the effectiveness of the
preliminary design before dedicating resources to its development. The study enabled us to gather objective data on
the design and determine how well the design supports our end users’ ability to complete tasks. Data This report
examines data from this study was used toand uses it to recommend design changes as necessary, as well as to
establish a baseline against which future usability evaluations can be measured.

Overall, we received very positive feedback about the new design. However, we also found several areas for
potential improvement. Specific issues that participants struggled with include:

 Navigating to the homepage

 Noticing all links in the left-hand navigation
 Locating the wW-4 form
 Noticing the “News for youYou” section

This report contains detailed information on these issues as well as proposed solutions. The recommended solutions
deal primarily with improvements that can be accomplished through better information design. These changes, while
less invasive and less resource-intensive than changes to base-functionality, still provide a significant return on
investment, offering improvements to the user experience without major modifications to code.
METHODOLOGY
During the tests, we asked participants were asked to perform common employee website tasks such as entering
vacation time and viewing pay advice. The tasks were designed to evaluate how easily participants were able to
complete the tasks using the new design. Details on how the sessions were conducted appear below. More detailed
information on the tasks participants were asked to complete appears in the next section.

Who We Tested

Over the course of two days, we tested a total of 14 participants (plus one pilot study) across varying departments Commented [JE1]: Do you want to say you conducted a
pilot study with ** participants?
and areas of the university. Participant characteristics break downs as follows:

Desired number
Characteristic of participants

Participant type
Pilot 1
Regular 14

Total number of participants 15

Website design
Design 1 7

Design 2 7

Overall characteristics
All participants:
 Are familiar with the university and its processes
 Are comfortable using online tools and websites to manage personal and
business transactions

Age
< 50 years 7
> 50 years 7

Gender
Male 6
Female 9
Participants were divided into two groups, with each group testing different design elements:
 Design 1 contained a “quick links” section and menu items arranged based on use
 Design 2 did not contain a “quick links” section and used menu items that are arranged alphabetically.

What Participants Did

Participants spent approximately 30 minutes with the test moderator on the following:

 Introduction to the session (5 minutes)

o Review and sign recording permissions
o Discussion:
 Participant’s’ experience with usability studies and focus groups
 Importance of their involvement in the study
 Moderator’s role
 The protocol and methodology for the rest of the session (i.e. thinking aloud)
 Tasks (20 minutes)
o Participants were asked to follow the instructions for performing basic tasks.
 Post-test debriefing (10 minutes)
o Participants were asked a series of open-ended questions about their experience and were also
asked to complete a brief questionnaire

To view the forms and materials used during the session, see Appendix A.

What Data We Collected

During the courseAs part of the usability test, we collected both performance and preference data aimed at
answering the following questions:

 How quickly can participants locate the specified link?

 Do participants go straight to the area the link is located, or do they search in other areas first?
 Do participants read the text in the communication channels?
 Can they answer questions based on information from the communication channels?
 How easily do participants grasp the organizational structure?
 How often to do participants need prompting to find something?

The We also collected data onfollowing types of data were collected:

 Performance:
o Errors of omission (leaving something outfailing to perform a necessary step)
o Errors of commission (doing somethingperforming an unnecessary step)
o Number of tasks completed with and without assistance - Two We provided and tracked two levels
of prompting when participants needed assistance were tracked:
 None - Participant completed a task without prompting.
  Try again - Participant completed a task when asked, “Can you think of any other place to
look?”
 Preference:
o Appropriateness of Online Student Center’s functions to users’ tasks
o Usefulness of terms and labeling
o Overall Ease ease of use overall
o Usefulness of terms and labeling

FINDINGS AND RECOMMENDATIONS

Task Overview

Descriptions The tables below showof the tasks participants were asked to complete appear below, along with a
summary of findings from those each tasks:

Task 1
You need to submit a new W-4 form to change
the amount of tax withheld from your pay.
Where would you go to do that?
Findings
8 participants completed the task with ease
6 participants needed prompting or had significant
difficulty completing the task

Task 2 Findings
14 participants completed the task with ease
You want to update your mailing address. How
would you use the employee website to do
that?

Task 3 Findings
14 participants completed the task with ease
You are planning to take a two-week vacation
to Hawaii at the end of April. Where would you
go to enter your vacation?
Task 4
You recently travelled to UCCS for an all- day
meeting. Where would you go to submit an
expense report for the trip?
Task 5
You know that open enrollment is coming up in
the next month or two. Where would you go to
find out more?
Task 6
You want to find your net pay for last month.
Where would you go to find that information?
Task 7
You need to access the HRMS system to
complete a task, w. Where would you go to do
that?
Findings
12 participants completed the task with ease
1 participant needed prompting or had significant
difficulty completing the task
1 participant encountered a technical issue that
prevented the participant from properly completing
completion of the task Commented [JE2]: Is this the only person who couldn’t
complete a task at all? Or does the “significant difficulty”
category above include that case?
Findings
6 participants completed the task with ease
8 participants needed prompting or had significant
difficulty completing the task
Findings
14 participants completed the task with ease
Findings
10 participants completed the task with ease
Commented [JE3]: Can we specify one?
4 participants needed prompting or had significant
difficulty completing the task
Observed Issues - Summary
Major issues are those that caused the users at least moderate difficulty when performing a task and that occurred
more than 50% of the time.

Criticality ranking is determined based on the severity and frequency of the problem. For details on the methods
used to assess this ranking, see Appendix B.

% of participants affected Problem

Issue Notes
by problem severity

Navigating to the Ranked as Sev 1 because navigation is

Total: 57%
homepage contained in left-nav and functionality is
Mockup 1: 29% 1
not dependent on returning to home
Mockup 2: 86%
page.
Noticing all links in the left- Ranked as Sev 3 because users who
hand nav Total: 50% experienced this problem were severely
Mockup 1: 71% 3 impacted in their ability to use the system.
Mockup 2: 29% Resulted in abandoned tasks.

Ranked as Sev 3 because users who

Total: 50%
Locating wW-4 form experienced this problem were severely
Mockup 1: 86% 3
impacted in their ability to use the system.
Mockup 2: 29%
Resulted in abandoned tasks.
Additionally,We also gathered statistics were included on how users reacted to the “News for youYou” section.
These statistics were included to help determine the level of training and/or communication that will be needed to
help maximize the functionality of this section.

% of participants affected Problem

Issue Notes
by problem severity
Ranked as Sev 1 because most users had
Trouble noticing “News Total: 21% a strategy for locating the news via
for youYou” section Mockup 1: N/A 1 other routes.
Mockup 2: 43%

Didn’t think of home Ranked as Sev 1 because most users had

page as obvious place for a strategy for locating the news via
Total: 43%
news other routes.
Mockup 1: 43% 1
Mockup 2: 43%

Thought of home page as Total: 14% Added for comparison of perspective. Commented [JE4]: Not sure what this means.
obvious place for news Mockup 1: 29% N/A
Mockup 2: N/a
Summary of Proposed Problems and Solutions
Problem
Navigating to the homepage
Noticing all links in the left-hand nav
Locating wW-4 form
Trouble noticing “News for youYou” section
Proposed solutions
- Add “Home” link to the left-hand nav.
- Several elements are interfering with users’
ability to notice all of the links in the left-hand
navigation. Those elements and their proposed
solutions are:
o Quick Links
 Move Quick Links to the
bottom of the left-hand
navigation
 Change title of Quick Links to
“Top 5 links”
 Modify Quick Links design so
that it looks more distinct
from the rest of the left-hand
navigation items.
o Design elements of left-hand
navigation
 Create more distinction
between left-hand menu
items so that the links don’t
visually blur together
 Add context sensitive help and cross linking to
help individuals locate specific items
- While this was not a significant issue for the
majority of our participants, we will need to
dorecommend some training and/or
communication around this section to
maximize the functionality
USABILITY FORMS AND DOCUMENTS
Videotape Release Form

I agree to participate in the study conducted and videotaped by the university.

I understand and consent to the use and release of the videotape. I understand that the information and videotape
is for research purposes only and that my name and image will not be used for any other purpose. I relinquish any
rights to the videotape and understand the videotape may be copied and used by the university without further
permission.

I understand that I can leave at any time.

I agree to immediately raise any concerns or areas of discomfort with the study administrator.

Your signature:____________________________________________________

Date:______________________________________________________________

Please print your name: ______________________________________________

Thank you!

We appreciate your participation.

Session Introduction Script
Thank you for agreeing to take part in our research study. Here is a consent form that we ask that you read and sign
before we begin the study. This form lets you know that today’s session will be videotaped. The information on the
video tape will be used for research purposes only and your name and image will not be used for any other purpose.

My name is LeeAnn and I will be giving you some instructions for the test and asking you a few questions when
you’re done. I’ll be working from a script to ensure that my instructions to everyone who participates in the study
are the same.

I’m here to learn what people think about the design of our new employee website. During the session, I will ask you
to use the new employee website to do a variety of things and will observe you while you do them

Please try to think out loud while you’re working. Basically, we want you to narrate whatever is going through your
mind while you’re working through the tasks.

Please know that we’re not testing you, and there is no such thing as a wrong answer. Your doing this helps us
understand what works or doesn’t work about the new employee website design.

One thing to keep in mind is that we’re running the study in one of our test environments so some functionality
might not be fully implemented at this time. That means that if we ask you to do something like update your email
address, you may be able to get to the screen where you would update your email address but you won’t be able to
actually update anything.

By the way, I’m an independent researcher who had nothing to do with the design of the site you’re about to try out.
So please be honest in your feedback – I need to know exactly what you think, not what you think I want to hear.

The whole session will take about 30 minutes. Do you have any questions before we begin?

Great, let’s get started.

You will be working on a total of seven tasks today. I will give you each task one at a time. Once you’re finished, I’ll
ask you a few questions about your experience. If you have any questions as we proceed, please let me know at any
point.

Because we’re asking you to narrate what you’re doing, please be sure to read all of the instructions out loud as you
complete each task. When you’re ready, you can begin.
Commented [JE5]: These two paragraphs aren’t in the
standalone script. Do we need both? If so, see my edits on
that document.
Tasks

1. You need to submit a new W-4 form to change the amount of tax withheld from your pay. Where would you
go to do that?

2. You want to update your mailing address. How would you use the employee website to do that?

3. You are planning to take a two-week vacation to Hawaii at the end of April. Where would you go to enter
your vacation request?

4. You recently travelled to UCCS for an all- day meeting. Where would you go to submit an expense report for
the trip?

5. You know that open enrollment is coming up in the next month or two. Where would you go to find out
more?

6. You want to find your net pay for last month. Where would you go to find that information?

7. You need to access the HRMS system to complete a task, where would you go to do that?
Post Task Interview

 What is your overall impression of the new employee website?

 What did you like best about the employee website?

 What did you like least about the employee website?

 Was there anything about the employee website that you found difficult or confusing?

 If you were to describe the employee website to a friend using a sentence or two, what would you say?

 Do you have any other final comments or questions?

APPENDIX B – CRITICALITY RANKING INFORMATION1
Problem Severity Ranking
Severity ranking Severity description
4 Unusable
3 Severe
2 Moderate
1 Irritant
1
Jeffery Rubin, and Dana Chisnell, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. (Indianapolis: Wiley Publishing,
Inc, 2008.) 262-263
Severity definition
The user either is not able or will not want to use a particular part of
the product because of the way that the product has been designed
and implemented.
Example: The product crashes unexpectedly whenever it is powered on
at altitude.
The user will probably use or attempt to use the product, but will be
severely limited in his or her ability to do so. The user will have great
difficulty working around the problem.
Example: Synchronizing the device to another device can only happen
when certain files are not in use. It isn’t obvious when the files are in
use.
The user will be able to use the product in most cases, but will have to
take some moderate efforts in getting around the problem.
Example: The user can make sure that all complementary applications
are closed while syncing the two devices.
The problem occurs only intermittently, can be circumvented easily, or
is dependent on a standard that is outside the product’s boundaries.
Could also be a cosmetic problem.
Example: The message area of the device’s small screen is at the very
top, dark blue, and often shaded by the frame of the screen.