Taking the Case: An Evaluation of Digital Case Study Libraries

Yeong-Tay Sun
Center for HCI at Virginia Tech
2202 Kraft Drive
Blacksburg, Virginia 24060
540-231-7409
YTSun@vt.edu

Laurian C. Vega
Center for HCI at Virginia Tech
2202 Kraft Drive
Blacksburg, Virginia 24060
540-231-7409
LHoby@vt.edu

D. Scott McCrickard
Center for HCI at Virginia Tech
2202 Kraft Drive
Blacksburg, Virginia 24060
540-231-7409
McCrickS@cs.vt.edu

Abstract: The field of Human-Computer Interaction (HCI) is intimately ingrained with the
emerging development of educational resource tools. Case study libraries are one such tool that
have proven to be of great interest due to their adaptability as learning resources. This study
explores the ability of a collection of such case study libraries to support its users, gradually
revealing useful heuristics information derived from design characteristics affecting the usability of
each system. Though current results are preliminary, several interesting questions regarding
usability issues have been raised that may warrant additional assessment.

1. INTRODUCTION
Design, in general, is nebulous. In Human-Computer Interaction (HCI), we design first with the user in mind.
However, even within the domain on HCI there is debate on the 'correct' way to start designing [5]. Therefore, there
are many issues that teachers of design principles (the fundamental rules of interface design) can expect to address,
from choices in development methodologies (such as usability engineering or goal-directed design) to learning
objectives in the classroom. An emergent focus in education is on the design and implementation of useful learning
tools that may help students excel in their understanding of specific topics. Methodologies, while indispensable for
introducing an established curriculum from which to base course material and exercises, are necessarily complex
due to their nature and invariably retain their own deficiencies such as timeliness, lack of regular feedback and lack
of breadth [9]. In light of this, the introduction of tools that could sufficiently support and expedite the study of
different concepts being taught in design courses today would be immensely useful, particularly to novices of those
subjects [1]. The field of HCI, with its goals for improving interface design and implementation, is well-suited to the
development of such tools intended for improving the learning curriculum. Out of the many different tools that have
been proposed, the use of case study libraries have proven to be of some immediate interest.

A case study involves a persistent analysis of a single event, possibly over some period of time. The basic structure
of such a study provides relevant data with some degree of contextual information, while a subsequent analysis of
the case's events eventually arrives at any number of conclusions. As a result, the reader of the case study will be
able to better understand why these events happened as they did, and may also gain insight as to what elements
might warrant a more extensive analysis in future studies. Such cases lend themselves to both the generation and
testing of hypotheses and, where appropriate, have proven themselves to be an immensely useful learning tool due to
their ability to present problems in the form of an intuitive "story-like" narrative or structure [3]. Additional studies
suggest that case-based approaches to learning promote key meta-cognitive skills including cognitive elaboration,
error management, reflection, self-regulation, and transfer of knowledge [3]. Perhaps the most useful feature of a
case study is the ability to simulate problems in a practical, realistic manner. This makes it easier to draw parallels to
real-world situations and enhance the relevance and applicability of the material to both students and instructors
alike. With a solid understanding of the underlying concepts of one case, one might be encouraged to make a logical
leap of intuition to discover parallels between previously unassociated cases [4].

Obviously, a single use-case study would have severely limited applications in both scope and utility; thus we turn
our attention to large collections of multiple use cases, and how such a resource can be beneficial to students in an
academic setting. In the context of this discussion, we introduce an implementation of such a library called the
Usability Case Study (UCS) which serves as a link between collections of relevant case studies and related artifacts
[8]. A prevailing characteristic of each of these cases is their attention to the detailed documentation of the steps and
artifacts generated during a project's conception. These may include requirements stages, task analysis, user-
modeling, various forms of prototyping, evolutionary development, and usability evaluation methods involved in
real-world instances of design examples, ultimately allowing for a prototyping of cases at various levels of
abstraction [3]. In the original study, this tool was employed in the task of supporting specific topics of a usability
engineering course over the length of a semester by illustrating the different aspects of a project's design process. By
the semester's end, reactions towards the UCS were generally positive. In particular, it was reported that students,
following a natural period of adaptation to the system, developed a more sophisticated understanding of usability
engineering practices through their readings of homework assignments and participation in case-based activities [3].
The overall result was a better technical understanding of how human-centered software design works. It should be
noted that this system was not intended as a replacement for regular lecturing or class sessions; rather, it was meant
to support and enhance the comprehension of specific topics so that they might consume less time from other areas
to be covered over the course's duration.

2. BACKGROUND & RELATED WORK

Curiously enough, recent experiences with the UCS aimed at mimicking the findings of this research have produced
results seemingly to the contrary [2]. In particular, anecdotal evidence indicates that these improvements in user
comprehension have been difficult to reproduce under similar settings. An informal study, which also utilized the
UCS to incorporate use cases into the design curriculum of a usability engineering class, was conducted over the
better part of a regular semester course. At the semester's end, it was generally determined that user satisfaction with
the system was somewhat lower than expected. In particular, it was remarked that the UCS proved to be overly
intimidating to users who were unfamiliar with the system, thereby lengthening the time of adoption necessary
before producing useful results [3]. Another negative report indicated that while many different pieces of a particular
case study were present in some form, explanations of how each component fit with the others, or even explanations
regarding an individual artifact's own significance, were minimal; that is, very little guidance was provided and it
was ultimately up to the user to form their own interpretations. The effects of this might be negligible on students
who were already familiar with a given concept or topic being studied, but proved to be an annoyance and possibly
an unhelpful distraction to students who required further assistance. A related complaint was that the degree of detail
present in the case studies, though organized in a relatively straightforward manner, nevertheless made it a daunting
and time-consuming task to sift through an individual case study, thereby lessening user enthusiasm towards the
system. In particular, a specific complaint voiced by the system's administrator claimed that the UCS did not
adequately address certain stages of design that were key to developing a comprehensive understanding of the
design process overall. Specifically, as we will also discuss later, the system allegedly failed to represent the
inherent complexity present during the design process by oversimplifying the represented artifacts without
adequately addressing the gradual progression in between. Further, the overall evolution of a project in which the
initial brainstorming of system features are refined through elimination, feasibility testing, and requirements
modifications was noticeably sparse, and so the representation might not be as true to life as could be expected.
Consequently, this could have the unfortunate effect of misrepresenting the design process overall. One could
further observe that many user sentiments regarding their experiences with the UCS actually coincided somewhat
with the initially negative reactions expressed in the original study, though the end result was certainly different. It is
also interesting to note that despite the fact that both studies initially encountered user complaints, there was a
fundamentally important distinction between the two groups. The complaints from the original experiment seemed
to be aimed at inefficiencies and missing features in the system, or self-diagnostic issues. In contrast, students from
the most recent study seemed to dismiss, or were genuinely baffled by the integration of the system as part of the
curriculum in the first place; that is, they couldn't quite seem to grasp the relevance of the system in the context of
what they were learning.

While this experience has certainly formed a point of contention, it should be emphasized that at this time we are not
attempting to make any specific assertions regarding the ability of the UCS to support its users (as there is
insufficient evidence to make such a contention). Obviously, there are quite a few different plausible explanations
for this phenomenon. The fault may lie with some previously undetected differences between the two groups
exposed to the system, or it may be with the individuals conducting the studies. However, there is also the distinct
possibility that the issue may lie within the fundamental design of the system itself. With these results as inspiration,
there are several courses of action that we could take. For example, we could compile the lists of reactions and
arguments from the system's users and try to incorporate these suggestions into the revision of a newer, better
system. However, the problem with the UCS may be a problem in general with case study libraries. At this point in
our research we would like to reassess case study libraries - in general - to generate and collect some heuristics for
designing systems like the UCS. That is to say, based on information collected and analyzed on our own accord, we
would like to have some assurance that the grievances of the users of the system are justified (and perhaps correlated
in some manner).

2.1 Test Libraries

To this end, a brief evaluation of three different online case study libraries incorporating useful functions or
distinguishing characteristics was conducted. Each of these libraries has satisfied our criteria for a successful
implementation of a use case library: they have either been integrated into the regular learning curriculum in an
academic setting, and/or they have been featured prominently as a section of a publicly accessible website with
relatively high visibility for some period of time. The focus of each library in itself is not especially significant, but
collectively they serve as a representative sampling of actual case study libraries currently in use. Despite their
differing subject matters, each website shares a certain number of similarities as well as unique differences, both of
which we would like to examine in detail.

2.1.1 Pathological Case Database

http://path.upmc.edu/cases.html
This library incorporates a fairly large database with over 480 cases represented, providing entries from the last
decade or so. Photographs or diagrams of pathological data (cultures, etc.) are depicted, as applicable or appropriate.
Diagnoses compare impacted organs of a patient to healthy, normally functioning model. A wide variety of defects
requiring different degrees and techniques of diagnosis are presented. An interesting navigational element is
incorporated by splitting each case study into two parts. The first part includes all relevant details of a case, along
with a preliminary diagnosis and other relevant information. At the bottom of each page is a link to the second half
of the case study which presents a detailed "Final Diagnosis" of a patient's condition. Presumably, the separation of
these two sections is to encourage critical thinking from the user by giving them the opportunity to speculate on their
own diagnosis based on previously available information, rather than digesting all of the conclusions in the same
motion. This library lacks powerful search and categorization tools.

UCS Similarities: Complex Search w/ advanced parsing, Mainly sidebar-driven, High-level context provided as
initial view, Standard template format, Multimedia, Well-structured artifacts, Consistent formatting, Real-world
examples

2.1.2 TED Case Studies

http://www.american.edu/TED/class/all.htm
This is a 14-year archive of cases (consisting of two collections bi-annually) covering trade issues that affect the
environment on a global scale; many countries are covered. Case studies are presented as a rather lengthy articles of
text; multimedia is incorporated as applicable. The large amounts of information draw parallels to real-world
scenarios where it may not be immediately obvious as to what is relevant and what isn't. Each case is generally
divided into sections based on a unified template, while mnemonics are used for easy case identification. On closer
inspection, there seems to be a great diversity among the authors/submitters for the cases included in this library.
Most likely, this is a culmination of resources that was taken from local sources and adapted to the format of this
library. Although the issues of quality control and content review inevitably come into play at some point, this is an
excellent way of keeping the pool of case studies diverse and relevant.

UCS Similarities: Key scenarios essential to case comprehension appear in their own section, along with relevant
supporting materials, Standard template format, Multimedia, Artifact interpretation is open, Consistent formatting,
Real-world examples

2.1.3 Case Law Database

http://www.cre.gov.uk/legal/casedatabase.html
These case studies span a number of years, with landmark cases being clearly delineated. Key points of the law that
factor heavily into the subsequent ruling of a case are outlined, while a summary of rulings appears towards the end;
this includes related laws and arguments influencing a decision. Cases are divided into seven sections according to
the most common groupings of cases. The body itself incorporates a combination of both bullet-style lists and
paragraphs of text, selecting the most suitable one for a given sub-section. Summaries, by nature, tend to emphasize
the notable points of an argument and would be better suited to a list-style layout. Case backgrounds, besides giving
specific information, are also meant to establish a case's context and are written in paragraph form to maintain their
"narrative" style. Besides contextual awareness this variety discreetly separates the text into manageable chunks,
resulting in an immensely more readable page.

UCS Similarities: Complex Search w/ advanced parsing, Mainly sidebar-driven, High-level context provided as
initial view, Standard template format, Well-structured artifacts, Real-world examples

2.1.4 Discussion

You may have noticed that none of the selected case study libraries are of a usability engineering-centric, or even
particularly technical (computer-wise) nature. This is mainly because the characteristics that we are attempting to
gather should be those that are common among all successful case study libraries. If we were to restrict our
observations to libraries that focus only on similar subject matters, we might very well be overlooking some
essential feature of a successful library, perhaps one with a predisposition towards not appearing in a technically-
oriented case library. Secondly, it is generally assumed that observations between similar libraries have already been
conducted to some degree (and likely in the original design stages of the UCS as well). By observing these common
characteristics we wish for the ability to recreate particular features from scratch based on their demonstrated worth,
rather than drawing on inspiration from a particular source.

3. EXPERIMENTAL PROCEEDINGS

3.1 Purpose

Such case libraries are only useful when they are both easy to use, and helpful for those individuals who will be
using them. The functional possibilities for access methods are many and varied. Certain preexisting notions of
optimal usability and interface layout certainly exist, but it is often difficult to discern exactly what elements in a
given implementation make one user interface inherently more usable over another. Additionally, a virtually
limitless number of factors may contribute to the success of a case study library such as access demographics, data
presentation, available tools, relevance of included material, quality of content, and so on. It is fairly self-evident
that even subtle changes to the user interface of a software program can have drastic effects on its subsequent
usability, for better or for worse. Alterations in the structure, length, emphasis, and many other components of an
individual use case can produce similar effects, and many parallels can be drawn between the two. Since we wish to
explore the basic components of a use case library that can shape a user's experience, both are equally relevant to the
goals of this study, and must be equally assessed to gain a comprehensive understanding of the system.

3.2 Basis

At this point, what would be the best method for gathering information that would provide useful insight into the
effective design and implementation of a use case library? Despite the fact that such skills may only be mastered
through practical experience, virtually all people tend to learn some strategies to improve their design approaches
through experience and feedback [6]. Assuming for a moment that we know nothing about designing a successful
case study resource, we can speculate that the relevant points of interest will most likely be reflected in the many
interactions between a resource and its users. In this way, these points may be revealed through the careful
observation of similar resources that are already successfully in use. A study of this nature would allow us to discern
the most influential and effective aspects of these libraries and incorporate them into our own work. In essence, this
study could be considered to be a momentary step back for a more complete system assessment (to "see the forest
for the trees", so to speak) before proceeding ahead. It is our expectation that the information gathered from this
study will eventually be used to support the development of an enhanced version of the original system that
produces a more desirable result.

3.3 Procedure and Overview

Our experiments involved nine (9) participants, seven (7) male and two (2) female. They were not in any inclusion
or exclusion criteria; however, the intent was to seek participants who were novice designers in computer science.
This was to minimize the effects of participants with past specialization in interface design or usability engineering
which might skew results. Additionally, since the goal of developing this software was to help this audience design
better systems, it was likewise in our best interests to select our participants from this population. Each participant
was randomly assigned a case study library, then participated in a three-part experiment. The first part involved a
series of tasks written on separate sheets of paper, to be completed in order. These tasks, involving a specific use
case library, can be found below in Table 1 and were designed for probing specific aspects of the user's interaction
with the system.

Question Pathology Case Trade Environment Employment Case Law

Studies Database
Starting from a website's "Department of Pathology "The TED Case Studies - An "Employment case law
front page, attempt to Online Case Studies" Online Journal" database"
navigate to the case study
database's Homepage,
titled respectively:
Starting from a case study is categorized under is categorized as being is categorized under "Direct
database's front page, "Molecular Diagnostics": published in January of 1995: Discrimination": involves a
attempt to locate a case involves a man with prior involves the effects of photographer with 22
study which: medical history of Behcet's tourism on the ecosystem of unsuccessful applications for
disease. national parks. promotion.

Attempt to find the Which of the following According to the article, According to the article, how
answers to the following symptoms is not exhibited what island has stopped the many steps are there in
questions (within each by the patient in the initial hunt for seals? identifying race
case study): diagnosis? [thin appearance; discrimination?
impaired vision; increased
blinking; symmetrical
reflexes]
Attempt to find the "Chemistry" data What were two factors that What was the officially cited
 Attempt to find the "Chemistry" data What were two factors that What was the officially cited
answers to the following measurements in this case contributed to the decline of reason for the termination of
questions (within each include which of the rug trades in Iran in the late the applicant's membership?
case study): following? [phosphorus, 1980s?
barium, fluorine, glucose,
potassium]
Identify the number of sections in this case that have appeared in other case studies explored thus far.
From a case study database's Homepage, identify the closest means of providing user feedback. This could be an e-mail
address or a scripted electronic form, for example. If none are available (or visible), navigate the site until one is found.
Table 1: Examples of Questions

Questions could range from something as simple as locating the front page of a case study library from a website's
main page to finding specific pieces of information within a specific case study. The emphasis should be made that
the primary goal was not to test the performance of the user, nor was it to specifically assess the correctness or
incorrectness of their answers, but to develop familiarity with a given case library. Once a participant completed all
of their tasks, they were asked to fill out a short likert-style questionnaire requesting feedback on various aspects of
the system. These questions, involving issues such as system usability and overall user satisfaction, were deemed
appropriate at this time since the participant would now be able to make informed responses based on their recent
experiences. Finally, an additional number of questions were administered verbally, expanding considerably upon
those in part two. These questions informally solicited opinions such as the sort of activities one might use a given
library for, or how certain characteristics of the case studies influenced the participant's impression of the library.
We generally considered this last portion of the experiment to provide the most useful information since we could
assess the participant's opinions of the system directly.

4. RESULTS AND INITIAL ANALYSIS

Based on the averaged results of the electronic survey in step 2 of this study, it appears that the participants
overwhelmingly selected the third use case library (Employment Case Law) as yielding the most satisfactory
usability experience. Coincidentally, it was also the library with the shortest average case length. Conversely, the
second use case library (Trade Environment Database) with the longest average use case lengths also garnered the
lowest average value for participant satisfaction. These results indicate that the density of information plays a
significant role in the usability of such a library. Users appear to be overly intimidated when the amount of text
extends much farther than two full screens. In these situations, the tendency is to visually latch on to anything
distinctive-looking in the page; often something as simple as a bolded line of text is adequate for distinguishing one
section from another. Generally, users reacted poorly to disruptions in formatting consistency across pages, feeling
that that the lack of consistent section headers and formatting requirements was overly disorienting and a fairly poor
design decision overall. The fact that these sections were inconsistent also reflected poorly on the "glance-ability" of
each page. That is, in the case of pages with a high density of presented information, some user felt that it was
generally futile to try and spot the information through sequential visual scanning. Also, this affected the
participants feelings of security in the information being presented. When the formatting was different users were
not sure whether the information was from the same library which raised concerns of being too far off task.

Interestingly enough, it has been observed that many participants tend to reserve the use of search functions as a last
resort. Rather, they initially prefer to scan the presented information visually, and in sequential order from top to
bottom until they decide that this method might be unfeasible. It may be that instead of trying to locate the specific
answer for a question, the participant is trying to assemble a contextual background for the question that might aid
them in locating the answer that much more quickly.

Another common complaint indicated that the overall hierarchy of information for a given case study library was
either so high-level as to be useless, or so dense as to be unusable. Users, apparently, are not adverse to different
views of information - that is, the ability to "drill-down" a particular piece of content to simplify or expand the data
to the desired level of complexity as necessary. At this point this highlights, perhaps, a way to apply Schneiderman's
mantra of “overview first, zoom and filter, and then details on demand” [7].
Most participants revealed that more often than not, the mere inclusion of multimedia elements (such as pictures)
seemed to have a marked effect on the user's reported usability of a case, regardless of whether or not the pictures
actually appeared to have any sort of tangible effect on the structure or flow of the case study. Interestingly, when
asked, participants said that the use of pictures allowed them to understand what the case was about. In a sense, it
allowed them to gain a high level understanding of what the case was about. Similarly, it made the participants feel
that the authors of the case study library took more time in crafting the case study. The participants then went on to
correlate this reaction to the case being better than others.

Other user reactions indicated, with surprising consistency, that they were somewhat skeptical of the legitimacy of a
piece of information retrieved from such a library because of the apparent lack of citations (the Employment Case
Law being the exception). The general sentiment was that there was only a limited trust of the library's information,
and that one could only safely use it in conjunction with a third-party search to confirm its validity; that is, they
wouldn't feel comfortable using it as a standalone resource, although some users decided to trust the given
information based on the subject matter alone.

5. CONCLUSION
This paper has presented a brief introduction to the appeal and usefulness of case study libraries in academic
settings, as well as some motivations and strategies for our heuristics gathering activities. We have also reviewed
some of the preliminary results from the experiments currently being collected for this study. The Case Law library
appears to have garnered the most positive impressions from participants by far, thanks to its concise narrative,
consistent layout and plain language. The "glance-ability" of this library was also improved by the subdivision of the
main body of text into a series of easily accessible sections. The pathology library elicited the most neutral responses
from experimental participants. It's use cases generally possessed the brevity shared with the Case Law library, but
were hampered by an abundance of technical jargon, with a similar effect on participant performance. Finally,
despite the extensive use of multimedia, the TED library cases proved to be less than popular to participants due to
their incredible length and extensive pretext, marginal search capabilities for a database consisting of well over five
hundred cases, and perpetually shifting page layouts, formats, colors, and virtually everything else from case to case.

6. FUTURE WORKS
By analyzing this information, we wish to build a set of heuristics that will be indispensable in our future studies of
educational resources, such as case study libraries, design and usability. Specific topics pending full analysis for
future research may include the notion of "browsing vs. searching", the use of multiple data hierarchies to organize
and streamline data access, the role of information density on a user's usability experience, and the role of trust as it
applies to use case resources. Eventually, we wish to augment the Use Case System to a state where its functionality
will be equivalent to the results found in the original study, and will serve as a useful learning tool for students in
their study of the principles of design. From there, we wish to continue building on the successful features of the
UCS, continuing to address issues that the original study may have overlooked.

7. REFERENCES

[1] Ardito, C., M. D. Marsico, et al. (2004). Usability of E-learning Tools. Proceedings of the Working
Conference on Advanced visual interfaces. Gallipoli, Italy, ACM Press: 80-84.

[2] Berry, B., L. Hobby, et al. (2006). Exploring Benefits of Visualization for Case Studies. World Conference
on Educational Multimedia, Hypermedia and Telecommunications 2006. P. Kommers and G. Richards.
Orlando, FL, USA, AACE: 485-492.

[3] Carroll, J. M. and M. B. Rosson (2005). "A case library for teaching usability engineering: Design
rationale, development, and classroom experience." Journal of Educational Resources in Computing 5(1):
3.
[4] Gick, M. L. and K. J. Holyyoak (1980). "Analogical problem solving." Cognitive Psychology 12: 306-355.

[5] Harrison, S., M. Back, et al. (2006). "It's Just a Method!": A Pedagogical Experiment in Interdisciplinary
Design. Proceedings of the 6th ACM conference on Designing Interactive Systems. University Park, PA,
USA, ACM Press: 261-270.

[6] Lewis, T. L., M. B. Rosson, et al. (2004). What Do The Experts Say?: Teaching Introductory Design from
an Expert's Perspective. Proceedings of the 35th SIGCSE Technical Symposium on Computer Science
Education. Norfolk, Virginia, USA, ACM Press: 296-300.

[7] Schneiderman, B. (1998). Designing the User Interface: Strategies for Effective Human-Computer
Interaction. Reading, MA, Addison Wesley.

[8] "Usability Case Study Library (UCS)." from http://ucs.ist.psu.edu/.

[9] Wright, T., J. Noble, et al. (2005). Using a System of Tutorials and Groups to Increase Feedback and Teach
User Interface Design. Proceedings of the 7th Australasian Conference on Computing Education.
Newcastle, New South Wales, Australia. 42: 187-192.