Sameer Ahuja Thesis

A Tale of Two Sites: An Explorative Study of the Design and
Evaluation of Social Network Sites
Sameer Ahuja
Thesis submitted to the Faculty of the

Virginia Polytechnic Institute and State University
in partial fulfillment of the requirements for the degree of
Master of Science
in
Computer Science & Applications
Dr. Manuel A. Pérez-Quiñones, Chair

Dr. Andrea L. Kavanaugh
Dr. Stephen H. Edwards
July 17, 2009

Blacksburg, Virginia
Keywords: Social Network Software, Usability Evaluation, Social Affordance, Design

Copyright 2009, Sameer Ahuja
A Tale of Two Sites: An Explorative Study of the Design and Evaluation of
Social Network Sites
Sameer Ahuja
(ABSTRACT)
Social Network Sites allow individuals to construct a public or semi-public profile within a
bounded system, articulate a list of other users with whom they share a connection, and view
and traverse their list of connections and those made by others within a system. Such sites
are generally centered around a particular activity, such as maintaining social relationships
or uploading user created content. Increasingly, niche domains such as education, healthcare
and software development have been exploring the creation of social network sites centered
around the activities of the domain. This has led to an increasing focus on the processes
involved in designing and evaluating these sites. We argue that social network sites require
a specialized focus in their design and evaluation on the social utility of the features on the
site. We have created two social network sites for niche communities: Colloki, a conversation
platform designed for members of local communities; and CATspace, a social repository of
Computer Science assignments, designed for use by CS instructors and students. In this
thesis, we describe the motivation, design and implementation of these two sites. We provide
a formative evaluation of these two sites, wherein we evaluate the usability, and study the
perceived social affordances of individual features across the two site. Finally, we discuss
future work towards building a framework for evaluating the social utility of Social Network
Sites at a formative stage.
biji
iii
Acknowledgments
My advisor, Dr. Manuel A. Pérez-Quiñones, has been a source of inspiration for me during
the past two years. I thank him foremost for his wonderfully cheerful, supportive and flexible
nature.
I’ve been lucky to have worked at a close level with Dr. Andrea Kavanaugh and Dr. Steve
Edwards in their respective areas of work. They both have a deep understanding of their
research areas, and it was fascinating to learn from them for the period that we worked
together. I thank both of them for their guidance, for their flexibility, and of course, for
serving on my committee.
Several people helped at different stages for this work to come through in the form that it has.
Ricardo, Ben, Vineeta and Mathew, have all helped with the development of these prototypes
at different stages. Dr. Pardha Pyla endured my old-age jokes and spent hours guiding me.
Fellow colleagues - Joon Kim, Candida and Szu-Chia provided support and guidance; and
Manas, my lab-mate for the better part this work, always kept things interesting at KWII
1132.
Finally, I’d like to thank my parents for making the small and big sacrifices in their lives, so
that I could achieve the small and big achievements of my life.
iv
Contents
1 Introduction 1
1.1 Defining Social Network Sites, Social Affordance and Social Actions . . . . . 2
1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3.1 Why Evaluate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.4 Overview of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2 Related Work 10
2.1 Social Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.1 Tagging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.2 Activity Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.3 User Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.4 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.5 Ratings and Votes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2 Social Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.1 Social Browsing and Search . . . . . . . . . . . . . . . . . . . . . . . 16
v
2.2.2 Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.3 Sharing Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2.4 Collaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3 Local Deliberation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3.1 Low Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.2 Automated relevance algorithms . . . . . . . . . . . . . . . . . . . . . 19
2.3.3 Citizen Participation . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.4 Centralization of content . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4 Computer Science Assignment Repositories . . . . . . . . . . . . . . . . . . . 20
3 Prototypes 23
3.1 Colloki . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.1.2 System Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.2 CATspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.2.2 System Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.2.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3 Social Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.3.1 Tagging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
vi
3.3.3 User Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.3.4 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4 Methodology 49
4.1 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.2.1 Selection of Participants . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.2.2 Usability Lab Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.2.3 Evaluation Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.3 Benchmark Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.3.1 Colloki . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.3.2 CATspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.4 Perceived Social Affordance . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.4.1 Simulated Community . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.4.2 Mapping social features to actions . . . . . . . . . . . . . . . . . . . . 56
5 Results 58
5.1 Overall Usability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.2.1 Tagging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
vii
5.2.3 User Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.2.4 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
6 Discussion and Conclusions 70
6.1 Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
6.1.1 Articulation of connections . . . . . . . . . . . . . . . . . . . . . . . . 70
6.1.3 Task Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
6.2 Evaluation Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
6.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Bibliography 83
A IRB Approvals 84
B Informed Consent Forms 87
C Pre-Evaluation Survey 94
D Colloki Tasks 98
E CATspace Tasks 101
F Colloki Post-Evaluation Questionnaire 104
G CATspace Post-Evaluation Questionnaire 112
viii
List of Figures
3.1 A sample topic page in Colloki . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2 Colloki Data Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.3 CATspace Homepage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.4 Representation of information that the different CATspace views display. . . 34
3.5 Object model and basic design of CATspace . . . . . . . . . . . . . . . . . . 37
3.6 CATspace data model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.7 Interaction between CATspace and Facebook . . . . . . . . . . . . . . . . . . 40
3.8 A user profile on Colloki . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.9 A user profile on CATspace . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5.1 Participant responses for questions on overall usability . . . . . . . . . . . . 59
5.2 Participant responses for questions on tagging . . . . . . . . . . . . . . . . . 61
5.3 Participant responses for questions on activity streams . . . . . . . . . . . . 64
5.4 Participant responses for questions on user profiles . . . . . . . . . . . . . . . 66
5.5 Participant responses for questions on comments . . . . . . . . . . . . . . . . 68
5.6 Participant responses for questions on votes/ratings . . . . . . . . . . . . . . 69
ix
List of Tables
1.1 Selected definitions of social software / social computing . . . . . . . . . . . 3
4.1 Expected mapping of Social Features to Social Actions . . . . . . . . . . . . 55
x
Chapter 1
Introduction
Mainstream Social Network Sites (SNS) have seen exponential growth in recent times. These
sites have become a cultural phenomenon, having managed to attain massive number of
participants and high engagement levels. Facebook is the prime example, and according to
their statistics, more than 3.5 billion minutes1 are spent on the site each day. Teenagers
(Age 12-17) and Gen Y (Age 18-32) are the most engaged demographics for SNS, with the
most recent PEW study indicating that 65% of online Teenagers and 67% of online Gen-Yers
use some form of Social Network Sites [39].
Seeing the growth of SNS, researchers in several domains are starting to explore the impli-
cations and possibilities of social network software in their respective domains. Researchers
see promise in the software’s ability to engage users and build social connections. Several
of these domains are ones that deal with building software for communities of people. Ex-
amples are software development [19], Healthcare [8], Education [58], Local Conversations
[5], Knowledge Communities [26] and so on. Complementary to the research movement,
startups and small companies are continuously developing and deploying beta or even alpha
stage websites that are bringing social networking concepts to new domains, such as travel2 ,
1
Facebook Press Release, http://www.facebook.com/press/info.php?statistics
2
Dopplr, http://dopplr.com
1
Sameer Ahuja Introduction 2
music3 and even research activities4 . Some of these softwares find immediate adoption and
a viral growth from the start, while others are relegated to ‘dead pools’5 , contributing as
prototypes to the global-scale social networking experiment.
1.1 Defining Social Network Sites, Social Affordance

and Social Actions
Before we consider the definition of social network sites, it is important to provide the defini-
tion of social software that we base our context on, and the relationship that exists between
social software and social network sites. The definition of social software varies across litera-
ture from different schools of thought and different research areas. These definitions vary in
the level of abstraction and coverage. Some selected definitions of social software in literature
are provided in Table 1.1.
Fortunately, social network sites satisfy roughly all of these definitions of social software, and
most schools of thought accept them as a kind of social software. boyd and Ellison narrow
down the scope of SNS by providing a definition that separates them from other forms of
social software [11]:
We define social network sites as web-based services that allow individuals to (1)
construct a public or semi-public profile within a bounded system, (2) articulate
a list of other users with whom they share a connection, and (3) view and traverse
their list of connections and those made by others within a system.
The requirement for a social network site to be a web-based service appears to be a rather
restricting technical constraint imposed by this definition, and it might be a consequence of
3
Last.fm, http://last.fm
4
Mendeley, http://www.mendeley.com; and CiteULike, http://www.citeulike.org
5
TechCrunch deadpool, http://www.techcrunch.com/tag/deadpool
Table 1.1: Selected definitions of social software / social computing
Source Definition
Schuler, ’94 [59] (Social Computing) Describing any type of computing
application in which software serves as an intermediary
or a focus for a social relation.
Wang et. al, ’07 [70] Computational facilitation of social studies and human
social dynamics as well as the design and use of ICT
technologies that consider social context.
Forrester Research, ’06 [14] A social structure in which technology puts power in
individuals and communities, not institutions.
Tepper, ’03 [67] Social Software refers to various loosely connected types
of applications that allow individuals to communicate
with one another, and to track discussions across the
web as they happen.
Wikipedia6 In its most common usage, social software means a range
of software programs which are often web-based, and
which allow users to interact and share data with other
users. However, it may also refer to the procedures of
society carried out by people since before the invention
of computers.
the fact that the web is the most ubiquitous technology platform in the world with most
social networking applications being written for it at the time of this writing. A social
networking site can exist over any technological platform that provides convenient access to
the intended users, it does not necessarily need to be a web based application7 .
There are other aspects to the definition that are a subject of debate in the research com-
munity. David Beer [7] argues that by categorizing a broad range of sites as ‘social network
sites’, the boyd-Ellison definition over-generalizes the classification and merges different kinds
of sites together: those that have social networking as the central activity (‘social network-
ing sites’), and those that center around some other activity (Such as video publishing in
YouTube). While the definition is a work under construction, it is representative of our state
of understanding of social network sites. Hence, for the rest of this writing, we will be using
the boyd-Ellison definition of social network sites.
Researchers have studied the social affordances of internet and online systems in the past
[73]. Social affordance, itself, has been defined as the quality of artifacts in any space
physical or virtual, which invite and facilitate social interaction among the participants in
that space [36]. This definition can be limiting in the context of social network sites, as
social interactions are but one kind of social actions. Rating a piece of content on a site,
for example, is not an interaction between two users, yet, it can be construed to be a social
action. Similarly, customizing one’s profile on a social network site is an individual activity,
yet it can be driven by social incentives and lead to social consequences [10]. So for the
purpose of this work, we broaden the definition, to define social affordance as:
The quality of an artifact in any space which invites and facilitates social actions
by the participants in that space.
To define social actions, let’s consider the concept of social capital. Social capital has been
given several definitions in several areas of research [4], and the research community is
7
An example of the same is location-based social networking applications such as Loopt (http://loopt.com)
that provide their interface via native applications on mobile devices.
torn on whether it is a fashionable (and short-lived) term proposed as a cure-all for the
maladies affecting contemporary communities, organizations and societies as a whole or
whether it has more long-term strategic - theoretical as well as applicable - meaning for
sociology and other social-science disciplines [3]. Sidestepping the arguments, let’s consider
the definition provided by James Coleman [57]. Coleman’s definition suggests that social
capital is anything that facilitates individual or collective action, generated by networks of
relationships, reciprocity, trust, and social norms. Based on this definition, we can define
social actions in the context of social network sites as:
Actions that affect the social capital of the community using the site, and this
would encompass the above-mentioned individual actions with social repercus-
sions.
1.2 Motivation
We look at social network sites from the practitioner’s perspective. The overarching problem
is ad-hoc implementation of social networking concepts. It is best defined by the questions
that we do not yet have answers to: How to design the software for social network sites? How
to do the requirements analysis? What are the design patterns to utilize? How to evaluate
the user interface for social network sites? Without an understanding of these high-level
concepts and a rigorous evaluation methodology, most decisions taken in the design and
implementation of software for social network sites are determined by the ‘gut feeling’ of the
practitioners.
Social networking software has an inherent complexity that makes the design and imple-
mentation of social network sites a unique undertaking that might not be best served by
traditional software engineering approaches. This complexity is sociological effects. The
success and growth of such software stems from the continuous tango between technologi-
cal advancement and incremental user participation, and the net product is neither cleanly
social nor technological in nature [9].
Sometimes, the way that the user community utilizes the software can lead to advancements
in the technology. An example of the same is the ‘@’ reply feature of Twitter8 . This feature
lets users address other users of the system in their updates by prefixing their usernames with
the ‘@’ symbol. This syntax came into being when the developers of the application realized
that several users were having directed public conversations by following the @username
convention to address other users in the system. The developers decided to enhance the
application to better support this community-generated convention.
On other occasions, technology can affect the social dynamics of the community, such as in
the case of Facebook introducing the ‘News Feed’ feature. The feature allows people using
the site to view a feed of the activity that people in their social network have performed on
the site in the recent past. This feature met with mass resistance when it was introduced in
2007, but consequently, it was central in driving certain kinds of social interactions amongst
users [12].
It is worthwhile to note that this tango between technological innovation and sociological
effects is not always perfect. Technology can fail to advance with the scale and requirements
of user community, as evidenced by the recent pulling of some Twitter features that the
application’s architecture was not being able to support9 . Complementarily, the user com-
munity can react negatively to technological innovation, as experienced by Facebook when
it rolled out the Beacon advertising platform in 200710 .
8
Twitter (http://twitter.com), is a microblogging service that lets people send and receive brief text
updates from a variety of devices and interfaces
9
Twitter had to remove an option that let users view their contacts’ replies to other people
(http://www.wired.com/epicenter/2009/05/twitter-screw-up/)
10
Facebook Beacon generated so much controversy amongst users that all major advertisers decided to
leave the platform (http://www.techcrunch.com/2007/12/03/more-facebook-advertisers-bail-from-beacon-
plus-new-concerns/)
1.3 Problem Statement
Social Network software, then, is defined not only by its feature characteristics, but also by
how the social context of the community utilizing the software affects its utilization [48]. This
realization leads to a rethinking of the conventional answers to the problems specified above.
Of special concern are the activities performed in software engineering before the software
is first released to the community. For example, how to design the application architecture
to support future features that cannot even be envisioned at the time of design? Or, how
to perform formative evaluations of a social network site, at a stage where it
has not been released and the user community has not started participating,
and get a sense of how successful it would be when released for use by that
community? In this thesis we specifically look at the problem of formative evaluation of
social network sites.
As part of this body of work, we design and develop two niche community social network
sites, Colloki and CATspace; and then perform a formative evaluation of the two softwares
with respect to their usability, and their social affordances. The research contributions of
this piece of work are to provide:
1. Foundational work for a framework for formative evaluation of perceived social affor-
dances of social network sites, provided via the case studies of Colloki and CATspace.
2. A characterization of SNS features and social actions, and a review of the current body
of work on them.
3. Formative evaluation of the usability and perceived social affordance of Colloki and
CATspace.
This work is intended to be one the first in a series of efforts to formalize processes for the
evaluation for social network sites, and towards the formalization of established user interface
patterns for social network sites.
1.3.1 Why Evaluate
But why do this evaluation at all? Or, in other words, why concern ourselves with trying
to evaluate the social utility of the site at a formative stage, when clearly it is not possible
to get a clear measurement of the utility of social features before the sociological effects
associated with the community using the tool start playing their role.
Formative evaluations of social network sites would help reduce the risk of being able to eval-
uate the social utility of the site only when the software has been released to the community
and a substantial amount of activity has been performed on the site. In fact, fundamental
issues with the features of the site might not even let it reach a state where the social capital
reaches a stage that it can start bringing utility to the members of the community. Even
when one considers the agile engineering approach of ‘release soon, release often’ [16], in the
case of social network software being released to a community of users, it is risky to release
the software too soon - depending on the context of the community; if the tool does not have
the appropriate privacy and security tools, or the social features are not engaging enough;
the users might go away disappointed with the software never to return.
We theorize that formative evaluation of social network sites will help find fundamental
problems with the way social features of the site are implemented. This also has a consequent
benefit for releasing changes to an in-production site - such changes can be evaluated using
a similar framework to formative evaluations so as to get a sense of the user response to the
features to prevent situations where new features are met with strongly negative reactions
[65].
1.4 Overview of the Thesis
This thesis is organized as follows. Chapter 2 reviews related work that has studied social
affordance, social features of social network sites and social actions taken by users of SNS. The
chapter also explores the past work on the two niche areas that the prototypes being studied
are involved with; namely, local deliberation and Computer Science assignment repositories.
Chapter 3 presents an overview of the two prototypes, providing detailed description of the
structure and implementation of the two systems, along with a comparison of the manner in
which the two systems utilize social features. Chapter 4 presents the methodology used for
the formative evaluation of usability and perceived social affordance of the two prototypes.
Chapter 5 presents the results of the study. Chapter 6 provides a discussion on the results
of the study, the questions that arise from this work, and possibilities for future work.
Chapter 2
Related Work
In this chapter, we discuss past work done in the area of characterizing social features and
understanding the user incentives around those features. Complementarily, we discuss social
mechanisms and actions that users of social network sites perform using the site’s features,
and their effects on the system. We then discuss the two niche domains of local conversations
and Computer Science (CS) assignment repositories, and provide examples and justifications
wherever possible from past work of the utilization of social network software in the particular
domains.
2.1 Social Features
Past work has looked at different aspects or features of social network sites. Different aspects
of SNS have received varying amounts of focus by the research community. For these features,
researchers have performed characterization of functional aspects, characterization of user
incentives in play, metric studies of the usage of features on mainstream SNS websites, and
so on. In this section we detail some of the significant work done in this direction. This
list of social features is by no means complete, it is a set of the most relevant features in
10
Sameer Ahuja Related Work 11
the context of this thesis, and is an intersection of the most popular social features and the
social features that we have chosen to implement and study in our prototypes.
2.1.1 Tagging
Tagging is one of the features that has received the most attention from the research commu-
nity, and its the only social feature that has a well defined framework for characterization.
As defined by [48], web-based tagging systems let users annotate a particular resource, such
as a web page, a blog post, an image, a physical location, or just about any imaginable ob-
ject with a freely chosen set of keywords (“tag”). We are just beginning to understand how
tagging works [29], its implications, and its possible uses [48]. Tags are even making their
way to personal information management tools, such as Google’s Mail program. Marlow
and colleagues [48] have studied tagging systems and articulated a framework for studying
tagging systems. They provide two organizational taxonomies to characterize social network
systems with support for tagging (they call them social tagging systems):
1. The user incentives at play: Future retrieval of tagged items, contribution and sharing,
attracting attention, play and competition, self presentation, opinion expression.
2. The system design and attributes: Tagging rights, tagging support, aggregation of
tags, type of object being tagged.
The two taxonomies are complementary, because one of them represents the original intent
of the designer of the system (System design and attributes) and can be determined at design
time, while the other characterization (User incentives) is such that it can only be determined
when the software is used by the community. User incentives are affected by personal needs
and social interests. While the design of the system can play a significant role in forming
user incentives, they are also affected by the social norms within the community using the
tool, which is a function of the characteristics of the individuals involved.
2.1.2 Activity Streams
Activity streams are flowing commentaries on users’ actions on the different sections of the
site. They are also known as ‘news feeds’ or ‘lifestreams’, and were first made popular
amongst social sites by Facebook in 2007. Most systems utilizing activity streams show the
subset of the stream that is most relevant to the user’s current view, such as a user’s activity
on his/her profile. Further, different implementations of activity streams can give preference
to a user’s close friends’ on the site in the stream that he/she sees. Some systems let users
delete items off their activity stream to hide some of their actions, while in other systems
the user can only get rid of an activity stream item by undo-ing the action that caused
it. Complementarily, systems can also provide filtering and grouping options for viewers to
group or restrict some kinds of items or items from some people from showing up in their
feed. Some systems also perform analysis on activity streams to present them in more useful
ways. Friendfeed1 , for example, restricts similar activities by a particular user that are visible
in the stream. The remaining activities are provided below the fold by clicking a link. This
prevents one user from dominating the activity stream of events.
Activity streams are an effective way of helping users discover content. In addition, they have
been found to motivate contribution by new users to the system. Burke et. al [12] recently
conducted a study using the logs of 140,000 new users on facebook to determine that activity
streams (Called ‘News Feed’ in Facebook terminology) can be vital for motivating new users
to contribute to the site. They found four social mechanisms at play that encourage new
users to contribute:
1. Social Learning: Newcomers whose friends share more content on the system go on
to contribute more content themselves.
2. Singling out: Newcomers who are singled out in content go on to contribute more
content.
1
Friendfeed, http://friendfeed.com
3. Feedback: Newcomers receiving more feedback on their initial content go on to con-

tribute more content.
4. Distribution: Newcomers whose initial content is distributed widely go on to con-

tribute more content.
Streams are the central vehicle for the above-mentioned social mechanisms in social network
sites. A complementary approach to explicit activity streams worth mentioning in this
context is the one taken by Amazon.com. Amazon records users’ activity streams, but
instead of letting other users view those streams, it instead uses algorithms to analyze those
streams to provide a different construct to its users - ‘Social navigation’ [21]. When users
perform actions on the site, the system provides recommendations for next actions based
on its analysis of past actions of users. These are displayed on the site in the form of
recommendations like “People who bought this book also bought...”.
2.1.3 User Profiles
User profiles are central to the concept of a Social Networking Site, as per the boyd-Ellison
definition [11]. Profiles provide users their identity on the system, and aid in discovery
of common interests between people and in articulating relationships. There is a strong
identity management component attached to user profiles, as detailed by danah boyd in her
ethnographic study of MySpace [10].
User profiles vary in the following three aspects:
1. Customizability of presentation: The customizability of MySpace pages via CSS,

images, videos and audios is a central aspect to its success with the youth demographic
[11]. Facebook, on the other hand, has strict controls in place over the display and
layout of information on the profile.
2. Information: Most SNS let users manipulate what information is provided on their
profiles in the system. Some systems let users import information from other social
network sites or websites into their profiles either as ‘widgets’2 , or as items of activities
in the users’ activity streams3 .
3. Articulated relationships: Different systems articulate relationships on user profiles

in different ways. Some systems such as facebook and MySpace have explicit two-way
relationships (‘friendships’); others like twitter support one-way explicit relationships
(‘followers’); while some systems articulate relationships between users via the content
that they contribute to the system, hence highlighting other users that have liked the
same content as the current user, or other users whose content the user subscribes to.
2.1.4 Comments
Most social network sites let users comment on content shared on the site. Some sites also
provide the ability to comment on user profiles. Comments are the primary conversational
medium on social network sites, and the comments of users on such systems can often express
social relationships [6].
2.1.5 Ratings and Votes
Several social applications utilize ratings and votes as an organizational tool. Amongst the
mainstream websites, digg.com places user votes at the center of the system, using them as a
social information filtering system [46, 47]. Other systems use ratings for personal and social
recommendations [30]. Ratings are also a fundamental component of reputation systems
2
Developer platforms for both MySpace (http://developer.myspace.com) and Facebook
(http://developers.facebook.com) provide for development of widgets that can be displayed on user
profiles
3
Friendfeed (http://friendfeed.com) is a service that lets users import their activities on other websites
into their activity stream on friendfeed
for users of social applications. These reputation systems hold immense utility in certain
areas and they have been found to facilitate social interaction, drive trust and limit aversive
behavior [37]. Ratings can also be used as input for other features on the site - such as
featured content, activity streams, leader-boards, and so on.
The semantics and presentation of voting/rating systems differ across sites. The manner
in which these systems are implemented and presented can have a significant effect on user
behavior. Studies of systems using ratings to derive recommendations for users have found
that such recommendations can have a cyclic effect on the ratings the users give to content on
the site [18] - that is, ratings help provide good recommendations, which encourage further
rating. Rating systems can provide different levels of granularity - ranging from a binary “I
like this” on facebook.com to 10 point ratings on imdb.com. Some systems also provide for
negative ratings, which are different from flagging - negative ratings do not indicate content
that is not acceptable in the system, but rather content that the users dislike. Systems that
support negative ratings can be more susceptible to unfair use [20].
2.2 Social Actions
We’ve defined social actions in Section 1.1 as actions that affect the social capital of the
community using the site. That is a broad definition and there is indeed a broad range of
actions that qualify for the definition at varying levels of abstraction. Forming a taxonomy
of social actions on social network sites is beyond the scope of this thesis. In this section,
we discuss past work on a set of social actions that we utilize as a barebones framework for
our evaluation of social features.
Ellison et al. [25] suggest that these social activities on social network sites provide users with
social capital. While in our definition and discussion we are looking at social actions from
the perspective of helping build social capital, there are other incentives and motivations
for users to perform these actions. Stafford et al. [62] provide a detailed study of “uses
and gratifications” of Internet use, “uses and gratifications” referring to the motivations of
specific uses and the satisfaction people gain from such use. The third form of gratification
from Internet that [62] mention is as a social environment. This is especially relevant in
social network sites, where the social environment aspect is exemplified in addition to the
other gratifications.
2.2.1 Social Browsing and Search
Lampe et al. [43] discuss social browsing and searching in the context of Facebook, but the
concepts are generalizable to the broader range of social network sites. Social browsing refers
to the use of the site for the act of finding people who are not in the user’s social network.
Social search refers to the use of social network site to get more information about people in
the user’s (offline) social network. Different social network environments lend differently to
the above actions - in the case of Facebook, researchers have found users to be more inclined
towards the action of social search than social browsing [38].
2.2.2 Interaction
We include any kind of direct interaction between two or more individuals via the medium
of the social network site in our definition of interaction. Social Network sites can support
interaction in different forms, however, the primary means of interaction is generally com-
ments and posts. Other forms include giving virtual gifts (facebook.com), and ‘nudging’
(twitter.com). These interactions have varying levels of ‘interestingness’, as determined by
the three variables of participation in related themes, participant cohesiveness and theme
diffusion [15]. They affect the social capital of the community in perhaps the most direct
manner of all social actions - interactions are used to strengthen or maintain current rela-
tionships or forge new ones. A study of Facebook usage by Lampe et al. [43] found ‘keeping
up with people’ to be the primary motivation of Facebook use for college students, and that
interaction is one of the primary means for doing so. A study by Golder et al. [32] reveals
how different forms of interaction are utilized for performing different functions on the social
capital by users of a social network site - they report that while the vast majority of messages
are sent to friends (90.6%), a large proportion (41.6%) is sent to friends outside of one’s local
network - indicating that messaging was primarily being used for building new connections.
In comparison, ‘pokes’ (a form of content-free interaction) were primarily exchanged within a
network (98.3% of all pokes were within a network) to maintain or strengthen relationships.
2.2.3 Sharing Content
Several social network sites are centered around the activity of users sharing content -
YouTube and Flickr being the most popular mainstream examples of the same. Even on sites
where sharing content is not the primary activity - such as Facebook or Twitter, content
sharing is one of the most oft-performed actions. Extensive work has been done in the past
to characterize the motivations for sharing content [64, 50, 71, 35]. Stoeckl et al. [64] charac-
terize the motivations into the following categories: Enjoyment, Information dissemination,
Personal documentation, Passing time, External economic incentives, and contact. As with
other forms of social actions, sharing content affects the social capital in varying amounts
and via different means. Users of social network sites use the actions of uploading content,
and controlling its access, to actively maintain and manipulate their social networks [44].
2.2.4 Collaboration
The concept of working together towards a common goal, or Collaboration, has a cyclic
cause-effect relationship with social capital - both are beneficial to each other [17]. An in-
triguing mainstream example of collaboration in online systems is provided by an immensely
successful system that does not precisely fall into our definition of social network sites -
Wikipedia. Wikipedia is a the Web-based user-created encyclopedia that is a prominent
example of a collaborative, user-generated content outlet [1]. Several forms of motivations

play a role in users collaborating to build the site’s knowledge base. One study claims the
most significant motivations for collaborating on Wikipedia content to be Fun, Ideology, Val-
ues, Understanding, Enhancement, Protective, Career and Social [52]. Collaboration also
occurs in several manners on Wikipedia. A central issue of Wikipedia is conflicts that occur
amongst contributors. While these are generally looked at in a negative light, the resolution
of these conflicts becomes a central activity on the site that can lead to positive effects of es-
tablishing consensus and strengthening common values amongst the participants [41], hence
affecting the social capital of the community in a direct manner.
2.3 Local Deliberation
Local action groups are crucial to create awareness and draw average citizens into dialogue
about local issues and concerns in democratic life [40]. These groups are centerpieces of local
democratic activity and act as an intermediary level of organization between the individual
and the government [69]. Most voluntary organizations face the typical challenges of lead-
ership burnout and limited resources [40]. There is growing evidence that information and
communication technology aids in resolution of these problems and increases participation
among the members of these voluntary organizations [40].
Often finding and participating in local conversations is hard to accomplish online. This is
for several reasons, the most important of them being:
1. There are fewer resources devoted by the news agencies to local issues.
2. Online deliberation at the local level often times occurs in particular groups and it is
difficult for online citizens to join and become active participants.
3. Social software systems that allow users to gather online and discuss their interests
(e.g., Digg, Slashdot, and other similar sites) work in part due to the sociological
effects of a large number of people participating and are not as effective when the
social network is small.
2.3.1 Low Scale
While mainstream media and conversations typically involve hundreds if not thousands of
participants, the most active local conversations involve at the most tens of people. With
reference to the social effects mentioned in Section Section 1.2, the lower scale places more
emphasis on the technology to leverage the limited sociological effects of people using the
technology.
2.3.2 Automated relevance algorithms
Local news and events have limited visibility in mainstream media, and any such existence
has limited references to it from local or mainstream media. Current state of the art al-
gorithms for search and discovery depend heavily on reverse-link counts [54]. This places
locally relevant items at a lower priority for such algorithms.
2.3.3 Citizen Participation
Past work [61, 40] talks of the three user classes of leaders, active citizens and lurkers. Theo-
ries on online deliberation have proposed the 1% rule: That only 1% of all users of an online
system contribute content to the system regularly, around 9% contribute content intermit-
tently, and 90% of the users do not contribute content at all4 .
To engage all kinds of users, any system that is based on user generated content should
4
Jakob Nielsen, Participation Inequality: Encouraging More Users to Contribute
(http://www.useit.com/alertbox/participation inequality.html)
provide appropriate features and means for all of the above mentioned user-classes to perform
their typical tasks. This means providing affordances for the social actions that facilitate both
lurking and participation, and providing incentives for participation, such as leaderboards
[33].
2.3.4 Centralization of content
A typical characteristic of local conversations is that they are distributed in nature and often
occur serendipitously [66]. There are several dimensions to why people prefer to post their
opinions at some places and not at others. People might use a medium because they are
familiar to it (such as forums), or because of features like private access (listservs), or just
because they feel that the medium is relevant for them to tell their story (Such as blogs
for personal opinions [51]). That is not to say that there isn’t work being done to build
centralized systems for collaborative deliberation [42]. However, we envision an ideal local
conversation system to be one that does not force people to one centralized conversation
location, but rather provides support for authors to selectively syndicate their opinions and
conversations onto the central system, and to consume content from the system in a similar
manner. It is with this understanding that we first built VizBlog [66], a visualization tool
that picked up conversations occurring in the local blogs of a region. Colloki is an extension
of the work we did with VizBlog, that has the ambitious long-term goal of providing a local
conversation platform which supports syndication from other conversational mediums.
2.4 Computer Science Assignment Repositories
Computer Science programs have seen a growing trend of online tools for managing course
related activities. Such tools include, but are not limited to online automated-grading
systems[24] and online repositories for course syllabi [68]. However, online repositories for
sharing programming assignments among instructors has not seen such progress. The most
notable examples of such efforts are the Nifty Assignments Repository at Stanford University
[2] and the Multimedia Educational Resource for Learning and Online Teaching (MERLOT)
[72].
The Nifty Assignments repository has grown as an effort of professors from different uni-
versities that come together as part of a committee during the annual SIGCSE conferences
(e.g. [55, 56, 2]). Although instructors may expect high quality programming assignments
from this source, they suffer from a relatively slow refinement and improvement process pro-
pitiated by the lack of involvement of instructors other than the authors of the assignments.
Also, the static nature of the list of assignments inhibits the CS education community to
track the use of those assignments and the modifications and improvements made through
time.
MERLOT attempts to improve some of the limitations of repositories like the Nifty Assign-
ments. MERLOT provides an online space where instructors may post their assignments
where they peer-reviewed by the community that they support [60, 34]. Despite MERLOT’s
moderate success, there is a need for online tools that adhere to the particular needs of
programming assignments [23, 53].
Overall, while teachers struggle to find quality programming assignments online [13], none
of the current NSDL collections help. This has led to a situation where most instructors do
not look for new repositories actively, and find the current ones to be of limited use [49].
There are a couple of problems with existing systems that has led to this situation - One, the
basic ‘library model’ that most of the current repositories are based on, does not provide any
greater utility than a simple web search for the assignment. Secondly, there are little to no
incentives for users to actively engage in building the repositories. Hence, most repositories
fail to engage the majority of instructors.
There is scope for an assignment repository to be much more than a digital library. Con-
tent brings with it conversations, collaboration and other social activities centered around
the content; and the repository could provide the social constructs for performing those
actions within the site. A community-based repositories might provide the opportunity for
instructors to connect at a personal level to advance their careers and promote their views
on CS education [22]. The online system could also foster the evolution of programming
assignments towards focusing on more culturally-diverse and socially-relevant problems [45].
Chapter 3
Prototypes
This section details the two prototype applications that were evaluated in the study, namely
Colloki and CATspace. The two systems are being developed for two separate niche com-
munities - Colloki is an open-source conversation tool meant for local communities, while
CATspace is a repository for Computer Science assignments, developed primarily for the
Computer Science instructor community.
The two applications, however, have a couple of similarities that make them worthwhile
to study together for this particular study - One, both the softwares are unique in their
particular domains as they bring social networking concepts to their respective domains.
They use similar features for doing this, however, these features are customized to suit their
particular domains. Secondly, they are both in the same stage of development. Both the
applications are in their first development iteration, and have a basic feature-set that is
complete in terms of essential functionality.
23
Sameer Ahuja Prototypes 24
3.1 Colloki
Local conversations online are an important means of building awareness and increasing
participation amongst citizens. However, most local websites are restricted to using forums or
collaborative blogs as the medium of conversation. Colloki is a local conversation system that
aims to utilize a set of social and organizational features for a more effective conversational
environment. In this section we provide a detailed description of Colloki’s system attributes
and implementation.
3.1.1 Introduction
In Section 2.3, we discuss the importance of local conversation tools for local action groups.
Now while while the mainstream web has seen explosive growth of social software systems in
the past few years, local online deliberation systems are still using the traditional discussion
forums and email listservs. We believe that online deliberation systems for small groups
have very unique design challenges that separate them from mainstream systems, and hence
mainstream social software systems do not translate very well to the local environment.
Colloki is an online “local conversation hub” that we are developing in close collaboration
with several civic organizations in Blacksburg, including Citizens First1 , a grassroots orga-
nization; and Blacksburg Electronic Village2 , which represents the web presence of many
local community groups. This design aims to utilize social features in a local environment
to provide what we believe is a more effective local conversation medium.
1
http://citizensfirstforblacksburg.org
2
http://www.bev.net
3.1.2 System Description
Content Organization: Topics, Tags and Ratings
Colloki lets users explicitly manage their online content for organizational and social pur-
poses. Personal accounts on Colloki support the creation of a social network, currently by
providing means for informal connections formed by commenting or rating others’ content,
and in the future by having people identify their friends online. These articulated relation-
ships are the centerpieces of social network software [11] and are instrumental in building
the social capital [25] . We will use the social connectivity provided by such relationships to
allow events and information to be shared by others.
All of the content within the system is organized in sections of interest called Topics that
are aggregation of particular subsets of information. Topics are usually a small number
of significant issues that a local community is facing. Browsing one of these topics is like
browsing a subsection of the newspaper (e.g. the sports section, or the stock market sec-
tion). Each section will have different type of content depending on how it is defined and
used by the citizens participating in it. Topics like Upcoming Town Elections, Revisions to
Comprehensive Plan, Downtown Revitalization are examples of possible local issue labels
to organize content. In our research we have found that local civic organizations tend to
focus on a particular small set of issues. We have organized Colloki in an analogous fashion,
so that local citizens can quickly identify the issues at hand. Community leaders have a
significant role in defining initial sections. Citizens can suggest new topics, and if enough
other citizens vote for the creation of that new topic, the new topic is created. Figure 3.1
shows a topic page in a test deployment of Colloki that some members of the Computer
Science department in Virginia Tech have been using for conversations for a few weeks as of
this writing.
Tagging is another means of organizing content on the site that is orthogonal to the topic
approach. A tag is a non-hierarchical term that reflects some aspect of a content item.
Figure 3.1: A sample topic page in Colloki

It is one of the most intriguing features of social software, and we are just beginning to
understand the utility [48] and implications [28] of tagging. Users of Colloki can browse
information within a topic via tags. The tag pages for a topic also provides content items
from other topics that have the same tag assigned to them. Further, users can browse
through the profiles of other users on the system based on the tags that those users have
assigned to their content, hence aiding people in discovering content and people based on
their interests.
Finally, if a user likes a particular content item, she can provide her approval in the system
via a thumbs up. Colloki then uses these ratings to decide how to sort items on the topic
homepage. For example, if an item is very popular, it does not go off the first page of the
topic, even when newer items have come up. We also provide interface controls to filter and
sort the list based on different parameters.
The system does not support negative ratings. However, a user can flag a post if she feels
that the content is inappropriate for the site. Administrators can later un-flag or delete the
post, as appropriate.
To counter the lack of user recommended content and for the purpose of aggregating relevant
content on the site, in the future we plan to build a bot user’ - A computer program that
aggregates several sources of information - including blog search results for topic-specific
terms and aggregated feeds from trusted local news websites and local blogs; filters them
based on the set of keywords that the site administrator provides to the system; and then
posts the most appropriate stories on the site. If the human users on the site vote on the
story, it stays in the system and is cached and later archived. Otherwise, the system flushes
these automatic posts on a regular basis. This helps keep the system updated with the latest
relevant news items from around the web, and provides the users a constant flow of stories.
Colloki provides a flexible browsing mechanism of stories and content so that people can
easily navigate among the information that is being collected in the site.
User Contributed Content: Citizen Opinions
Colloki lets users enter their opinions in rich text in the web based interface. In the future,
it will have multiple ways for users to express their opinions. For example, opinions can be
typed text, video postings, or even audio postings. We will support doing so from mobile
devices, so if citizens want to stand at a particular location in town and file a story with
images from the location, they have the freedom to do so.
Further, we plan to provide means for bringing in opinions from other sources, blogs and
email listservs. We understand that such kinds of automated import features would require
selective import features - so the features would allow authors to bring in blog posts or
listserv emails to Colloki only if the blog post contains certain keywords. Such content can
also be auto-classified via the specification of topic in the body of the blog post or the email.
Opinions are organized in sections of interest called topics’ (Figure 3.1) that are an aggrega-
tion of a particular subset of information. Topics are usually a small number of significant
issues that a local community is facing. Topics like Upcoming Town Elections, Revisions to
Comprehensive Plan, Downtown Revitalization are examples of possible local issue labels to
organize content. Community leaders have a significant role in defining these sections. In
our research we have found that local civic organization tend to focus on a particular small
set of issues. They often have a community leader in charge of the issue. We are organizing
Colloki in an analogous fashion, so that local citizens can quickly identify the issues at hand.
Browsing one of these topics is like browsing a subsection of the newspaper (e.g. the sports
section, or the stock market section). Each section will have different type of content de-
pending on how it is defined and used by the citizens participating in it. Some might have
more links, others might have more opinions, etc.
With collaboration from the local town, town officials could use appropriate tags for com-
munications and Town Council agendas so that information is automatically classified into
the appropriate section of Colloki.
User Recommended Content: Links
Colloki will provide users with easy ways to share content from the web, in the form of
links. Links are a flexible form of user recommended content (Figure 3.1). In spirit, they
are similar to the bookmarks on del.icio.us, the Posted items’ feature of facebook.com and
the news stories posted on digg.com. A link can be a news story, an image, a video, or
any other form of web content that a user recommends to other users of colloki. This can
be done by submitting it on the site; or more conveniently, by using a bookmarklet or a
browser extension. Other users can then vote’ positively or negatively on the items. Hence,
organization is brought to the section using the same sum of the parts effect that systems
like digg.com utilize, but the initial posting of the story is done by community leaders.
User Contributed Content: Local Deliberation
Beyond the top level organization of opinions and stories around ‘Topics’, all online partici-
pants are allowed to comment on each other’s contributions. We will allow any content item
to have a series of comments or posts following it, in a manner similar to blog comments.
This supports discussion and deliberation by citizens as a response to the postings of com-
munity leaders. We will develop an easy way for cross referencing, to support people posting
comments that easily link to other stories/comments within the site.
Promoting real and virtual world conversations: Activity streams
Activity streams (Latest Activity’ in Figure 3.1) are flowing commentaries on users’ actions
on the different sections of the site. They were first made popular amongst social sites by
Facebook in 2007. Activity streams can be customized to give preference to a user’s close
friends’ on the site in the stream that he/she sees. They are an effective way of helping users
discover content that they might be interested in. Colloki provides an activity stream of the
site’s activity at a system level, and topic-level streams on the topic homepages.
Content Syndication: Feeding the Web 2.0
User’s content can be syndicated out to other services (e.g., their photos of Montgomery
County or their comments on revisions to the Town Plan). The goal is to provide as many
ways to get information out to people as possible. We allow information in Colloki to be
reused by other systems, thus giving way to a wider network of information about local
issues.
We provide RSS feeds by sections, recent stories, particular users, tags, and any other or-
ganization scheme in the system. This allows users to be alerted when new information is
available in whichever way they find most appropriate. In the future, we will also provide
options to send updates via emails, SMS, Twitter, and subscription to calendar feeds using
popular applications like Google Calendar, Apple’s iCal and Microsoft Exchange.
3.1.3 Implementation
Colloki is based on the Ruby On Rails framework. Its architecture, hence, is standard Model
View Controller. The base data model is shown in Figure 3.2. The key entities are users
and stories. Stories are individual items of content on the site - a story is either a link or
a post. Other tables support features of the application such as tagging (Taggings, Tags),
comments, activity items, and topics. In the figure, a directed arrow represents a foreign key
relationship.
3.2 CATspace
CATspace is a social web-based repository for Computer Science assignments that allows
instructors to share, discover, and iterate on programming assignments. The objective of
the project is to facilitate the reuse of these assignments and promote their evolution and
iterative enhancement through social networking capabilities. CATspace is built as a Face-
Stories Votes
Taggings ID ID
ID Title vote
tagged_id Description story_id
tagged_type URL user_id
tag_id Kind
Views Count
Popularity
Topic_id
Tags user_id
ID
title
Users Topics
ID ID
login Titile
Comments email Description
ID crypted_password user_id
body salt
user_id session token
story_id session expiration
activation code
activated at
realname
website
bio
location
Activity_Items twitter_id
ID delicious_id
Activity friendfeed_id
story_id linkedin_url
user_id facebook_url
reset_code
Figure 3.2: Colloki Data Model

book application using the Facebook Developer Platform 3 . This allows us to leverage the
social graph of Facebook for the social features of the application. We expect that the ubiq-
uity of Facebook will facilitate the appropriation and acceptance of CATspace in the CS
education community by providing a space where instructors may collaborate, regardless of
their institutional affiliations.
3.2.1 Introduction
CATspace is a web-based repository that allows instructors to share, discover, and improve
programming assignments. The tool supports computer science instructors in developing,
refining, and discovering programming assignments in multiple ways. Instructors can add
assignments to the repository with descriptions and solution code files, provide feedback to
other assignments, contribute to assignments they are co-authors on, and find new assign-
ments through tags, recommendations, and social graphs.
Through the use of this tool instructors should be able to minimize the time investment
in developing new programming assignments and provide long term contributions to the
computer science educators community by adding or providing feedback for assignments.
The process of contribution, reuse, and refinement of programming assignments should result
in the evolution of higher quality content for programming courses and help to mitigate the
duplication of efforts.
3.2.2 System Description
As mentioned in Section 3.2.1 the goals of CATspace are sharing, discovering, and improving
assignments. In this section we describe the decisions and reasoning behind the different
sections of our user interface. An example screenshot of CATspace is available in Figure 3.3
The main driver of our interface design are the two collections of data that CATspace is
3
Facebook Developer Platform, http://developers.facebook.com
Figure 3.3: CATspace Homepage

Search
People Assignments
My CATspace
Home
USERS ASSIGNMENTS
Figure 3.4: Representation of information that the different CATspace views display.
primarily concerned with, assignments and users. Access to this information is provided in
two distinct methods, one, is a depth oriented search and the other, is a breadth oriented
search. By the term depth oriented search we mean that one collection is completely search-
able and viewable without regards to the other collection, this is illustrated in Figure 3.4 by
the People and Assignments views. These views provide the users the ability to search and
order the different assignments and users within the system.
However, where the more interesting aspects of our design reside are in the breadth views,
illustrated in Figure 3.4 by My CATspace and Home. Since our application resides within
Facebook, we have access to a rich social map for each of our users and can inspect the
relationships between them. These relationships give CATspace the ability to make more in-
telligent suggestions to our users, by combining information about the relationships between
users and the assignments that different users have expressed interest in.
Home
The Home view is a representation of all of the pertinent information that CATspace can
determine about the user through leveraging the social graph and expressed interests. There
are sections that show recent activity about tags and people that he/she has expressed
interest in. Also, there is a section that shows CATspace’s recommended assignments based
on interests the user has specified. Home also shows the user other users with similar interests
so that they can grow the social map. Our hope is that this page will encourage our user
base to make discoveries of new colleagues and assignments based on expressed interests and
social data.
Assignment(s)
The Assignments view has multiple purposes, search results, ordered lists, and individual
assignments are all displayed in this view. The search results and ordered list functions of
the view are meant to provide users of the system with a more traditional way to discover
assignments. In this view the user finds assignments they might be interested in without the
help of the social graph.
The single assignment mode of the assignments view is meant to be the hub of all activity
for an assignment. This is where the editing of the assignment’s metadata, source files, and
various other fields takes place. This editing is provided with as little navigation as possible
and fields are editable directly on the page, by using ajax technologies. This is also where a
user that is interested in the assignment ultimately navigates to, as such downloading and
providing feedback for the assignment are both done here.
People
The People view, similar to the assignment view, servers dual purposes as well. In this view,
there are lists of all users that are on CATspace and lists of your Facebook friends that are
using CATspace, this enables users to leverage existing connections and possibly find more.
Also, the view of an individual user’s profile is displayed in this view, outlining all of their
latest activity on CATspace and displays some of the users in their social map.
My CATspace
The My CATspace view is a window of all of the activity associated with a user’s assignments,
any assignments that this user is an author for and any feedback associated with these
assignments are displayed here. This view is meant to house any of the activities of the user
on CATspace, for instance, this is the jumping off point of creating assignments as well as
displaying any information about already created assignments and their associated activity.
3.2.3 Implementation
The architecture of CATspace strictly follows the MVC design pattern, as seen in Figure 3.5,
and is implemented using Ruby on Rails. In CATspace views, each page of the application
has its own class. Since some of the views of CATspace are more interactive than others,
some of the views have a further breakdown of classes into sections. In the controllers there
is a one-to-one mapping of classes, except for the application controller which is the root
controller class that other controllers inherit from. Each action in the controller maps to
a view, and generally utilizes more than one model. For example, the “show assignment”
action would utilize the assignment model to display the assignment, and the user model
to authenticate the user. The models correspond to data sources. The data sources can be
database tables, web services, files on the server, or a mixture of these. For example, the
user model comprises of a table in the database, and also Facebook’s web services. The
assignment model abstracts a database table and a collection of files in the web server’s
filesystem.
We utilize some other Ruby on Rails mechanisms such as partials in the view layer that
CATspace
Layouts
Views AJAX Response
Home Users Assignments Activity
Application
Items
profile / my catspace index show/edit new IFrame
index
preferences AJAX Views
search tag Partials
Assignment List
People List
Activity Item
Controllers
Activity
Home Users Assignments
Items
Application
Models
Facebook
Assignment Comment Activity Item
Users
Facebook
Authorship
Publisher
Libraries & Helpers
Application Helper
JavaProps Read/Write
Attachment Handling (Plugin)
Spawn Processes (Plugin)
Externals Facebook Web Catspace Database

ZIP (RubyZip Gem) Services API
Figure 3.5: Object model and basic design of CATspace

provide common modular functionality available to all the views in CATspace.
Data Model
The data model for CATspace is shown in Figure 3.6. The key entities are users and assign-
ments. Other tables support features of the application such as tagging (Taggings, Tags),
comments, activity items, and authorship roles. In the figure, a directed arrow represents a
foreign key relationship. As one might notice, the user model is devoid of any personal infor-
mation. This is because the personal information is maintained by Facebook and retrieved
within our application via Facebook’s developer APIs.
Facebook Integration
From a user perspective, facebook forms a proxy between him/her and the application. From
the application’s perspective, Facebook App Platform environment proxies user requests.
There are two key ways for the application to interact with Facebook in this model (Figure
3.7):
1. Use FBML encoding to respond to user queries. Facebook interprets the FBML re-
sponse and replaces it with the appropriate information before responding to the user.
2. Use Facebook APIs independent of FBML to query for user data.
Developing the application within Facebook’s framework provided both benefits and unique
challenges. The primary benefit, as mentioned previously, is that the application is able to
leverage the rich social graph of Facebook. However, developing for Facebook is not without
difficulties. Some of the difficulties our project ran into include having to implement an ajax
and animation javascript library as we could not use existing open source libraries such as
prototype.js; and having to add separate session authentication for file handling, because
ID
Taggings
tagged_id
tagged_type
{ ID
Title
Synopsis
Rating
Assignments
tag_id Attachment_Name
License
Language
Topics
Tags Goals
ID Potential
title Info_URL
Lang_version
course Authorship
est_experience ID
est_time Role
est_size Order
... assignment_id
Comments user_id user_id
ID
body
user_id
assignment_id
Users
ID
FB_ID
session_key
Activity_Items
session_expires
ID
ROLE
ROLE
Activity
pref_comment_notify
assignment_id
pref_tags_follow
user_id
pref_email
pref_download_notify
pref_rate_notify
pref_tags_notify (...)
Figure 3.6: CATspace data model

Facebook App Platform
User HTTP CATspace APIs

HTTP, FBML
User
Figure 3.7: Interaction between CATspace and Facebook
Facebook, at the time of writing the application, did not support returning zip mime type
while responding via its proxy.
Edit-in-place for Assignments
One of the most important goals for CATspace is to make creation and editing of assignments
fairly simple and fast. Long and tedious editing interfaces discourage users from actively
contributing to the system. CATspace provides assignment authors a dynamic assignment
editing interface that leverages AJAX for providing a WYSIWYG experience. For the fields
of assignment title, description and the several metadata fields (e.g., license, license URL,
language etc.), the fields are by default represented as standard text on the page. However,
when the user hovers the mouse over these fields, the visual representation of the field is
modified to represent highlighting by adding a background color and a border. If the user
then clicks on the field, then the field converts into an actual HTML text field filled with
the value that was being represented in text before. This text field is followed with a couple
of buttons to save the new value of the field or to cancel the edit action. The user can then
modify content and click save. This sends an AJAX request to the back-end and if a success
response is received, the HTML text field and the buttons vanish, and are replaced by the
original text representation of the field, except that the text content is changed to the new
value.
The tags list is represented as links in its standard form on the page. Hence, in this case,
instead of overloading the hover action, a small edit tags link is provided next to the tag list.
The user can click it to convert the the list into an editable comma separated list inside a
text field.
This form of editing provides a richer, faster user experience than a standard edit page for
the assignment:
1. Faster: When an author is gradually building up his or her assignment and filling
it in with information, then in most cases he or she is just looking to edit one field
at a time - for example, a metadata item. Editing via an edit-in-place field is faster,
because lesser data is sent and received from the server for an AJAX request to update
a field than in the case where the user navigates to an edit page and submits a form
wherein all the fields are sent again to the back-end regardless of whether they have
been updated or not.
2. Asynchronous: Since AJAX calls are asynchronous in nature, they do not affect the
rest of the page elements during their processing. Hence, the user can actually go and
start editing another field while the save request of the field he/she just edited is still
being processed. Contrast this to an edit form for the assignment page - in which case
the user would have to wait both for the edit page to initially loaded and then wait
for the data to be stored to the database before he/she can return to the assignment
page.
3. Reduces the chance of data loss: A related advantage is that by the use of edit-in-
place fields, the assignment’s state is stored at an atomic level - whenever the author
edits a field, that property of the assignment is updated in the database. Contrast this
to server-side generation of pages, in which case the user would go to a separate page
to edit all the fields. If the user is editing multiple fields on the form and for some
reason the browser session is lost before (s)he can save the form, then all the updated
information would be lost.
3.3 Social Features
Both of the above applications utilize social features. The Related Work chapter discusses
past work on these social features. This section characterizes and compares the specific
implementations of these social features across these two applications.
3.3.1 Tagging
Both Colloki and CATspace use tags in similar ways. However, there are subtle differences.
To characterize the tagging aspects of these two systems, we use the framework defined in
[48]. As detailed in Section 2.1.1, the user incentive characterization can only be determined
for a functional system that is being used by the community. Hence, we characterize the two
systems only in terms of system attributes, as these tools do not have an active user base in
their respective communities yet.
1. Tagging Rights: Both Colloki and CATspace are restricted to self-tagging, that is, a
user can only tag resources that she creates.
2. Tagging Support: Both Colloki and CATspace have blind tagging support. They do
not (yet) provide features to suggest tags to users while tagging resources. Suggestive
tagging can benefit a tagging system by helping it converge to a folksonomy [31].
3. Aggregation: Being self-tagging systems, both Colloki and CATspace follow the
bag-model for tag entry. In other words, a tag is a property of a resource and a user.
However, there is a difference in the way Colloki and CATspace implement aggregation.
While in Colloki, two users can have duplicate tags for the same resource for some types
of resources (namely, links); in CATspace, each resource (assignment) is unique and
only one user can tag it.
4. Type of Object: In Colloki the objects that users can tag are posts and links, as
defined in Section 3.2.2. In CATspace, the objects are CS assignments that instructors
upload.
5. Source of Material: CATspace allows only resources that are supplied by the par-
ticipants. Colloki’s resources can be from the participants (posts, links to participants’
content), or could be any web resource (links).
6. Resource Connectivity: CATspace provided linked connectivity between resource

via tags; while Colloki provides both linked and grouped connectivity between re-
sources. Colloki provides grouped connectivity via the concept of ‘Topics’ which group
related posts and links together (See Section 3.1.2).
7. Social Connectivity: Both CATspace and Colloki provide linked social connectivity
independently and with respect to tags. Users are connected when they use the same
tag to tag their resources (and by other means, which do not involve tags). Both the
systems also provide grouped user connectivity. Both the systems use means to connect
users by forming grouping, like top contributors in Colloki, and recommended users in
CATspace.
Given the above characteristics, we expect tagging on these two systems to provide the
navigational constructs that provide affordance for the social actions of social browsing,
social search and sharing content. As the system’s usage in the community increases, we
also expect users to find affordance for collaboration.
Both Colloki and CATspace implement activity streams in a fairly similar manner. On
CATspace, However, there are some differences. In Colloki, activity streams are shown on
the homepage and the topic pages. In CATspace, these are more widely available - The
homepage, the user profile, and assignment page; all show activity streams.
CATspace also provides the ability to delete items from a user’s activity stream. In Colloki
the only way to remove an activity stream is to undo the activity (Deleting a comment or
un-voting an item). These are conscious design decisions taken based on the context of the
application. CATspace exposes more information as part of the activity stream - the actions
shown on the stream are publishing assignments, commenting, rating, tagging and editing
of published assignments. Instructors might sometimes want to remove items that they feel
might mislead viewers, or are just not important enough to be published to the stream. In
Colloki, all actions that are published on the stream (Creation of posts/links, commenting,
voting) are public in nature, and removing items from the stream goes against the very
public and open nature of the system. There are no filtering/grouping features in either
application yet.
Based on our design choices, we expect activity streams to be an engaging means to navigate
across the content on the system. Hence the social actions that we expect the feature (as
implemented in the two systems) to provide strong affordance for social search and social
browsing.
3.3.3 User Profiles
User profiles, as mentioned in Section 6.1.2, are central to social network sites. These two
applications are no different. Colloki’s profiles are shown in Figure 3.8, and CATspace’s
user profiles are shown in Figure 3.9. Both applications use the profiles to showcase the
content that the user has contributed to the system, and provide the users ability to provide
Figure 3.8: A user profile on Colloki

additional information about themselves.
Colloki has several fields in the profile that let the user provide name, a short bio, an
avatar image, location, and link to website. In addition, users can provide their usernames
on other social systems (LinkedIn.com, Twitter.com, Facebook.com, Friendfeed.com, and
delicious.com) that are converted to icons on the user profile that link to the user’s profile(s)
on those systems. In addition, the profile provides a showcase of the latest content and
comments by the user, and a tag cloud of the user’s tags.
CATspace is different because it builds over facebook’s social graph. It pulls the user’s profile
photo from Facebook and shows it in the profile along with the user’s name. The user can
supply their affiliation separately in the CATspace application. There are no other custom
fields, the rest of the profile is a showcase of the user’s assignments on CATspace (Figure 3.9)
User profiles are central information sources for people, so by definition we expect them to
strongly support the actions of social browsing and social search; and provide good affordance
for sharing.
3.3.4 Comments
Both Colloki and CATspace have fairly straightforward implementations of commenting.

The feature is designed to essentially facilitate conversations, hence we expect it to strongly
support interaction. Comments also help in discovery of people’s thoughts and opinions,
hence we foresee benefits in the social actions of social search and social browsing. By
providing a means to conversation, we also expect users to find comments to be beneficial
for collaboration.
Figure 3.9: A user profile on CATspace (Picture used with the permission of Dr. Stephen
Edwards, 2009)
Colloki and CATspace implement user ratings/votes in very different ways. In Colloki, users
can ‘vote’ for an item that they have not created. Voting is an atomic activity and has a
binary score - you either vote or you do not. It also provides users the ability to un-vote a
piece of content after they have voted for it. An activity stream item is generated when users
vote for an item, and the only way to remove that item is to un-vote the content. Hence,
there is no way to secretly vote for a piece of content. votes are used to derive the sort order
on the default view of topic pages, hence they are central to the organizational structure of
Colloki.
CATspace provides users a 5-star rating system. Users can change their rating of an as-
signment, but once having rated an assignment, cannot un-rate it. Rating activity does not
show up on the activity stream of the system. These ratings are then used to provide one of
the sorting orders in the assignment views. In the future, we will also use ratings to provide
leaderboards - listings of the top rated assignments and the top rated users (via association)
in the system.
Ratings and votes provide building of the social capital via community-powered organization
of content and via reputation scores that lead to user interface artifacts such as leaderboards.
These kind of social mechanisms are out of the scope of our current study, however, in the
actions that we are measuring in our study, we expect users to find ratings and votes helpful
in finding relevant content, an activity could then lead to benefits for social search and social
browsing.
Chapter 4
Methodology
This chapter presents the methodology used for performing the evaluation study for the two
tools. This study was conducted as a usability evaluation along-with pre and post evaluation
surveys. We had ten participants in total, five in each of the individual studies. All par-
ticipants were graduate students at Virginia Tech. All five participants of CATspace study
worked in the Computer Science Department, while the participants of Colloki study were
from other departments across campus. Each participant was given a short Pre-Evaluation
survey that comprised of questions on internet usage and academic background.The partic-
ipant were then provided a short video demonstration of the site they were evaluating, and
then asked to perform a set of tasks on that site.This was followed by a comprehensive post-
evaluation questionnaire on the site, that had questions on usability and perceived social
affordance.
4.1 Goals
The main goals of this study is to perform a formative evaluation of the two sites. This
evaluation, in addition to evaluating the usability of the two sites, should explore how the
49
Sameer Ahuja Methodology 50
the perceived social affordance of the features for the social actions of social browsing, social
search, interaction, sharing and collaboration matched the design expectations of the features
of CATspace and Colloki, and the outcomes thereof.
4.2 Experimental Setup
4.2.1 Selection of Participants
We recruited 10 graduate students from the town of Blacksburg as participants for our
evaluation. The evaluation used two different groups of 5 users each for the two applications.
For Colloki, we recruited graduate students studying at Virginia Tech, with the additional
condition that the participants need to be studying in a department other than Computer
Science. This is because the benchmark tasks of Colloki can be simpler for people who have
contextual information about the Department of computer Science at Virginia tech, and its
members. To avoid any such circumstances, we made sure that none of the participants had
contextual information about the Department that would help them in the tasks.
For CATspace, we recruited graduate students studying at Virginia Tech, with the additional
condition that the participants need to studying in the Department of Computer Science.
This is because we felt that the graduate student community in the Computer Science
department provides a good approximation of the kind of community that we expect to be
using CATspace when it is deployed in the wild.
Both the user groups were invited via individual emails to participate in the study. The
participants received a gift coupon worth $5.50 each in return of their participation.
4.2.2 Usability Lab Setup
The usability evaluation was setup in a closed lab in the Knowledgeworks II building of the
Corporate Research Center at Virginia Tech. This room was not accessible to the general
public when the study was in session. The evaluator and the participant were both present in
the room at the same time. Once the tutorial video was completed, communication between
the evaluator and the participant occurred only when the participant had questions or com-
ments. In general, there was no communication between the evaluator and the participant,
during the time of the actual tasks.
The setup included the following:
1. Computer with Internet access: A computer with Internet connection was used
for the evaluation. Shortcuts to the prototype and the demo video were provided on
the desktop of the computer system.
2. Screen capture software: A screen capture software was used to record the users’
on-screen activity. These recordings helped us find critical incidents in the evaluations.
4.2.3 Evaluation Protocol
The study detailed here is formative in nature. The purpose of formative evaluation is to
validate or ensure that the goals of the instruction are being achieved and to improve the
instruction, if necessary, by means of identification and subsequent remediation of problem-
atic aspects [74]. According to [63], “When the cook tastes the soup, that’s formative; when
the guests taste the soup, that’s summative.” Formative evaluation is intended as a basis
for improvement.
Both the studies (CATspace and Colloki) were approved by IRB at Virginia Tech, please
see Appendix A for the letters of approval. When the participants arrived in Room 1132
at Department of Computer Science Building, Knowledge Works II they were greeted by
the evaluator, and the purpose of the evaluation was explained to them, along with the
evaluation setup, the procedure, confidentiality of data and the approximate duration of the
evaluation. They were then handed the Informed Consent Forms to read and sign if they
were willing to participate in the evaluation.
Once the participants signed the Consent forms they were instructed to keep one copy for
their records. Then, we asked them to fill out an online pre-evaluation survey. This survey
included a short demographic survey and a survey that asked the participants questions
about the depth and the breadth of their usage of social network sites. After completing
the survey, each participant was asked to view a video demonstration of the software. The
video goes through the main features of the software that the users would be using in their
evaluations.
Following the video demo we handed the participants a sheet that contained a set of pre-
designed benchmark tasks. These were a set of tasks derived from the scenarios, and are
described in Section 4.3. We then reminded the users that they are not going to be timed
on any of the tasks.
Following these tasks, we asked the participants to fill out a post-evaluation questionnaire.
The post-evaluation questionnaire is designed to provide feedback about the tool and the
evaluation procedures. This questionnaire form helped us get information about the partici-
pants experience using the tool with respect to usability, and the perceived social affordance
of the tool’s features. It also provided space for free-form feedback by the participants.
4.3 Benchmark Tasks
4.3.1 Colloki
For the tasks, the users were asked to assume that they had just joined the Computer
Science department at Virginia Tech as a graduate student and were interested in learning
more about the department’s student community.
1. Use the following credentials to log on to the site -

Username: participant
Password: participant
2. Browse through the list of links and posts in the topic ‘Local Restaurants’. Is there
a restaurant you know in the area that is missing? Add a link to the website of that
restaurant to the topic along with some description and tags. (You can use web search
or other tools to lookup local restaurants and/or their websites.)
3. Go to the ‘Graduate Life’ topic, and view the most recent links in that topic. Vote for
the link that you like the most. (You can click through on the links and visit the sites
to decide your favorite link). Now, leave a comment on the link mentioning why you
like it.
4. Which is the oldest post in the topic ‘Graduate Life’ ?
5. Which one of the five topics listed on the homepage do you think is the most popular
amongst the slurp community?
6. Which one of the five topics listed on the homepage do you think has been the most
active recently amongst the slurp community?
7. From the department website, you know that Sameer Ahuja works in the same research
group that you are interested in. Find him in Slurp. Once you’ve found him, find out
two people that he has had the most interactions with (via comments) on slurp.
8. Manuel A. Pérez-Quiñones is the professor who heads the research group that you are
interested in. His user id on Slurp is ‘mapq’. His top tag on Slurp represents his
primary research area: ‘PIM’. Name one more user in the system whose top tag is
‘PIM’.
4.3.2 CATspace
For the tasks, the users were asked to assume the role of a fictional professor in Computer
Science, who teaches the introductory CS class and is interested in the topics of algorithms,
arrays and linked lists. They were logged into the application as that user.
1. Go to your CATspace preferences page and add the tags ‘CS1’ and ‘arrays’ to your list
of tag subscriptions.
2. You are looking for an assignment for your upcoming class on two dimensional arrays.
Find an assignment that’s tagged both ‘CS1’ and ‘2d array’.
3. Find which of your Facebook friends are already using CATspace. Name one of these
friends.
4. Name two other people that the person you named in the last question has as co-authors
on his/her assignments.
5. ‘Stephen Edwards’ is a very well known CS instructor working in your area of teaching
who had earlier mentioned to you that he is already on CATspace. Find his profile on
CATspace. How many assignments has Stephen published on CATspace? Which is
the latest assignment that he’s published? What’s his top rated assignment?
6. Is ‘CS1’ amongst the top tags for Stephen? Find another who has ‘CS1’ as one of
his/her top tags.
7. You decide to contribute to the CATspace community by adding one of your own
assignments. Create a new assignment with the title ‘Single dimensional arrays’ and
tag it with tags ‘CS1’ and ‘1d arrays’. Upload the assignment.zip file available on the
desktop to this assignment’s file.
8. After making these changes, you realize that the assignment also has a question on
linked lists. Edit the assignment’s title to ‘Single dimensional arrays and linked lists’,
Table 4.1: Expected mapping of Social Features to Social Actions
Social Social Interact- Sharing Collabor-

Browsing Search ion Content ation
Tagging High High High Good
Activity Streams High High
User Profiles High High
Comments Good Good High Good
Ratings and Votes Good
and add ‘linked lists’ to the list of tags.
9. You’re now finishing up the assignment. Let’s make a final set of changes. Set the
metadata property called ‘Course’ for this assignment to ‘CS1’. Also, update the
readme.txt file in the assingment’s files - change the first line of the file from ‘Arrays’
to ‘Arrays and Linked Lists’.
10. Publish the assignment.
4.4 Perceived Social Affordance
The goal behind measuring perceived social affordance is to get a sense of whether users
see the social utility of the software. This utility that users see comprises of software and
social effects. In the evaluation, we control for the social effects by providing each user with
exactly the same state of the system when they evaluate it. This state was represented by
a simulated set of users, content and interaction. Below, we describe the guidelines we used
for simulating the community around the software.
4.4.1 Simulated Community
At this stage, the software does not have a community of people using the content. However,
for a meaningful evaluation, the users need to be introduced to a state of the system with
some content already present. For this purpose, we simulated a community of users and
their content. The kind of community and content to simulate is contextual to the kind of
application being built, but the following general guidelines were followed while building the
communities:
1. The scale of the content on the site should be just enough to provide the users an
indication of how features work. For example, in a listing of content, any content that
falls onto the second page is extra. One should be careful to not place too much content
on the site, as this might start affecting the user’s perception of the social utility to
greater extents. As the general guidance, for all the central views in the prototypes,
we filled the views with content enough to fill up the first fold/page.
2. The content should model the kind of content that members of the community are
expected to post.
4.4.2 Mapping social features to actions
Table 4.1 shows the expected mapping of features to social actions. Each element in this
grid is a mapping between a single social feature and a single action; and it represents how
well the social feature is perceived to be for that particular social action. The social actions
listed here are sufficiently abstract to be representative for a set of characteristic actions.
Yet the actions listed are easily differentiated from each other. The values in this table have
been filled based on our design expectations from the features, and the user responses were
evaluated based on these expectations.
To measure the individual values, we provide users statements for each grid element, and ask
them to evaluate them on the Likert scale. For example, to evaluate the perceived affordance
of tags for interaction in CATspace, we provide the following statement to the users:
Tags will be helpful for me to interact with other people on CATspace.
These questions are asked in the post evaluation questionnaire. The complete questionnaires
are provided in Appendix F and Appendix G.
Chapter 5
Results
In this section, we discuss the results from the evaluation study. We break down the results
based on features, and provide results for both sites together to facilitate comparison. Given
the low scale of the study (5 participants for each site), we’ve concentrated on results that
show a clear opinion across participants, in terms of a unanimous agree or disagree on the
likert scale questions. To provide a visual aid for understanding the results, we provide figures
in each section that show the user responses to the likert-scale questions in a color-coded
grid of responses, along with the averages. The questions that we asked the participants are
labelled as A1 to A43 for the Colloki study, and B1 to B55 for CATspace participants.
5.1 Overall Usability
Overall, participants found the two sites easy to use (Questions A1, B1 in Figure 5.1).
The participants also unanimously found the basic tasks of creation and editing of content
(Questions A2, B2, B3, B4 in Figure 5.1) to be easy to use.
However, for the two questions that we asked related to articulating relationships via tags
(Questions A5, B7) or comments (Question A6); we found that the users clearly did not
58
Sameer Ahuja Results 59
Figure 5.1: Participant responses for questions on overall usability

find it easy to use the features in their current state to perform the task of articulating
relationships. We consider that an important task because its a central task with respect
to the definition of SNS that we are using, and also because it supports the social action of
social browsing.
Some of the free-form feedback users provided gives us a deeper understanding of why par-
ticipants did not find these tasks easy to use. One of the participants of Colloki’s evaluation
mentioned in the feedback:
The Tagging thing was odd. I had to click on a tag which took me to a page that
had all posts with the tag. Then from there I had to click on one of the posts,
then go to its author and see if he or she had that tag in the tag cloud. That’s
cumbersome.
Another participant, this time from the CATspace evaluation, mentioned:
I think this profile looks a little empty. You can have a list of users with similar
stuff here, kindof like facebook friend list. Actually, you could show the person’s
facebook friends (who are on CATspace) right here.
The results and these qualitative feedbacks clearly indicate that the system needs to aid
the users in articulation of relationships amongst users of the system, by providing explicit
interface elements that showed the relationships we deem important, ideally in user profiles.
An example of such an interface element is providing a list of users who have a similar set
of top tags, on the user’s profile in the system.
Figure 5.2: Participant responses for questions on tagging

5.2 Features
5.2.1 Tagging
At a basic level, participants on both evaluations found tags easy to understand conceptually
(Figure 5.2), and found it easy to create tags and navigate the systems based on tags. This
assures us that there aren’t any conceptual issues with the implementation. However, the
implementation of tagging fell one step short of providing users affordance to perform slightly
more complex tasks, like finding people using similar tags. Participants on the Colloki study
did not agree that finding similar people was easy (Question A5 in Figure 5.1). The CATspace
users gave varied responses to the same question, with two strongly agreeing, two disagreeing,
and one user was neutral (Question B7 in Figure 5.1). As discussed above, this task related
to articulation of relationships on the network, and has direct impact on the social actions of
social search and social browsing. User responses indicate that the system needs to explicitly
articulate relationships based on tags in user profiles in CATspace. In Colloki, while tags
are used for organization orthogonally to topics, we are not as concerned about relationships
based on tags, as we are on relationships based on interaction.
With respect to social actions, the perceived utility of tagging in Colloki for different social
actions generally matched the design expectations. In the case of CATspace, however, a
couple of social actions saw a wide variety in users’ opinions of them.
Colloki participants generally agreed with the utility of tagging for all five social actions
evaluated (Figure 5.2). The participants felt that tagging would be most helpful for effec-
tively sharing content with other users on the site. Interaction got the least score as the
action that tagging facilitates. These results are in accordance with the design goals for the
tagging feature, as discussed in Section 3.3.1.
In CATspace, however, the participant reactions were more varied. The three actions that
we deemed most important for users to perceive in our design were social browsing, social
search and sharing content. Participants generally strongly agreed in the utility of tagging
for social browsing - the average likert scale score for the related question (B13) was 4.4, with
a standard deviation of 0.89. However, the scores for sharing assignments were spread out
across the participant - while one of the participants strongly disagreed with the assertion,
another was neutral, and the rest strongly agreed with the statement that tagging would
be helpful for sharing content. For the other two social actions that we had deemed that
tagging would be useful for, participants were generally neutral to the utility of tagging for
social search, but were more positive towards its utility for collaboration (Average score of
4.2, with standard deviation of 1.1)
There were significant differences in the way that participants of the Colloki and CATspace
studies evaluated activity streams. As visible from the responses to A16 in Figure 5.3,
there were varied opinions on how intuitive and easy to understand activity streams were.
These findings are in contrast to those of CATspace, where all of the participants responded
positively (B18 in Figure 5.3). We discuss possible explanations to the same later in this
section.
In our design of activity streams, we had expected them to be useful for social discovery
and social our expectations for activity streams as vehicles of social browsing and social
discovery. Clearly, the participants did not see that utility (Responses to A18, A19, B20
and B21 in Figure 5.3). This cannot be attributed to the lack of conceptual understanding
of activity streams - most of the participants were regular users of Facebook, a site where
‘News Feeds’ about friends is a central concept.
The above seems to point that there might be issues in our design or implementation of
activity streams. Surveying participant videos provides some hints - none of the participants
noticed used the activity stream for navigation. Also, both in Colloki and CATspace, when
users were asked to find information that was available in a straightforward manner within
Figure 5.3: Participant responses for questions on activity streams

activity streams (Task 6 in Colloki, Task 4 in CATspace), they instead chose other means to
reach the information. This might indicate that activity streams were not visually prominent
enough in the interfaces to be noticeable. In Colloki, another issue related to activity streams
was that they were not ubiquitous - the users’ profiles did not have the stream of activities
that those users had performed, and similarly the pages for individual posts and links did
not show the activity performed on those items.
5.2.3 User Profiles
User profiles, as discussed before, are a primary component of social networking sites, and
hence a critical feature for the system to get right. Participants found user profiles easy
to understand, and they consistently saw its utility in both the systems. In terms of social
actions, we expected user profiles to be useful primarily for social browsing and social search.
The results were in accordance with our expectations - participants fairly unanimously con-
sidered user profiles to be helpful for social browsing and social search.
Users of both Colloki and CATspace also found user profiles to be unanimously useful for
another form of social action - interaction (Questions A41 and B41 in Figure 5.4). This
is interesting, because user profiles on both the system did not provide any direct means
for interacting with the person. They did, however, provide links to other places on the
system, or outside of the system, where interaction with the user was possible. One possible
explanation for the users to vote unanimously in this regard is that they brought their
expertise of existing social network sites with them. When they thought ‘user profiles’, they
thought a place to find about and interact with someone. Evidence for this conceptual
thinking is provided by one participant’s feedback to CATspace’s user profiles -
[...] but where can I leave him a message? Do I have to go to the Facebook
profile for that? How do I go to the Facebook profile from here?
Participants thought of user interaction as one of the obvious things to do on a user’s profile.
Figure 5.4: Participant responses for questions on user profiles

This feedback has a consequence to our designs for the next iteration of the product: Knowing
that people expect to have the social affordance of interaction on a user profile, we can build
features that are directed towards that goal.
5.2.4 Comments
Both the applications had fairly straightforward implementations of commenting. We ex-

pected users to find commenting helpful primarily for interaction and collaboration. Partici-
pants on both studies agreed with us for interaction (Questions A27 and B28 in Figure 5.5).
For collaboration, however, it was a different story - In both the studies, a couple of users
were neutral about the utility of collaboration. We attribute this partly due to a fault in
our study, that we discuss in Section 6.1.3.
As detailed in Section 3.3.4, we rated social browsing and social search as secondary ben-
efits of commenting - a person’s comment tells you something about the person, and in
these two systems it is fairly simple to navigate to people’s comments. In the results, we
found unanimous agreement on the affordance of commenting for social browsing in both
the applications, but not social search.
Ratings and votes bring community controlled organization to social network sites. The
users on Colloki found the voting feature to be easy to use almost unanimously - one user
was neutral. On CATspace, the feature was not completely developed - the user interface
for making the ratings had a couple of bugs. Hence we did not have a task for rating an
assignment, only for organizing the view based on already-existing ratings.
Figure 5.5: Participant responses for questions on comments

Figure 5.6: Participant responses for questions on votes/ratings

Chapter 6
Discussion and Conclusions
This chapter is divided into three sections - first, we discuss the outcomes of the study for
the two social network sites we evaluated. Second, we summarize and discuss the evaluation
procedure. Finally, we discuss future work towards evaluation of social network sites.
6.1 Outcomes
Based on the results, we obtained the following main outcomes for the two social networking
sites:
6.1.1 Articulation of connections
From the results, a fairly clear design flaw with both the applications was that they did not
provide a direct means to articulate and navigate social relationships. The recommendation
for Colloki is to either clearly articulate social relationships (formed via using the same tags
or commenting on each others’ posts) on user profiles, and let users navigate those; or to
provide a different, more straightforward concept of social connections - such as mutual
relationships of facebook, or the unidirectional ‘follower’ relationships of twitter.
70
Sameer Ahuja Discussion and Conclusions 71
For CATspace, the problem is more complex - The underlying platform of Facebook already
provides explicit social connections, and there are implicit social connections formed by
people using the same tags. Neither is clearly articulated on user profiles. We see utility
in exposing both of the relationships as they facilitate different social actions - articulating
facebook connections may help people find more about people they already have as social
connections, hence facilitating social search; while articulating content-based relationships
facilitates social browsing by listing users who have posted or are subscribed to similar tags.
Hence, the first step will be articulating both on the CATspace user profiles. In the next
evaluation, it might make sense to ask users to evaluate the two relationships for social
affordances and to evaluate whether articulation of two kinds of relationships confuses the
users in any manner.
Users on both sites rated activity streams lower than we expected. They consistently used
other means of navigation for finding relevant pieces of information. The key takeaway for us
was to reset our expectations. Both Colloki and CATspace are content centric, and provided
clear affordances for navigation. Activity streams are great for serendipitous discovery,
however; when asked to perform tasks, users responded by utilizing the feature that most
clearly provided the affordance for the task required - which in both cases was not the activity
stream. This also raises an interesting question - are social affordances of features in a site
a zero-sum game (that is, if add a feature that provides a certain social affordance for some
action, does the affordance for that action decrease for the other features)? It may very well
be true to a certain extent, and there is a clear analogue in conventional usability engineering
to this situation - If you let the user do the same thing in several ways, you might end up
confusing the user, causing overall perception of usability to drop. So its important to think
of what the key features for social actions are, and hence in our expectations account for
the reduction in perceived affordance, caused by having a feature facilitating a social action
which is also facilitated by a more central feature to the application.
6.1.3 Task Sets
In our analysis of the results, we realized that the social action of collaboration received low
scores on all social features. On further examination, we found that collaboration was the
only social action that the users did not perform as part of their task scenarios. We think
that this causes the ratings to distort, especially in this controlled environment. Hence, we
decided to ignore all results for collaboration for any of the outcomes, and more importantly:
we tuned the study for future instances to provide task scenarios that reflected on all the
social actions that we are measuring.
6.2 Evaluation Procedure
Going back to the overarching problem that we established in Chapter 1:
How to perform formative evaluations of a social network site, at a stage where

it has not been released and the user community has not started participating,
and get a sense of how successful it would be when released for use by that
community?
So how does the evaluation we performed in this thesis help in determining how successful
the site would be in the community? In the context of social actions based evaluation,
the question of ‘how successful is the site is going to be’ can be broken down into three
constituent questions:
1. Do the users of the site perceive the features of the site to have social affordances for
the actions that we expect the features to have social affordances for?
2. Will the users, even when they do perceive the social affordances, actually perform the
social actions?
3. Finally, if the users do perform these social actions, how would that relate to the central
goals of the application?
Formative evaluation gives us a good sense of the answer to the first question. The second
question deals with user incentives and motivations, and the question is out of scope for
formative evaluation, and the results obtained cannot predict usage. Finally, what does it
mean for the application’s goals if the users perform the expected actions? The actions have
been framed as positive in our work - that’s derived from past literature that has related
these actions to the rise of social capital for the community. The rise in social capital is what
connects the site’s goals to the social actions, and it brings along the social utility that social
network sites hope to achieve. However, one cannot overrule the possibility of negative effects
of certain social affordances of features, or certain social actions. Feedback from participants
during formative evaluation can help catch some of these negative consequences at an early
stage, but in essence this is something that relates to the second question above - it is very
contextual to the community that ends up using the software.
Following is a summary of the evaluation procedure.
1. Design features for the site while considering which social actions those features are
expected to facilitate. With this information, construct the expectations matrix. The
social actions listed in this thesis are not an exhaustive list, but they provide a good
foundation of social actions to consider for most common contexts.
2. Perform an evaluation with a set of participants that are ideally a representative set
from the community the site is intended for. This evaluation is designed to perform a
formative usability and social affordance evaluation. As part of this evaluation,
(a) Perform a pre-evaluation on the participants on their expertise with social network
sites, and particularly, the social features that they are going to be evaluated on.
(b) Provide a demonstration or a guided tour of the features to the participants.
(c) Provide the participants with the prototype site. Simulate a community and
content on the site, based on the guidelines provided in Section 4.4.1. Provide
the users with a set of tasks to perform on the site. These tasks are to be derived
as per standard usability engineering guidelines - from scenarios of usage. These
scenarios should cover the primary task(s) of the application (Such as uploading
and editing assignments in CATspace, adding links in Colloki); the primary social
tasks of articulating social connections and navigating those connections to reach
user profiles; and finally, tasks based on the expected social actions. The tasks
are a means for not just measuring the task completion, but also to provide the
users a sense of the social utility of the software.
(d) Provide the participants with a post-evaluation questionnaire, that asks them
three sets of questions - Likert scale questions on the usability of the features of
the site, Likert scale questions on the perceived social affordance of the site, and
finally ask them for free-form feedback. This last set can be substituted for by a
short interview.
3. With these results, find strong mismatches in expectations and results of perceived
social affordance. Such mismatches might mean one or more of three things - there
might be a design issue with the software, there might be an issue with the study (For
example, the tasks might be covering some social actions and not others), or it might
mean that the expectation was wrong - that the users either would not perceive the
utility with the software until the sociological effects start playing their role or that
the feature was not as important for the particular social action. Any number of these
three aspects might require tuning. Participants’ past experience with SNS, the critical
incidents from the task performance, and the feedback from the participants can help
provide more hints to root causes, and lead to recommendations.
6.3 Future Work
This thesis is a first step towards studying the design and evaluation of social network sites
from a practitioner’s perspective. There are some key areas where we see future work in the
area. One, it would be interesting to extend the social actions based framework to measuring
user behavior on these sites when they are released to the community, and see if there are
any strong generalizable relationships between perceived affordances and actual use.
Secondly, an interesting trend had started to emerge in this study, a trend that was out of
scope for this study to explore, but merits further study. This trend was that the features
appeared to have similar affordances across the sites - that is, the set of actions users users
seemed to think a particular feature facilitated seemed to be similar across sites. This was
largely due to the fact that since these two applications had the same designers, they had very
similar implementations of features. However, it would be interesting to study affordances
for social actions for particular features - If there seems to be a base ‘behavior’ for a feature,
it might be worthwhile to start thinking of these features as design patterns with the set of
default social actions being facilitated as a default property of the pattern.
Finally, this study was run as a controlled formative study. By nature, its a low-cost, high-
turnaround approach. Another approach to consider for future work on formative evaluation
of social network sites is a more longitudinal study, wherein a focus group uses the software
for period of time and then performs an evaluation. Such studies for social software have
been performed before [27], and its apparent that the participant responses might be more
in-depth in a longitudinal study, given that they use the software for a much longer period.
Also, it might be easier to evaluate some kinds of social actions - such as collaboration. In
case of a longitudinal study, a group of users can be given a common task to be performed,
whereas in the case of a controlled study it becomes harder to provide tasks that give a real
sense of collaboration. However, longitudinal studies might face the issue of users linking
the technology’s performance to the performance of the community of participants in the
study. They are also costlier to perform in terms of resources and time.
Bibliography
[1] User-generated content drives half of u.s. top 10 fastest growing web brands. Nielsen
NetRating, 2006.
[2] F. Ackerman, T. McGuire, T. Scott, and J. Peterson. Nifty assignments. Journal of

Computing Sciences in Colleges, 24(1):257–262, 2008.
[3] F. Adam and B. Roncevic. Social capital: recent debates and research trends. Social
Science Information, Jan 2003.
[4] P. Adler and S. Kwon. Social capital: the good, the bad, and the ugly. KNOWLEDGE
AND SOCIAL CAPITAL: FOUNDATIONS AND APPLICATIONS, Eric L. Lesser,
ed., Butterworth-Heinemann, Boston, MA, pages 89–115, 2000.
[5] S. Ahuja, M. Pérez-Quiñones, A. Kavanaugh, and C. Tauro. Colloki: Rethinking lo-

cal conversations on the web. Proceedings of Directions and Implications of Advanced
Computing - Conference on Online Deliberation, 2008.
[6] N. Ali-Hasan and L. Adamic. Expressing social relationships on the blog through links
and comments. Proceedings of the International Conference on Weblogs and Social
Media, Mar 2007.
[7] D. Beer. Social network (ing) sites. . . revisiting the story so far: A response to danah
boyd & nicole ellison. Journal of Computer-Mediated Communication, Jan 2008.
76
Sameer Ahuja Bibliography 77
[8] M. K. Boulos and S. Wheeler. The emerging web 2.0 social software: an enabling suite
of sociable technologies in health and health care education. Health Information &
Libraries Journal, Jan 2007.
[9] D. Boyd. The significance of social software. BlogTalks reloaded: Social software research
& cases, pages 15–30.
[10] D. Boyd. Why youth social network sites: The role of networked publics in teenage
social life. The John D. and Catherine T. MacArthur Foundation Series on Digital
Media and Learning, Jan 2007.
[11] D. Boyd and N. Ellison. Social network sites: Definition, history, and scholarship.
Journal of Computer-Mediated Communication, Jan 2008.
[12] M. Burke, C. Marlow, and T. Lento. Feed me: motivating newcomer contribution
in social network sites. CHI ’09: Proceedings of the 27th international conference on
Human factors in computing systems, Apr 2009.
[13] A. Carbone, J. Hurst, I. Mitchell, and D. Gunstone. Principles for designing program-
ming exercises to minimise poor learning behaviours in students. Proceedings of the
Australasian conference on Computing education, Jan 2000.
[14] C. Charron, J. Favier, and C. Li. Social computing. How Networks Erode Institutional
Power, Jan 2006.
[15] M. Choudhury, H. Sundaram, A. John, and D. Seligmann. What makes conversations

interesting?: themes, participants and consequences of conversations in online social
media. WWW ’09: Proceedings of the 18th international conference on World wide
web, Apr 2009.
[16] A. Cockburn. Agile software development. Jan 2002.
[17] D. Cohen and L. Prusak. In good company: How social capital makes organizations
work. Ubiquity, Jan 2001.
[18] D. Cosley, S. Lam, I. Albert, J. Konstan, and J. Riedl. Is seeing believing?: how
recommender system interfaces affect users’ opinions. CHI ’03: Proceedings of the
SIGCHI conference on Human factors in computing systems, Apr 2003.
[19] E. Courses and T. Surveys. A study of collaborative software development using group-
ware tools. Information Technology: Research and Education, Jan 2003.
[20] C. Dellarocas. Immunizing online reputation reporting systems against unfair ratings
and discriminatory behavior. Proceedings of the 2nd ACM Conference on Electronic
Commerce, Jan 2000.
[21] A. Dieberger, P. Dourish, K. Höök, and P. Resnick. Social navigation: Techniques for
building more usable systems. interactions, 7(6):36–45, Dec 2000.
[22] J. DiMicco, D. R. Millen, W. Geyer, C. Dugan, B. Brownholtz, and M. Muller. Moti-

vations for social networking at work. pages 711–720, 2008.
[23] S. H. Edwards, J. Börstler, L. N. Cassel, M. S. Hall, and J. Hollingsworth. Develop-

ing a common format for sharing programming assignments. ACM SIGCSE Bulletin,
40(4):167–182, 2008.
[24] S. H. Edwards and M. A. Perez-Quinones. Web-cat: automatically grading programming

assignments. pages 328–328, 2008.
[25] N. Ellison, C. Steinfield, and C. Lampe. Spatially bounded online social networks and
social capital. International Communication Association, Jan 2006.
[26] T. Erickson and W. Kellogg. Knowledge communities: Online environments for support-
ing knowledge management and its social context. Sharing Expertise: Beyond Knowl-
edge Management, Jan 2003.
[27] S. Farnham, S. Kelly, W. Portnoy, and J. Schwartz. Wallop: designing social software
for co-located social networks. Proceedings of the 37th Annual Hawaii International
Conference on System Sciences, Jan 2004.
[28] U. Farooq, T. Kannampallil, Y. Song, and C. Ganoe. Evaluating tagging behavior in

social bookmarking systems: metrics and design heuristics. Proceedings of the 2007
international ACM conference on Supporting group work, Jan 2007.
[29] G. Furnas, C. Fake, L. Ahn, J. Schachter, S. Golder, K. Fox, M. Davis, C. Marlow, and
M. Naaman. Why do tagging systems work? CHI ’06: CHI ’06 extended abstracts on
[30] S. Ghita, W. Nejdl, and R. Paiu. Semantically rich recommendations in social net-
works for sharing, exchanging and ranking semantic context. Lecture notes in computer
science, Jan 2005.
[31] S. Golder and B. Huberman. The structure of collaborative tagging systems. Journal
of Information Science, Jan 2006.
[32] S. Golder, D. Wilkinson, and B. Huberman. Rhythms of social interaction: Messaging

within a massive online network. Communities and Technologies 2007: Proceedings of
the Third International Conference on Communities and Technologies, Michigan State
University, East Lansing, Michigan, June 28-30, 2007, Jan 2007.
[33] F. Harper, S. Li, Y. Chen, and J. Konstan. Social comparisons to motivate contributions
to an online community. Lecture notes in computer science, Jan 2007.
[34] M. Hawkes, D. Talley, and J. Webster. An introduction to the multimedia educational

resource for learning and online teaching (merlot). Journal of Computing Sciences in
Colleges, 17(1):255–256, 2001.
[35] N. V. House. Flickr and public image-sharing: distant closeness and photo exhibition.
CHI ’07 extended abstracts on Human factors in computing systems, pages 2717–2722,
2007.
[36] J. Howarth, P. Pyla, B. Yost, and Y. Haciahmetoglu. Designing a conference for women
entering academe in the sciences and engineering. Advancing Women in Leadership
Online Journal, 23, 2007.
[37] C. Jensen, J. Davis, and S. Farnham. Finding others online: Reputation systems for
social online spaces. Proceedings of the SIGCHI conference on Human factors in com-
puting systems: Changing our world, changing ourselves, Jan 2002.
[38] A. Joinson. Looking at, looking up or keeping up with people?: motives and use
of facebook. CHI ’08: Proceeding of the twenty-sixth annual SIGCHI conference on
[39] S. Jones and S. Fox. Generations online in 2009. PEW Internet and American Life
Project, 2009.
[40] A. Kavanaugh, H. Kim, M. Pérez-Quiñones, and P. Isenhour. Models of government

blogging: Design trade-offs in civic engagement. Communities and Technologies 2007:
Proceedings of the Third International Conference on Communities and Technologies,
Michigan State University, East Lansing, Michigan, June 28-30, 2007, Jul 2007.
[41] A. Kittur, B. Suh, B. Pendleton, and H. Chi. He says, she says: Conflict and coordina-
tion in wikipedia. Proceedings of the SIGCHI conference on Human factors in computing
systems, pages 453–462, 2007.
[42] M. Klein, L. Iandoli, and I. Federico. Supporting collaborative deliberation using a large-
scale. Proceedings of Directions and Implications of Advanced Computing - Conference
on Online Deliberation, 2008.
[43] C. Lampe, N. Ellison, and C. Steinfield. A face (book) in the crowd: Social searching
vs. social browsing. Proceedings of the 2006 20th anniversary conference on Computer
supported cooperative work, pages 167–170, Jan 2006.
[44] P. Lange. Publicly private and privately public: Social networking on youtube. Journal
of Computer-Mediated Communication, Jan 2008.
[45] L. Layman, L. Williams, and K. Slaten. Note to self: make assignments meaningful.
volume 39, pages 459–463, 2007.
[46] K. Lerman. Social networks and social information filtering on digg. CoRR,
abs/cs/0612046, 2006.
[47] K. Lerman. Social information processing in news aggregation. Internet Computing,

IEEE, 11(6):16–28, 2007.
[48] C. Marlow, M. Naaman, D. Boyd, and M. Davis. Ht06, tagging paper, taxonomy, flickr,
academic article, to read. HYPERTEXT ’06: Proceedings of the seventeenth conference
on Hypertext and hypermedia, Aug 2006.
[49] S. Mitchell and W. Lutters. Assessing the value of computer science course material
repositories. 19th Conference on Software Engineering Education and Training Work-
shops, CSEETW ’06, Jan 2006.
[50] B. Nardi, D. Schiano, and M. Gumbrecht. Blogging as social activity, or, would you
let 900 million people read your diary? Proceedings of the 2004 ACM conference on
Computer supported cooperative work, Jan 2004.
[51] B. Nardi, D. Schiano, M. Gumbrecht, and L. Swartz. Why we blog. Communications

of the ACM, 47(12), Dec 2004.
[52] O. Nov. What motivates wikipedians? Communications of the ACM, 50(11):60–64,

2007.
[53] E. Orhun. Difficulties in integrating digital libraries into teaching: computer science
teaching with the merlot collection. pages 311–316, 2006.
[54] L. Page, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bring-
ing order to the web. 1999.
[55] N. Parlante. Nifty assignments. pages 112–113, 2008.
[56] N. Parlante, J. Cigas, A. B. Shiflet, R. Sooriamurthi, M. Clancy, B. Noonan, and

D. Reed. Nifty assignments. volume 39, pages 497–498, 2007.
[57] A. Portes. Social capital: its origins and applications in modern sociology. Annual
review of sociology, Jan 1998.
[58] B. Saulnier. “child is father to the man”: Social software in the is 2007 curriculum?
Information Systems Education Journal, Jan 2007.
[59] D. Schuler. Social computing. Communications of the ACM, Jan 1994.
[60] L. A. Smith, E. T. Smith, and T. F. Melder. A taste of merlot: tutorial presentation.

Journal of Computing Sciences in Colleges, 22(5):147–148, 2007.
[61] M. Smith. Invisible crowds in cyberspace. Communities in cyberspace, Jan 1999.
[62] T. Stafford, M. Stafford, and L. Schkade. Determining uses and gratifications for the
internet. Decision Sciences, Jan 2004.
[63] R. Stake. Standards-based & responsive evaluation. 2004.
[64] R. Stoeckl, P. Rohrmeier, and T. Hess. Motivations to produce user generated content:
differences between webloggers and videobloggers. Proceedings of the 20th Bled eCon-
ference: eMergence: Merging and Emerging Technologies, Processes, and Institutions.,
Jan 2007.
[65] L. Story. Apologetic, facebook changes ad program. New York Times, Jan 2007.
[66] C. Tauro, S. Ahuja, M. Pérez-Quiñones, A. Kavanaugh, and P. Isenhour. Deliberation in

the wild: a visualization tool for blog discovery and citizen-to-citizen participation. dg.o
’08: Proceedings of the 2008 international conference on Digital government research,

May 2008.
[67] M. Tepper. The rise of social software. netWorker, 7(3), Sep 2003.
[68] M. Tungare, X. Yu, W. Cameron, G. Teng, M. A. Pérez-Quiñones, L. Cassel, W. Fan,

and E. A. Fox. Towards a syllabus repository for computer science courses. volume 39,
pages 55–59, 2007.
[69] S. Verba and N. Nie. Participation in america: Political democracy and social equality.
Jan 1987.
[70] F. Wang, K. Carley, D. Zeng, and W. Mao. Social computing: From social informatics
to social intelligence. IEEE Intelligent Systems, Jan 2007.
[71] Y. Wang and D. Fesenmaier. Assessing motivation of contribution in online communi-

ties: An empirical investigation of an online travel community. Electronic Markets, Jan
2003.
[72] T. Weech. Merlot: Multimedia educational resource for learning and online teaching.
Proceedings of the Informing Science + IT Education Conference, 2002.
[73] B. Wellman, A. Quan-Haase, J. Boase, and W. Chen. The social affordances of the
internet for networked individualism. Journal of Computer-Mediated Communication,
Jan 2003.
[74] C. Weston, L. McAlpine, and T. Bordonaro. A model for understanding formative

evaluation in instructional design. Educational Technology Research and Development,
Jan 1995.
Appendix A
IRB Approvals
84
OfficeofofResearch
Office Research Compliance
Compliance
InstitutionalReview
Institutional Review Board
Board
2000Pratt
1880 Kraft Drive,
Drive Suite 2000 (0497)
(0497)
Blacksburg,
Virginia 24061
24061
540/231-4991
540/231-4991 Fax
Fax: 540/231-0959
540/231-0959
e-mailmoored@vt.edu
E-mail: moored@vt.edu
www.irb.vt.edu
www.irb.vt.edu
FWA00000572( expires 1/20/2010)
DATE: June 23, 2009 IRB # is IRB00000667
MEMORANDUM
TO: Manuel A. Perez-Quinones

Approval date: 6/23/2009
Continuing Review Due Date:6/8/2010
FROM: David M. Moore Expiration Date: 6/22/2010
SUBJECT: IRB Expedited Approval: “Evaluation of an Online System for Local Deliberation” ,
IRB # 09-567
This memo is regarding the above-mentioned protocol. The proposed research is eligible for
expedited review according to the specifications authorized by 45 CFR 46.110 and 21 CFR 56.110.
As Chair of the Virginia Tech Institutional Review Board, I have granted approval to the study for a
period of 12 months, effective June 23, 2009.
As an investigator of human subjects, your responsibilities include the following:
1. Report promptly proposed changes in previously approved human subject research

activities to the IRB, including changes to your study forms, procedures and
investigators, regardless of how minor. The proposed changes must not be initiated
without IRB review and approval, except where necessary to eliminate apparent
immediate hazards to the subjects.
2. Report promptly to the IRB any injuries or other unanticipated or adverse events
involving risks or harms to human research subjects or others.
3. Report promptly to the IRB of the study’s closing (i.e., data collecting and data
analysis complete at Virginia Tech). If the study is to continue past the expiration
date (listed above), investigators must submit a request for continuing
review prior to the continuing review due date (listed above). It is the researcher’s
responsibility to obtain re-approval from the IRB before the study’s expiration date.
4. If re-approval is not obtained (unless the study has been reported to the IRB as
closed) prior to the expiration date, all activities involving human subjects and
data analysis must cease immediately, except where necessary to eliminate
apparent immediate hazards to the subjects.
Important:
If you are conducting federally funded non-exempt research, please send the applicable OSP/grant
proposal to the IRB office, once available. OSP funds may not be released until the IRB has
compared and found consistent the proposal and related IRB applicaton.
cc: File
Invent the F u t u r e
V I R G I N I A P O L Y T E C H N I C I N S T I T U T E U N I V E R S I T Y
VIRGINIA POLYTECHNIC INSTITUTE AND A SNT A D S T A T E U N I V E R S I T Y
TE U N I V E R S I T Y
An equal opportunity, affirmative action i n s t i t u t i o n
85
OfficeofofResearch
Office ResearchCompliance
Compliance
InstitutionalReview
Institutional ReviewBoard
Board
2000Pratt
1880 Kraft Drive,
Drive Suite 2000 (0497)
(0497)
Blacksburg,
Virginia 24061
24061
540/231-4991Fax:
540/231-4991 Fax540/231-0959
540/231-0959
e-mail moored@vt.edu
E-mail: moored@vt.edu
www.irb.vt.edu
www.irb.vt.edu
FWA00000572( expires 1/20/2010)
DATE: July 8, 2009 IRB # is IRB00000667
MEMORANDUM
TO: Manuel A. Perez-Quinones

Sameer Ahuja
Stephen H. Edwards Approval date: 6/24/2009
Continuing Review Due Date:6/9/2010
FROM: David M. Moore Expiration Date: 6/23/2010
SUBJECT: IRB Amendment 1 Approval: “Evaluation of CATspace: A CS Assignment

Repository” , IRB # 09-577
This memo is regarding the above referenced protocol which was previously granted approval by the
IRB on June 24, 2009. You subsequently requested permission to amend your IRB application.
Since the requested amendment is nonsubstantive in nature, I, as Chair of the Virginia Tech
Institutional Review Board, have granted approval for requested protocol amendment, effective as of
July 8, 2009. The anniversary date will remain the same as the original approval date.
As an investigator of human subjects, your responsibilities include the following:
1. Report promptly proposed changes in previously approved human subject research

activities to the IRB, including changes to your study forms, procedures and
investigators, regardless of how minor. The proposed changes must not be initiated
without IRB review and approval, except where necessary to eliminate apparent
immediate hazards to the subjects.
2. Report promptly to the IRB any injuries or other unanticipated or adverse events
involving risks or harms to human research subjects or others.
3. Report promptly to the IRB of the study’s closing (i.e., data collecting and data
analysis complete at Virginia Tech). If the study is to continue past the expiration
date (listed above), investigators must submit a request for continuing
review prior to the continuing review due date (listed above). It is the researcher’s
responsibility to obtain re-approval from the IRB before the study’s expiration date.
4. If re-approval is not obtained (unless the study has been reported to the IRB as
closed) prior to the expiration date, all activities involving human subjects and
data analysis must cease immediately, except where necessary to eliminate
apparent immediate hazards to the subjects.
cc: File
Invent the F u t u r e
V I R G I N I VAI R P
G IONLI A
Y T E CP O
H LNYI TCE CI H
N NSI T
C I T U TI NES TUI TNUI TVEE R S AND
I T Y AS N
T ADT ES T AUTNE I V
U E
N R
I VS EI R
T S
YI T Y
An equal opportunity, affirmative action institution
86
Appendix B
Informed Consent Forms
87
!"#$"%"&'()*+,-./%".'"%0,",1,-'&%2'0,&,-'1%"!-#0",+''
!!
"3456789'.53:83;'456'(<6;=>=?<3;:'=3'#8:8<6>@'(65A8>;:'"3B5CB=3D'/E7<3'0EFA8>;:''
!!
!!
"#$%&!'(!)*'+&,$-!./0%10$#'2!'(!02!'2%#2&!343$&5!('*!%',0%!6&%#7&*0$#'2!
!!
82/&3$#90$'*:3;-!<*=!>021&%!?=!)@*&ABC1#D'2&3E!<*=!?26*&0!F0/02019GE!H05&&*!?G1+0=!!
!!
"G'(1#()0-')H',/"0'#-0-&#./I(#)J-.,''
!!
"G#3!#3!02!&IJ%'*0$'*4!3$164!$'!&/0%10$&!0!,'5512#$4!K&7!0JJ%#,0$#'2!6&/&%'J&6!('*!31JJ'*$#29!
%',0%!,'2/&*30$#'23E!,0%%&6!L'%%'M#=!"G&!J1*J'3&!'(!$G#3!*&3&0*,G!#3!$K'B('%6-!N2&!$'!J&*('*5!02!
&/0%10$#'2!'(!$G&!1307#%#$4!026!3',#0%!0(('*602,&!'(!$G&!,'5512#$4!0JJ%#,0$#'2E!026!$K'E!$G&!
3$164!#3!J0*$!'(!0!%0*9&*!&(('*$!$'!$0M&!0!(#*3$!3$&J!$'K0*63!('*50%#A0$#'2!'(!J*',&61*&3!026!#331&3!
#2!$G&!6&3#92!026!&/0%10$#'2!'(!,'5512#$4!3'($K0*&=!
!!
""G'(#).-21#-0''
!!
O#*3$E!$G&!J0*$#,#J02$!K'1%6!7&!9#/&2!0!3G'*$!)*&B./0%10$#'2!31*/&4!$G0$!K#%%!,'5J*#3&!'(!
P1&3$#'23!'2!#2$&*2&$!1309&E!K&73#$&!1309&E!026!0,06&5#,!70,M9*'126=!
!
H&,'26E!$G&!J0*$#,#J02$!K'1%6!7&!03M&6!$'!/#&K!0!3G'*$!/#6&'!6&5'!'(!'2&!'(!$G&!$K'!K&7!
0JJ%#,0$#'23=!
!
"G#*6E!$G&!J0*$#,#J02$!K'1%6!7&!03M&6!$'!J&*('*5!0!3&$!'(!$03M3!'2!$G0$!0JJ%#,0$#'2=!
?!/#6&'!*&,'*6#29!'(!$G&!3,*&&2!K'1%6!7&!$0M&2!KG&2!$G&!J0*$#,#J02$!K'1%6!7&!J&*('*5#29!$G&!
*&P1#*&6!$03M3=!Q'!016#'!K'1%6!7&!*&,'*6&6E!026!2'!*&,'*6#29!'(!$G&!J0*$#,#J02$!K'1%6!7&!
$0M&2!KG#%&!J&*('*5#29!$G&!$03M3=!
!
O'1*$GE!$G&!J0*$#,#J02$!K'1%6!7&!03M&6!$'!(#%%!0!J'3$B&/0%10$#'2!P1&3$#'220#*&!'2!$G&!0JJ%#,0$#'2!
703&6!'2!$G&!&/0%10$#'2!$03M3=!
!
"G&!3$164!K#%%!7&!,'261,$&6!#2!R''5!2'=!SSTU!#2!F2'K%&69&!V'*M3!88!:<&J0*$5&2$!'(!L'5J1$&*!
H,#&2,&;E!#2!$G&!L'*J'*0$&!R&3&0*,G!L&2$&*!0$!W#*9#2#0!"&,G=!"G&!066*&33!#3-!R''5!SSTUE!UUXU!
F*0($!<*#/&E!Y%0,M371*9E!W?!UZX[S=!!
!
"G&!3$164!K#%%!%03$!('*!2'!%'29&*!$G02!02!G'1*=!
!
!!
"""G'#"0K0''
88
!
"#$%&'&(#%&)*!&*!%+&,!$-,-#$'+!.&//!*)%!(/#'-!%+-!(#$%&'&(#*%!#%!0)$-!%+#*!0&*&0#/!$&,1!)2!(+3,&'#/!
)$!(,3'+)/)4&'#/!+#$05!!
!!
!"#$%&'&(!)*$$
!!
"#$%&'&(#%&)*!&*!%+&,!,%673!.&//!+-/(!&0($)8-!6*7-$,%#*7&*4!)2!9)'&#/!:22)$7#*'-;!#,!&%!#((/&-,!%)!
&*%-$#'%&8-!#((/&'#%&)*,5!<%!.)6/7!#/,)!+-/(!($)8&7-!2)6*7#%&)*#/!.)$1!2)$!%+-!2)$0#%&)*!)2!
7-,&4*!($&*'&(/-,!#*7!-8#/6#%&)*!($)'-76$-,!)2!')006*&%3!,&%-,5!:77&%&)*#//3;!%+&,!-8#/6#%&)*!
.)6/7!+-/(!6,!$-2&*-!=)//)1&;!#!/)'#/!')*8-$,#%&)*!%))/;!.+&'+!.-!&*%-*7!%)!$-/-#,-!.+-*!&%!
$-#'+-,!#!0#%6$-!,%#%-;!#,!#!2$--!)(-*>,)6$'-!%))/!2)$!/)'#/!')006*&%&-,5!
!
!!
"#$&+)&')$,($-','./!).$-'0$1,'(!0&')!-2!).$$
!!
?+-!$-,6/%,!)2!%+&,!,%673!.&//!@-!1-(%!,%$&'%/3!')*2&7-*%&#/5!A)6$!.$&%%-*!')*,-*%!&,!$-B6&$-7!2)$!
%+-!$-,-#$'+-$,!%)!$-/-#,-!#*3!7#%#!&7-*%&2&-7!.&%+!3)6!#,!#*!&*7&8&76#/!%)!#*3)*-!)%+-$!%+#*!
(-$,)**-/!.)$1&*4!)*!%+-!($)C-'%5!!?+-!&*2)$0#%&)*!3)6!($)8&7-!.&//!+#8-!3)6$!*#0-!$-0)8-7!
#*7!)*/3!#!,6@C-'%!*60@-$!.&//!&7-*%&23!3)6!76$&*4!#*#/3,-,!#*7!#*3!.$&%%-*!$-()$%,!)2!%+-!
$-,-#$'+5!A)6$!(-$,)*#//3!&7-*%&2&#@/-!&*2)$0#%&)*!.&//!@-!1-(%!6*7-$!/)'1!#*7!1-3!&*!%+-!)22&'-!
)2!D$5!E#*6-/!:5!"F$-G>H6&I)*-,;!J))0!K)5!LLMN;!O*)./-74-!P)$1,!<<5!!://!%+-!8&7-)!$-')$7&*4,!
.&//!@-!7-,%$)3-7!)*'-!%+-3!+#8-!@--*!%$#*,'$&@-75!!
!!
D#%#!.&//!@-!,%)$-7!,-'6$-/3!#*7!.&//!@-!0#7-!#8#&/#@/-!)*/3!&*!%+-!')*%-Q%!)2!$-,-#$'+!!
(6@/&'#%&)*,!#*7!7&,'6,,&)*5!K)!$-2-$-*'-!.&//!@-!0#7-!&*!)$#/!)$!.$&%%-*!$-()$%,!%+#%!')6/7!/&*1!
3)6!%)!%+-!7#%#!*)$!.&//!3)6!-8-$!@-!&7-*%&2&-7!#,!#!(#$%&'&(#*%!&*!%+-!($)C-'%5!://!7#%#!4#%+-$-7!
.&//!+#8-!3)6$!*#0-!$-0)8-7!#*7!)*/3!#!6,-$!*60@-$!.&//!&7-*%&23!-#'+!6,-$!76$&*4!#*#/3,-,!
#*7!#*3!.$&%%-*!$-()$%,!)2!%+-!$-,-#$'+5!!
!!
"!#$1,/3&'*-)!,'$$
!!
R3!(#$%&'&(#%&*4!&*!%+&,!($)C-'%!3)6!.&//!@-!$-.#$7-7!#!')6()*!.)$%+!SN5N!#%!%+-!R#'1,%$--%,!
J-,%#6$#*%;!/)'#%-7!#%!MTU!9!E#&*!9%;!R/#'1,@6$4;!V:!MWTXT5!
!!
"!!#$(4&&0,/$),$5!)604-5$$
!!
A)6!#$-!2$--!%)!.&%+7$#.!2$)0!%+&,!,%673!#%!#*3!%&0-!2)$!#*3!$-#,)*!#*7!.&%+)6%!(-*#/%35!!
!!
"!!!#$-334,"-2$,($4&*&-416!!
!!
?+&,!$-,-#$'+!+#,!@--*!#(($)8-7;!#,!$-B6&$-7;!@3!%+-!<*,%&%6%&)*#/!J-8&-.!R)#$7!2)$!($)C-'%,!
&*8)/8&*4!+60#*!,6@C-'%,!#%!V&$4&*&#!")/3%-'+*&'!<*,%&%6%-!#*7!9%#%-!Y*&8-$,&%3;!#*7!@3!%+-!
D-(#$%0-*%!)2!=)0(6%-$!9'&-*'-5!!
!!
89
!"#$%&'(!)!%&*(+,$'-,%.*,!/!0!(!-,$!
!!
"!#$%&'()*+%,!)-*..!($!/)*(+0+/)(.!+'!(1+2!2(&3,4!)'3!"!5'$6!$7!'$!*.)2$'!"!0)''$(!/)*(+0+/)(.8!"!6+%%!
5../!(1.!)0(+#+(+.2!)'3!+'7$*9)(+$'!3+20&22.3!0$'7+3.'(+)%4!2+'0.!$(1.*2!6+%%!:.!/)*(+0+/)(+'-!+'!
(1+2!*.2.)*018!!
!!
!!
"#$%&'(!)!%&*(+,$%-'1!,,!.*$!
!!!
"!1)#.!*.)3!)'3!&'3.*2()'3!(1.!+'7$*9.3!0$'2.'(!)'3!0$'3+(+$'2!$7!(1+2!/*$;.0(8!!"!1)#.!1)3!)%%!
9,!<&.2(+$'2!)'26.*.38!!"!1.*.:,!)05'$6%.3-.!(1.!):$#.!)'3!-+#.!9,!#$%&'()*,!0$'2.'(!7$*!
/)*(+0+/)(+$'!+'!(1+2!/*$;.0(8!!"7!"!/)*(+0+/)(.4!"!9),!6+(13*)6!)(!)',!(+9.!6+(1$&(!/.')%(,8!!"!
)-*..!($!):+3.!:,!(1.!*&%.2!$7!(1+2!/*$;.0(8!!
!!
!!
=====================================!!!!!!!!!!=========================!
>+-')(&*.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!?)(.!!
!!
!!
!!
==============================!! ! !!!!!!!!==========================!
@)9.!A/%.)2.!/*+'(B!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!C$'()0(D!E1$'.FG9)+%!AH/(+$')%B!!
!!
!!
>1$&%3!"!1)#.!)',!/.*(+'.'(!<&.2(+$'2!):$&(!(1+2!*.2.)*01!$*!+(2!0$'3&0(4!"!9),!0$'()0(D!!
!!
>)9..*!I1&;)!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!2)9..*8)1&;)J#(8.3&!
"'#.2(+-)($*!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.K9)+%!!
!!
!!
?*8!L)'&.%!I8!EM*.NKO&+P$'.2!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!/.*.NJ#(8.3&!!!
"'#.2(+-)($*4!Q)0&%(,!I3#+2$*!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.K9)+%!!
!!
!!
?*8!I'3*.)!R)#)')&-1!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!5)#)'J#(8.3&!!!
"'#.2(+-)($*4!C$KC1)+*!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.K9)+%!!
!!
!!
?)#+3!L8!L$$*.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!STUKVWXKTYYX!F!9$$*.3J#(8.3&!!!
C1)+*4!"Z[!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\.%./1$'.F.K9)+%!!
90
!"#$"%"&'()*+,-./%".'"%0,",1,-'&%2'0,&,-'1%"!-#0",+''
!!
"3456789'.53:83;'456'(<6;=>=?<3;:'=3'#8:8<6>@'(65A8>;:'"3B5CB=3D'/E7<3'0EFA8>;:''
!!
!!
"#$%&!'(!)*'+&,$-!./0%10$#'2!'(!34"560,&-!4!37!455#829&2$!:&6'5#$'*;!
!!
<2/&5$#80$'*=5>-!?*@!A021&%!4@!)B*&CDE1#F'2&5G!?*@!7$&/&2!H@!.IJ0*I5G!709&&*!4K1+0@!!
!!
"G'(1#()0-')H',/"0'#-0-&#./I(#)J-.,''
!!
"K#5!#5!02!&L6%'*0$'*;!5$1I;!$'!&/0%10$&!0!,'9912#$;!*&6'5#$'*;!'(!37!055#829&2$5G!,0%%&I!
34"560,&@!"K&!61*6'5&!'(!$K#5!*&5&0*,K!#5!$J'D('%I-!M2&!$'!6&*('*9!02!&/0%10$#'2!'(!$K&!
150N#%#$;!02I!5',#0%!0(('*I02,&!'(!34"560,&G!02I!$J'G!$K&!5$1I;!#5!60*$!'(!0!%0*8&*!&(('*$!$'!$0O&!
0!(#*5$!5$&6!$'J0*I5!('*90%#C0$#'2!'(!6*',&I1*&5!02I!#551&5!#2!$K&!I&5#82!02I!&/0%10$#'2!'(!
,'9912#$;!5'($J0*&@!
!!
""G'(#).-21#-0''
!!
P#*5$G!$K&!60*$#,#602$!J'1%I!N&!8#/&2!0!5K'*$!)*&D./0%10$#'2!51*/&;!$K0$!J#%%!,'96*#5&!'(!
Q1&5$#'25!'2!#2$&*2&$!1508&G!02I!0,0I&9#,!N0,O8*'12I@!
!
7&,'2IG!$K&!60*$#,#602$!J'1%I!N&!05O&I!$'!/#&J!0!5K'*$!/#I&'!I&9'!'(!34"560,&@!
!
"K#*IG!$K&!60*$#,#602$!J'1%I!N&!05O&I!$'!6&*('*9!0!5&$!'(!$05O5!'2!34"560,&@!
4!/#I&'!*&,'*I#28!'(!$K&!5,*&&2!J'1%I!N&!$0O&2!JK&2!$K&!60*$#,#602$!J'1%I!N&!6&*('*9#28!$K&!
*&Q1#*&I!$05O5@!R'!01I#'!J'1%I!N&!*&,'*I&IG!02I!2'!*&,'*I#28!'(!$K&!60*$#,#602$!J'1%I!N&!
$0O&2!JK#%&!6&*('*9#28!$K&!$05O5@!
!
P'1*$KG!$K&!60*$#,#602$!J'1%I!N&!05O&I!$'!(#%%!0!6'5$D&/0%10$#'2!Q1&5$#'220#*&!'2!$K&!066%#,0$#'2!
N05&I!'2!$K&!&/0%10$#'2!$05O5@!
!
"K&!5$1I;!J#%%!N&!,'2I1,$&I!#2!:''9!2'@!SSTU!#2!V2'J%&I8&!W'*O5!<<!=?&60*$9&2$!'(!3'961$&*!
7,#&2,&>G!#2!$K&!3'*6'*0$&!:&5&0*,K!3&2$&*!0$!X#*8#2#0!"&,K@!"K&!0II*&55!#5-!:''9!SSTUG!UUYU!
V*0($!?*#/&G!Z%0,O5N1*8G!X4!U[Y\S@!!
!
"K&!5$1I;!J#%%!%05$!('*!2'!%'28&*!$K02!02!K'1*@!
!
!!
"""G'#"0K0''
!
)0*$#,#60$#'2!#2!$K#5!*&5&0*,K!J#%%!2'$!6%0,&!$K&!60*$#,#602$!0$!9'*&!$K02!9#2#90%!*#5O!'(!6K;5#,0%!
'*!65;,K'%'8#,0%!K0*9@!!
91
!!
!"#$%&'&(!)*$$
!!
"#$%&'&(#%&)*!&*!%+&,!,%-./!0&11!+21(!&3($)42!-*.2$,%#*.&*5!)6!7)'&#1!866)$.#*'29!#,!&%!#((1&2,!%)!
&*%2$#'%&42!#((1&'#%&)*,:!;%!0)-1.!#1,)!+21(!($)4&.2!6)-*.#%&)*#1!0)$<!6)$!%+2!6)$3#%&)*!)6!
.2,&5*!($&*'&(12,!#*.!24#1-#%&)*!($)'2.-$2,!)6!')33-*&%/!,&%2,:!8..&%&)*#11/9!%+&,!24#1-#%&)*!
0)-1.!+21(!-,!$26&*2!=8>,(#'29!#!=)3(-%2$!7'&2*'2!#,,&5*32*%!$2(),&%)$/!#((1&'#%&)*9!0+&'+!
02!&*%2*.!%)!$212#,2!0+2*!&%!$2#'+2,!#!3#%-$2!,%#%29!#,!#!6$22!)(2*?,)-$'2!%))1!6)$!%+2!
=)3(-%2$!7'&2*'2!&*,%$-'%)$,!')33-*&%/:!
!!
"#$&+)&')$,($-','./!).$-'0$1,'(!0&')!-2!).$$
!!
>+2!$2,-1%,!)6!%+&,!,%-./!0&11!@2!<2(%!,%$&'%1/!')*6&.2*%&#1:!A)-$!0$&%%2*!')*,2*%!&,!$2B-&$2.!6)$!
%+2!$2,2#$'+2$,!%)!$212#,2!#*/!.#%#!&.2*%&6&2.!0&%+!/)-!#,!#*!&*.&4&.-#1!%)!#*/)*2!)%+2$!%+#*!
(2$,)**21!0)$<&*5!)*!%+2!($)C2'%:!!>+2!&*6)$3#%&)*!/)-!($)4&.2!0&11!+#42!/)-$!*#32!$23)42.!
#*.!)*1/!#!,-@C2'%!*-3@2$!0&11!&.2*%&6/!/)-!.-$&*5!#*#1/,2,!#*.!#*/!0$&%%2*!$2()$%,!)6!%+2!
$2,2#$'+:!A)-$!(2$,)*#11/!&.2*%&6&#@12!&*6)$3#%&)*!0&11!@2!<2(%!-*.2$!1)'<!#*.!<2/!&*!%+2!)66&'2!
)6!D$:!E#*-21!8:!"F$2G?H-&I)*2,9!J))3!K):!LLMN9!O*)012.52!P)$<,!;;:!!811!%+2!4&.2)!$2')$.&*5,!
0&11!@2!.2,%$)/2.!)*'2!%+2/!+#42!@22*!%$#*,'$&@2.:!!
!!
D#%#!0&11!@2!,%)$2.!,2'-$21/!#*.!0&11!@2!3#.2!#4#&1#@12!)*1/!&*!%+2!')*%2Q%!)6!$2,2#$'+!!
(-@1&'#%&)*,!#*.!.&,'-,,&)*:!K)!$262$2*'2!0&11!@2!3#.2!&*!)$#1!)$!0$&%%2*!$2()$%,!%+#%!')-1.!1&*<!
/)-!%)!%+2!.#%#!*)$!0&11!/)-!242$!@2!&.2*%&6&2.!#,!#!(#$%&'&(#*%!&*!%+2!($)C2'%:!811!.#%#!5#%+2$2.!
0&11!+#42!/)-$!*#32!$23)42.!#*.!)*1/!#!-,2$!*-3@2$!0&11!&.2*%&6/!2#'+!-,2$!.-$&*5!#*#1/,2,!
#*.!#*/!0$&%%2*!$2()$%,!)6!%+2!$2,2#$'+:!!
!!
"!#$1,/3&'*-)!,'$$
!!
R/!(#$%&'&(#%&*5!&*!%+&,!($)C2'%!/)-!0&11!@2!$20#$.2.!#!')-()*!0)$%+!SN:N!#%!%+2!R#'<,%$22%,!
J2,%#-$#*%9!1)'#%2.!#%!MTU!7!E#&*!7%9!R1#'<,@-$59!V8!MWTXT:!
!!
"!!#$(4&&0,/$),$5!)604-5$$
!!
A)-!#$2!6$22!%)!0&%+.$#0!6$)3!%+&,!,%-./!#%!#*/!%&32!6)$!#*/!$2#,)*!#*.!0&%+)-%!(2*#1%/:!!
!!
"!!!#$-334,"-2$,($4&*&-416!!
!!
>+&,!$2,2#$'+!+#,!@22*!#(($)42.9!#,!$2B-&$2.9!@/!%+2!;*,%&%-%&)*#1!J24&20!R)#$.!6)$!($)C2'%,!
&*4)14&*5!+-3#*!,-@C2'%,!#%!V&$5&*&#!")1/%2'+*&'!;*,%&%-%2!#*.!7%#%2!Y*&42$,&%/9!#*.!@/!%+2!
D2(#$%32*%!)6!=)3(-%2$!7'&2*'2:!!
!!
!"#$%&'(!)!%&*(+,$'-,%.*,!/!0!(!-,$$
!!
92
!"#$%&'()*+%,")-*.."($"/)*(+0+/)(."+'"(1+2"2(&3,4")'3"!"5'$6"$7"'$"*.)2$'"!"0)''$("/)*(+0+/)(.8"!"6+%%"
5../"(1.")0(+#+(+.2")'3"+'7$*9)(+$'"3+20&22.3"0$'7+3.'(+)%4"2+'0."$(1.*2"6+%%":."/)*(+0+/)(+'-"+'"
(1+2"*.2.)*018""
""
""
!"#$%&'()($%*'+,#$-&.(,,(/*#!
"""
!"1)#."*.)3")'3"&'3.*2()'3"(1."+'7$*9.3"0$'2.'(")'3"0$'3+(+$'2"$7"(1+2"/*$;.0(8""!"1)#."1)3")%%"
9,"<&.2(+$'2")'26.*.38""!"1.*.:,")05'$6%.3-."(1."):$#.")'3"-+#."9,"#$%&'()*,"0$'2.'("7$*"
/)*(+0+/)(+$'"+'"(1+2"/*$;.0(8""!7"!"/)*(+0+/)(.4"!"9),"6+(13*)6")(")',"(+9."6+(1$&("/.')%(,8""!"
)-*.."($"):+3.":,"(1."*&%.2"$7"(1+2"/*$;.0(8""
""
""
=====================================""""""""""========================="
>+-')(&*.""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""?)(.""
""
""
""
=============================="" " """"""""=========================="
@)9."A/%.)2."/*+'(B"""""""""""""""""""""""""""""""""""""""""""""""""""""""C$'()0(D"E1$'.FG9)+%"AH/(+$')%B""
""
""
>1$&%3"!"1)#.")',"/.*(+'.'("<&.2(+$'2"):$&("(1+2"*.2.)*01"$*"+(2"0$'3&0(4"!"9),"0$'()0(D""
""
>)9..*"I1&;)"""""""""""""""""""""""""""""""""""""""""""""""""""""""""2)9..*8)1&;)J#(8.3&"
!'#.2(+-)($*"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""".K9)+%""
""
""
?*8"L)'&.%"I8"EM*.NKO&+P$'.2""""""""""""""""""""""""""""""""/.*.NJ#(8.3&"""
!'#.2(+-)($*4"Q)0&%(,"I3#+2$*"""""""""""""""""""""""""""""""""""""""".K9)+%""
""
""
?*8">(.#.'"R8"G36)*32"""""""""""""""""""""""""""""""""""""""""""28.36)*32J#(8.3&"
!'#.2(+-)($*4"C$KC1)+*""""""""""""""""""""""""""""""""""""""""""""""""""".K9)+%""
""
""
?)#+3"L8"L$$*."""""""""""""""""""""""""""""""""""""""""""""""STUKVWXKTYYX"F"9$$*.3J#(8.3&"""
C1)+*4"!Z[""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""\.%./1$'.F.K9)+%""
""
"
93
Appendix C
Pre-Evaluation Survey
94
!"#$#%&'(&)*+,-.(#/)*+,,&*"#-
-
-
01-234-35678-93-:3;-;<7-6=7-53>>34?8@-47A-BCC>?DB6?38<E-!>7B<7-B99-B8:-36=7F-47A-BCC>?DB6?38<-B<-8779791-
-
!"#$% !$&$'()% .#%)$(+#% !$&$'()% 401$%(%2$$3% 5('$)-% 6$&$'%
#"*$+%(%,(-% /01$%(%,(-% #"*$+%(%
2$$3%
GBD7A33H1D3I- - - - - - -
)4?667F1D3I- - - - - - -
J:<CBD71D3I- - - - - - -
+FH;61D3I- - - - - - -
G>?DHF1D3I- - - - - - -
K3;);A71D3I- - - - - - -
L7>?D?3;<1D3I- - - - - - -
L?@@1D3I- - - - - - -
!"#$%&'''(
)*+$,&$(,--(.$+/01(
- - - - - - -
- - - - - - -
- - - - - - -
- - - - - - -
- - - - - - -
- - - - - - -
- - - - - - -
-
-
-
-
-
95
!"#$%&'#()*+#,-#&.')/)')01#%&/0#2,3#405-,56!"#$%#&'!(!#)!*#+,,-./+&.$%(#01./2#+--#&'+&#+,,-345#67$%&!%&8#.%#&'.(#
7301'),*#.&*#60&*#&88#()*+1#,-#.,*'0*'9#8)(0#4%,',19#/)+0,19#631).9#:8,;#4,1'19#*0<1#&5').8019#0'."#=++#6,50#
<0:1)'01#&1#*00+0+"#
#
!"#$% !&'(")*% 0./,'1")*% 4,55$)#")*%,)% 8,#")*9%('#")*%,(% :(,-7")*% :(,-7")*%
+,)#$)#%-"#&% 2,3(%,-)% +,)#$)#%,(% !"#$#%&'()*%+,%+% *+-,./'( *+-,./'(
,#&$(%.$,./$% +,)#$)#% .(,6"/$7% .(,6"/$7% +,)#$)#%
>&.0:,,(".,6# # # # # # #
?<)''05".,6# # # # # # #
@214&.0".,6# # # # # # #
A5(3'".,6# # # # # # #
>8).(5".,6# # # # # # #
B,3?3:0".,6# # # # # # #
C08).),31".,6# # # # # # #
C);;".,6# # # # # # #
!"#$%&'''(
)*+$,&$(,--(.$+/01(
# # # # # # #
# # # # # # #
# # # # # # #
# # # # # # #
# # # # # # #
# # # # # # #
# # # # # # #
# # # # # # #
# # # # # # #
#
%
%
%
96
!"#$%&'()*+,-./$&#'0*$.,
! !
"#!$%&'!(((((((((!)*+#!!
!
!
,#!-&.'!((((((((((!
!
!
/#!0112345678'!!
!
!!!!((((!-529&85!
!!!!((((!05:&*'!(((((((((((((((((((!
!
!!!;<!05:&*=!+>63!5:&!<7??7@68%!A2&+5678#!
!
!
B#!$*&!)72!4!
!
!!!(((!C41:&?7*+!+529&85!
!!!(((!D4+5&*+!+529&85!
!!!(((!E:F!+529&85!
!
97
Appendix D
Colloki Tasks
98
!"#$#%
!
!
"#$!%&'!(&#)*+'+!,*-.!%!,'/!/%0'+!%((1*2%-*#3!4#&!.#1+*35!2#3)'&0%-*#30!%3+!0.%&*35!1*360!%7#350-!7'7/'&0!
#4!-.'!5&%+$%-'!0-$+'3-!2#77$3*-8!%-!-.'!9#7($-'&!:2*'32'!+'(%&-7'3-!#4!;*&5*3*%!<'2.=!<.'!%((1*2%-*#3!*0!
!"##$%&'(#)*+,-&.#$"($&/"0!1&01$&(12&-!+'7#!)*+'#!#4!-.'!%((1*2%-*#3!/8!21*26*35!-.'!+'06-#(!0.#&-2$-!-*-1'+!
'(#)*+&%$32,-&45!$&62)&1"7$&/"0!1$%&01$&78%$29&+#$"($&!#8!:&25&01$&%$(:02+&(12*0!)0&#";$##$%&<(#)*+<&02&2+$5&
-.'!,'/0*-'>!%3+!('&4#&7!-.'!4#11#,*35!-%060!#3!-.'!0*-'?!
!
!
@= A0'!-.'!4#11#,*35!2&'+'3-*%10!-#!1#5!#3!-#!-.'!0*-'!B!
!
A0'&3%7'?!(%&-*2*(%3-!
C%00,#&+?!(%&-*2*(%3-!
!
!
D= =*2/($&01*2)>1&01$&#8(0&2?&#85:(&"5%&+2(0(&85&01$&02+8!&@A2!"#&B$(0")*"50(C-&D(&01$*$&"&*$(0")*"50&62)&:52/&85&
-.'!%&'%!-.%-!*0!7*00*35E!F++!%!1*36!-#!-.'!,'/0*-'!#4!-.%-!&'0-%$&%3-!-#!-.'!-#(*2!%1#35!,*-.!0#7'!
+'02&*(-*#3!%3+!-%50=!G"#$!2%3!$0'!,'/!0'%&2.!#&!#-.'&!-##10!-#!1##6$(!1#2%1!&'0-%$&%3-0!%3+H#&!-.'*&!
,'/0*-'0=I!
!
!
J= E2&02&01$&@E*"%)"0$&A8?$C&02+8!9&"5%&78$/&01$&&'#(%)*+*,(%-.,$#!*3!-.%-!-#(*2=!;#-'!4#&!-.'!1*36!-.%-!8#$!1*6'!
-.'!7#0-=!G"#$!2%3!21*26!-.&#$5.!#3!-.'!1*360!%3+!)*0*-!-.'!0*-'0!-#!+'2*+'!8#$&!4%)#&*-'!1*36I=!K#,>!1'%)'!%!
2#77'3-!#3!-.'!1*36!7'3-*#3*35!,.8!8#$!1*6'!*-=!
!
!
L= M.*2.!*0!-.'!'-/*#(%0'#(!85&01$&02+8!&@E*"%)"0$&A8?$CF&G*80$&62)*&"5(/$*&;$#2/-!
!
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN!
!
!
!"#$%&'$("))"*+,-$./'0%+",01$200/3'$%&2%$4"/$&25'$6/0%$6"+,'7$%&'$8"39/%'#$:;+',;'$7'92#%3',%$2%$<+#-+,+2$
!"#$%&'%&%()&*+&,"%',+*"-,%&-*%&)"%.-,")"',"*%.-%/"&)-.-(%01)"%&21+,%,$"%*"3&),0"-,4'%',+*"-,%#100+-.,56!
!
!
O= M.*2.!#3'!#4!-.'!4*)'!-#(*20!1*0-'+!#3!-.'!.#7'(%5'!+#!8#$!-.*36!*0!-.'!7#0-!(#($1%&!%7#350-!-.'!01$&(!
2#77$3*-8E!
!
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN!
!
!
!
99
!" #$%&$'()*'(+',$*'+%-*',(.%&/'0%/,*1'()',$*'$(2*.34*'1('5(6',$%)7'$3/'8**)',$*'2(/,'3&,%-*'9*&*),05'
32()4/,',$*'/069.'&(226)%,5:'
'
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;'
'
'
<" =9(2',$*'1*.39,2*),'>*8/%,*?'5(6'7)(>',$3,'@32**9'A$6B3'>(97/'%)',$*'/32*'9*/*39&$'49(6.',$3,'5(6'39*'
!"#$%$&#$'(!")(*!"'(+!,(!"(-./%0)(1"2$(34/56$(74/"'(+!,8(7!"'(4/#(#94(0$40.$',$3,'$*'$3/'$31',$*'2(/,'
%),*93&,%()/'>%,$'C-%3'&(22*),/D'()'/069."'E),*9'5(69'3)/>*9/'8*0(>"'
'
-:,$$%(;+/<:5&(/&$%(!'(4"(-./%0=(>>>>>>>>>>>>>>>('
'
F/*9'%1/'(+',>('6/*9/'$*'$3/'$31'2(/,'%),*93&,%()/'>%,$'C-%3'&(22*),/DG';;;;;;;;;;;;;;;;;;'
'
'
H" I3)6*0'A"'J*9*K'L6%)()*/'%/',$*'.9(+*//(9'>$('$*31/',$*'9*/*39&$'49(6.',$3,'5(6'39*'%),*9*/,*1'%)"'M%/'
/&$%(!'(4"(-./%0(!&(?,:0@A)(B!&(#40(#:C(4"(-./%0(%$0%$&$"#&(+!&(0%!,:%3(%$&$:%2+(:%$:=(?DEFA)(G:,$((4"$(
,4%$(/&$%(!"(#+$(&3&#$,(9+4&$(#40(#:C(!&(?DEFA)'
'
;;;;;;;;;;;;;;;;;;;;;'
'
'
100
Appendix E
CATspace Tasks
101
!"#$%&'()#"*+*)
!
!
"#$%&'()!*%!'!+'(),--.!'&&/*('0*-1!02'0!/)0%!"3!*1%045(0-4%!&-%06!%2'4)6!(-77)10!-16!'18!4'0)!"3!'%%*917)10%:!
!"#$%#&'$()*&(*#&%*+,(&-#.+&/0-#+&+1&(*#&$22"0)$(0+3&45&)"0)6037&(*#&-#%6(+2&%*+,()8(&(0("#-&9:;<%2$)#&-#.+=>&
;1()!<-5!2'=)!>'0(2)8!02)!=*8)-6!<-5!>*//!,)!&4-=*8)8!'!,4->%)4!>*18->!>*02!02)!'&&/*('0*-1!-&)1!*1!*0:!
!
+-4!02)!?-//->*19!0'%.%6!<-5!>*//!,)!/-99)8!*10-!02)!'&&/*('0*-1!'%!'!?*(0*-1'/!&4-?)%%-4!*1!"-7&50)4!3(*)1()6!
>2-%)!0)'(2)%!02)!*104-85(0-4<!"3!(/'%%:!@-5!'4)!*10)4)%0)8!*1!02)!0-&*(%!-?!'/9-4*027%6!'44'<%!'18!/*1.)8!/*%0%:!
A)4?-47!02)!?-//->*19!0'%.%!-1!02)!%*0)B!
!
C: ?+&(+&5+8,&:;<%2$)#&2,#1#,#3)#%&2$7#&$3-&$--&(*#&($7%&9:@A=&$3-&9$,,$5%=&(+&5+8,&"0%(&+1&($7&%84%),02(0+3%>!
!
!
D: @-5!'4)!/--.*19!?-4!'1!'%%*917)10!?-4!<-54!5&(-7*19!(/'%%!-1!0>-!8*7)1%*-1'/!'44'<%:!+*18!'1!'%%*917)10!
(*$(B%&($77#-&4+(*&9:@A=&$3-&9C-&$,,$5=>&D,0(#&(*#&$%%073.#3(B%&(0("#&4#"+'>&!
!
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE!
!
!
F: +*18!>2*(2!-?!<-54!+'(),--.!?4*)18%!'4)!'/4)'8<!5%*19!"#$%&'():!G'7)!-1)!-?!02)%)!?4*)18%!*1!02)!,/'1.!
,)/->:!
!
!
!
H: G'7)!0>-!-02)4!&)-&/)!02'0!02)!&)4%-1!*1!I5)%0*-1!F!2'%!'%!(-J'502-4%!-1!2*%K2)4!'%%*917)10%:!
!
!
!
L: 9@(#2*#3&E-'$,-%=&0%&$&/#,5&'#""&63+'3&:@&03%(,8)(+,!>-4.*19!*1!<-54!'4)'!-?!0)'(2*19!>2-!2'8!)'4/*)4!
7)10*-1)8!0-!<-5!02'0!2)!*%!'/4)'8<!-1!"#$%&'():!+*18!2*%!&4-?*/)!-1!"#$%&'():!M->!7'1<!'%%*917)10%!2'%!
@(#2*#3&284"0%*#-&+3&:;<%2$)#F&D*0)*&0%&(*#&"$(#%(&$%%073.#3(&(*$(&*#B%&284"0%*#-F&D*$(B%&*0%&(+2&,$(#8!
'%%*917)10N!
!
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE!
!
!
!
102
!" !"#$%&'(#)*+,-".#./0#.+1#.)-"#2+3#&.01/0,4#56,7#),+./03#8/+#/)"#$%&'(#)"#+,0#+2#/6"9/03#.+1#.)-":#
#
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$#
#
#
%" &'(#)*+,)*#-'#+'.-/,0(-*#-'#-1*#234567+*#+'88(.,-9#09#7)),.:#'.*#';#9'(/#'<.#755,:.8*.-5"#2/*7-*#7#.*<#
)""6-,*0,.#86./#./0#.6.;0#<&6,-;0#76*0,"6+,);#)33)="(#),7#.)-#6.#86./#.)-"#$%&'(#),7#$'7#)33)="(:#>1;+)7#./0#
755,:.8*.-"=,6#;6;0#)?)6;)@;0#+,#./0#70"A.+1#.+#./6"#)""6-,*0,.B"#26;0:#
#
#
>" 3;-*/#87?,.:#-1*5*#+17.:*5@#9'(#/*7A,=*#-17-#-1*#755,:.8*.-#7A5'#175#7#B(*5-,'.#'.#A,.?*)#A,5-5"#C),-#-1*#
)""6-,*0,.B"#.6.;0#.+#$&6,-;0#76*0,"6+,);#)33)="#),7#;6,A07#;6"."(C#),7#)77#$;6,A07#;6"."(#.'#-1*#A,5-#';#-7:5"#
#
#
D" D+EB30#,+8#26,6"/6,-#E1#./0#)""6-,*0,.:#F0.B"#*)A0#)#26,);#"0.#+2#G/),-0":#&0.#./0#*0.)7).)#13+103.=#G);;07#
$%+E3"0(#2+3#./6"#)""6-,*0,.#.+#$%&'(:#H;"+C#E17).0#./0#30)7*0:.I.#26;0#6,#./0#)""6-,*0,."B#;,A*5#E#+17.:*#-1*#
;,/5-#A,.*#';#-/0#26;0#23+*#$H33)="(#.+#$H33)="#),7#F6,A07#F6"."(:#
#
FG" H(0A,51#-1*#755,:.8*.-"#
#
#
#
#
#
#
#
103
Appendix F
Colloki Post-Evaluation Questionnaire
104
!"#$%&'()*($+",-.*/#$+",,(+0/-
-
-
- 1$0",2)3- 4+#(20//- 5/*$0()- 620//- 1$0",2)3-
4+#(20//- 620//-
- - - - - -
78-1)*09-:(#-/(#3-$"-*#/- 7- ;- <- =- >-
- - - - - -
;8-?-@"*,A-+$-/(#3-$"-B0/($/-9"#$#-"0-)+,C#-",- 7- ;- <- =- >-

1)*09-
- - - - - -
<8-?-@"*,A-+$-/(#3-$"-@+,A-$D/-)($/#$-9"#$#-(,A- 7- ;- <- =- >-

)+,C#-+,-1)*09-
- - - - - -
=8-?-@"*,A-+$-/(#3-$"-@+,A-$D/-E"#$-9"9*)(0- 7- ;- <- =- >-

9"#$#-(,A-)+,C#-+,-1)*09-
- - - - - -
>8-?-@"*,A-+$-/(#3-$"-@+,A-*#/0#-:D"-D('/- 7- ;- <- =- >-

#+E+)(0-$"9-$(2#-
- - - - - -
F8-?-@"*,A-+$-/(#3-$"-@+,A-:D+BD-*#/0#-D('/- 7- ;- <- =- >-

B"EE/,$/!"#$"%&'("#)(%*+,"-#,),"&$!"./$0,-
- - - - - -
G8-?-@"*,A-+$-/(#3-$"-@+,A-$D/-$"9-B",$0+H*$"0#- 7- ;- <- =- >-

@"0-9(0$+B*)(0-$"9+B#-
- - - - - -
!"#$% - - - - -
I8-?-@"*,A-+$-/(#3-$"-$(2-E3-9"#$#-(,A-)+,C#- 7- ;- <- =- >-
- - - - - -
105
! "#$%&'()! *+,-'$..! /.0#$-(! 1'$..! "#$%&'()!
*+,-'$..! 1'$..!
23!4!5%0&6!+#!.-,)!#%!7$%8,.!9%,#,!-&6!(+&:,! ;! <! =! >! ?!

0,+&'!#-',!
! ! ! ! ! !
;@3!A-',!8+((!7.!B.(950(!5%$!C.!#%!5+&6!9%,#,! ;! <! =! >! ?!

-&6!(+&:,!$.(.D-&#!#%!C)!+&#.$.,#!
! ! ! ! ! !
;;3!A-',!8+((!7.!B.(950(!5%$!C.!#%!5+&6!9.%9(.! ;! <! =! >! ?!

8+#B!,+C+(-$!+&#.$.,#,!-,!C.!
! ! ! ! ! !
;<3!A-',!8+((!7.!B.(950(!5%$!C.!#%!5+&6!C%$.! ;! <! =! >! ?!

-7%0#!9.%9(.!4!-($.-6)!:&%8!
! ! ! ! ! !
;=3!A-',!8+((!7.!B.(950(!5%$!C.!#%!+&#.$-E#! ;! <! =! >! ?!

8+#B!%#B.$!9.%9(.!%&!"(0$9!
! ! ! ! ! !
;>3!A-',!8+((!7.!B.(950(!+&!,B-$+&'!C)!9%,#,! ;! <! =! >! ?!

-&6!(+&:,!8+#B!%#B.$!0,.$,!%5!"(0$9!
! ! ! ! ! !
;?3!A-',!8+((!7.!B.(950(!+&!8%$:+&'!8+#B!0,.$,! ;! <! =! >! ?!

%&!"(0$9!#%!$.-EB!,%C.!E%CC%&!'%-(!!
! ! ! ! ! !
!"#"$%&'#%()(%*!! ! ! ! ! !
;F3!4!5%0&6!$.E.&#!-E#+D+#)!(+,#!+&#0+#+D.!-&6! ;! <! =! >! ?!

.-,)!#%!0,.!
! ! ! ! ! !
;G3!H.E.&#!1E#+D+#)!8+((!7.!B.(950(!5%$!C.!#%! ;! <! =! >! ?!

5+&6!9%,#,!-&6!(+&:,!$.(.D-&#!#%!C)!+&#.$.,#!
! ! ! ! ! !
106
! "#$%&'()! *+,-'$..! /.0#$-(! 1'$..! "#$%&'()!
*+,-'$..! 1'$..!
234!5.6.&#!16#+7+#)!8+((!9.!:.(;<0(!<%$!=.!#%! 2! ?! @! A! B!
<+&>!;.%;(.!8+#:!,+=+(-$!+&#.$.,#,!-,!=.!
! ! ! ! ! !
2C4!5.6.&#!16#+7+#)!8+((!9.!:.(;<0(!<%$!=.!#%! 2! ?! @! A! B!
<+&>!=%$.!-9%0#!;.%;(.!D!-($.->)!E&%8!
! ! ! ! ! !
?F4!5.6.&#!16#+7+#)!8+((!9.!:.(;<0(!<%$!=.!#%! 2! ?! @! A! B!
+&#.$-6#!8+#:!%#:.$!;.%;(.!%&!"(0$;!
! ! ! ! ! !
?24!5.6.&#!16#+7+#)!8+((!9.!:.(;<0(!+&!,:-$+&'! 2! ?! @! A! B!
=)!;%,#,!-&>!(+&E,!8+#:!%#:.$!0,.$,!%<!"(0$;!
! ! ! ! ! !
??4!5.6.&#!16#+7+#)!8+((!9.!:.(;<0(!+&!8%$E+&'! 2! ?! @! A! B!
8+#:!0,.$,!%&!"(0$;!#%!$.-6:!,%=.!6%==%&!
'%-(!!
! ! ! ! ! !
!"##$%&'( ! ! ! ! !
?@4!D!<%0&>!+#!.-,)!#%!=-E.!6%==.&#,! 2! ?! @! A! B!
! ! ! ! ! !
?@4!G%==.&#,!8+((!9.!:.(;<0(!<%$!=.!#%!<+&>! 2! ?! @! A! B!
;%,#,!-&>!(+&E,!$.(.7-&#!#%!=)!+&#.$.,#!
! ! ! ! ! !
?A4!G%==.&#,!8+((!9.!:.(;<0(!<%$!=.!#%!<+&>! 2! ?! @! A! B!
;.%;(.!8+#:!,+=+(-$!+&#.$.,#,!-,!=.!
! ! ! ! ! !
?B4!G%==.&#,!8+((!9.!:.(;<0(!<%$!=.!#%!<+&>! 2! ?! @! A! B!
=%$.!-9%0#!;.%;(.!D!-($.->)!E&%8!
! ! ! ! ! !
107
! "#$%&'()! *+,-'$..! /.0#$-(! 1'$..! "#$%&'()!
*+,-'$..! 1'$..!
234!5%66.&#,!7+((!8.!9.(:;0(!;%$!6.!#%! =! 2! >! ?! @!
+&#.$-<#!7+#9!%#9.$!:.%:(.!%&!"(0$:!
! ! ! ! ! !
2A4!5%66.&#,!7+((!8.!9.(:;0(!+&!,9-$+&'!6)! =! 2! >! ?! @!
:%,#,!-&B!(+&C,!7+#9!%#9.$!0,.$,!%;!"(0$:!
! ! ! ! ! !
2D4!5%66.&#,!7+((!8.!9.(:;0(!+&!7%$C+&'!7+#9! =! 2! >! ?! @!
0,.$,!%&!"(0$:!#%!$.-<9!,%6.!<%66%&!'%-(!!
! ! ! ! ! !
!"#$%& ! ! ! ! !
2E4!F!;%0&B!+#!.-,)!#%!G%#.!;%$!:%,#,!-&B!(+&C,! =! 2! >! ?! @!
! ! ! ! ! !
>H4!I%#.,!7+((!8.!9.(:;0(!;%$!6.!#%!;+&B!:%,#,! =! 2! >! ?! @!
-&B!(+&C,!$.(.G-&#!#%!6)!+&#.$.,#!
! ! ! ! ! !
>=4!I%#.,!7+((!8.!9.(:;0(!;%$!6.!#%!;+&B!:.%:(.! =! 2! >! ?! @!
7+#9!,+6+(-$!+&#.$.,#,!-,!6.!
! ! ! ! ! !
>24!I%#.,!7+((!8.!9.(:;0(!;%$!6.!#%!;+&B!6%$.! =! 2! >! ?! @!
-8%0#!:.%:(.!F!-($.-B)!C&%7!
! ! ! ! ! !
>>4!I%#.,!7+((!8.!9.(:;0(!;%$!6.!#%!+&#.$-<#! =! 2! >! ?! @!
7+#9!%#9.$!:.%:(.!%&!"(0$:!
! ! ! ! ! !
>?4!I%#.,!7+((!8.!9.(:;0(!+&!,9-$+&'!6)!:%,#,! =! 2! >! ?! @!
-&B!(+&C,!7+#9!%#9.$!0,.$,!%;!"(0$:!
! ! ! ! ! !
108
! "#$%&'()! *+,-'$..! /.0#$-(! 1'$..! "#$%&'()!
*+,-'$..! 1'$..!
234!5%#.,!6+((!7.!8.(9:0(!+&!6%$;+&'!6+#8! >! ?! 2! @! 3!
0,.$,!%&!"(0$9!#%!$.-<8!,%=.!<%==%&!'%-(!!
! ! ! ! ! !
!"#$%&$'()*#"% ! ! ! ! !
2A4!B!:%0&C!0,.$!9$%:+(.,!.-,)!#%!0&C.$,#-&C! >! ?! 2! @! 3!
! ! ! ! ! !
2D4!E,.$!F$%:+(.,!6+((!7.!8.(9:0(!:%$!=.!#%!:+&C! >! ?! 2! @! 3!
9%,#,!-&C!(+&;,!$.(.G-&#!#%!=)!+&#.$.,#!
! ! ! ! ! !
2H4!E,.$!F$%:+(.,!6+((!7.!8.(9:0(!:%$!=.!#%!:+&C! >! ?! 2! @! 3!
9.%9(.!6+#8!,+=+(-$!+&#.$.,#,!-,!=.!
! ! ! ! ! !
2I4!E,.$!F$%:+(.,!6+((!7.!8.(9:0(!:%$!=.!#%!:+&C! >! ?! 2! @! 3!
=%$.!-7%0#!9.%9(.!B!-($.-C)!;&%6!
! ! ! ! ! !
@J4!E,.$!F$%:+(.,!6+((!7.!8.(9:0(!:%$!=.!#%! >! ?! 2! @! 3!
+&#.$-<#!6+#8!%#8.$!9.%9(.!%&!"(0$9!
! ! ! ! ! !
@>4!E,.$!F$%:+(.,!6+((!7.!8.(9:0(!+&!,8-$+&'!=)! >! ?! 2! @! 3!
9%,#,!-&C!(+&;,!6+#8!%#8.$!0,.$,!%:!"(0$9!
! ! ! ! ! !
@?4!E,.$!F$%:+(.,!6+((!7.!8.(9:0(!+&!6%$;+&'! >! ?! 2! @! 3!
6+#8!0,.$,!%&!"(0$9!#%!$.-<8!,%=.!<%==%&!
'%-(!!
!
!
!
!
!
!
109
!
!"#$%&'()'$*'+,-.+$(+/$.,0'1$2'+'3-,)$.3$3'(,41')$.3$5&416$2'&.7#$8.4$9(+$(::$3'(,41')$,0(,$/.4$:.+;,$)''$-+$
"#$!%&'"(!
!
!
!"#$%&"' (")"*+$,'
)*++&,+! !
-."&/&"0!1"2$*3'! !
4533$,"&,+! !
65"&,+! !
7'$2!8259&%$'! !
!"#$%!"#$%&'(&"'))"*&%+,-&
! !
! !
! !
!
!
::(!-2$!05;!.5,.$2,$<!*=5;"!*,0!>*0'!"#*"!;'$2'!3&+#"!*=;'$!"#$!9$*";2$'!?25/&<$<!5,!1%;2?@!-,0!*<<&"&5,*%!
.533$,"'!05;!#*/$!5,!?*2"&.;%*2!9$*";2$'@!8%$*'$!*<<!"#$3!=$%5>(!
!
!
!"#$%&"' -.//")$,'
)*++&,+! !
-."&/&"0!1"2$*3'! !
4533$,"&,+! !
65"&,+! !
7'$2!8259&%$'! !
./0&1!"#$%&'(&"'))"*&%+,-&
! !
! !
110
!"#$%&'()'$(**$(+,$-./'0$1-22'+.)$-+$3&405$/'0'#$6789'$108.8:4'$-;$;'(.40')<$(**8.8-+(&$;'(.40')$./(.$,-4$./8+9$
=8&&$>'$4)';4&<$(+*$)-$-+#?$$
$
111
Appendix G
CATspace Post-Evaluation
Questionnaire
112
!"#$%&'()*($+",-.*/#$+",,(+0/-
!
!
! "#$%&'()! *+,-'$..! /.0#$-(! 1'$..! "#$%&'()!
*+,-'$..! 1'$..!
! ! ! ! ! !
23!415,6-7.!8-,!.-,)!#%!0,.! 2! 9! :! ;! <!
! ! ! ! ! !
93!=!>%0&?!+#!.-,)!#%!7$.-#.!-&!-,,+'&@.&#!%&! 2! 9! :! ;! <!
415,6-7.!
! ! ! ! ! !
:3!=!>%0&?!+#!.-,)!#%!@-&+60(-#.!?+>>.$.&#! 2! 9! :! ;! <!
-##$+A0#.,!%>!-&!-,,+'&@.&#!%&!415,6-7.!
! ! ! ! ! !
;3!=!>%0&?!+#!.-,)!#%!-??!-&?!.?+#!>+(.,!%>! 2! 9! :! ;! <!
-,,+'&@.&#,!%&!415,6-7.!
! ! ! ! ! !
<3!=!>%0&?!+#!.-,)!#%!+?.&#+>)!#B.!#%6!#-',!>%$! 2! 9! :! ;! <!
6.%6(.!
! ! ! ! ! !
C3!=!>%0&?!+#!.-,)!#%!>+&?!#B.!(-#.,#!-7#+D+#)!+&! 2! 9! :! ;! <!
415,6-7.!
! ! ! ! ! !
E3!=!>%0&?!+#!.-,)!#%!>+&?!6.%6(.!#B-#!B-D.! 2! 9! :! ;! <!
,+@+(-$!#-',!
! ! ! ! ! !
F3 =!>%0&?!+#!.-,)!#%!>+&?!6.%6(.!8B%!-$.!7%G 2! 9! :! ;! <!
-0#B%$,!%&!-,,+'&@.&#,!8+#B+&!415,6-7.!
! ! ! ! ! !
113
! "#$%&'()! *+,-'$..! /.0#$-(! 1'$..! "#$%&'()!
*+,-'$..! 1'$..!
23!4!5%0&6!+#!.-,)!#%!5+&6!7)!8-9.:%%;!5$+.&6,! ?! @! A! B! C!
%&!<1=,>-9.!
! ! ! ! ! !
!"#$% ! ! ! ! !
?D3!4!5%0&6!+#!.-,)!#%!#-'!7)!-,,+'&7.&#,! ?! @! A! B! C!
! ! ! ! ! !
??3!4!5%0&6!+#!.-,)!#%!:$%E,.!-,,+'&7.&#,! ?! @! A! B! C!
0,+&'!#-',!
! ! ! ! ! !
?@3!=-',!E+((!:.!F.(>50(!5%$!7.!#%!5+&6! ?! @! A! B! C!
-,,+'&7.&#,!$.(.G-&#!#%!7)!+&#.$.,#!
! ! ! ! ! !
?A3!=-',!E+((!:.!F.(>50(!5%$!7.!#%!5+&6!>.%>(.! ?! @! A! B! C!
E+#F!,+7+(-$!+&#.$.,#,!-,!7.!
! ! ! ! ! !
?B3!=-',!E+((!:.!F.(>50(!5%$!7.!#%!5+&6!7%$.! ?! @! A! B! C!
-:%0#!>.%>(.!4!-($.-6)!;&%E!
! ! ! ! ! !
?C3!=-',!E+((!:.!F.(>50(!5%$!7.!#%!+&#.$-9#! ?! @! A! B! C!
E+#F!%#F.$!>.%>(.!%&!<1=,>-9.!
! ! ! ! ! !
?H3!=-',!E+((!:.!F.(>50(!+&!,F-$+&'!7)! ?! @! A! B! C!
-,,+'&7.&#,!E+#F!%#F.$!0,.$,!%5!<1=,>-9.!
! ! ! ! ! !
?I3!=-',!E+((!:.!F.(>50(!+&!E%$;+&'!E+#F!0,.$,! ?! @! A! B! C!
%&!<1=,>-9.!#%!$.-9F!,%7.!9%77%&!'%-(!!
! ! ! ! ! !
114
! "#$%&'()! *+,-'$..! /.0#$-(! 1'$..! "#$%&'()!
*+,-'$..! 1'$..!
!"#$%#&'(#)*)#+!! ! ! ! ! !
234!5!6%0&7!(-#.,#!-8#+9+#)!(+,#!+&#0+#+9.!-&7! 2! :! ;! <! =!
.-,)!#%!0,.!
! ! ! ! ! !
2>4!?-#.,#!18#+9+#)!@+((!A.!B.(C60(!6%$!D.!#%! 2! :! ;! <! =!
6+&7!-,,+'&D.&#,!$.(.9-&#!#%!D)!+&#.$.,#!
! ! ! ! ! !
:E4!?-#.,#!18#+9+#)!@+((!A.!B.(C60(!6%$!D.!#%! 2! :! ;! <! =!
6+&7!C.%C(.!@+#B!,+D+(-$!+&#.$.,#,!-,!D.!
! ! ! ! ! !
:24!?-#.,#!18#+9+#)!@+((!A.!B.(C60(!6%$!D.!#%! 2! :! ;! <! =!
6+&7!D%$.!-A%0#!C.%C(.!5!-($.-7)!F&%@!
! ! ! ! ! !
::4!?-#.,#!18#+9+#)!@+((!A.!B.(C60(!6%$!D.!#%! 2! :! ;! <! =!
+&#.$-8#!@+#B!%#B.$!C.%C(.!%&!G1H,C-8.!
! ! ! ! ! !
:;4!?-#.,#!18#+9+#)!@+((!A.!B.(C60(!+&!,B-$+&'! 2! :! ;! <! =!
D)!-,,+'&D.&#,!@+#B!%#B.$!0,.$,!%6!G1H,C-8.!
! ! ! ! ! !
:<4!?-#.,#!18#+9+#)!@+((!A.!B.(C60(!+&!@%$F+&'! 2! :! ;! <! =!
@+#B!0,.$,!%&!G1H,C-8.!#%!$.-8B!,%D.!
8%DD%&!'%-(!!
! ! ! ! ! !
,-..$/#%& ! ! ! ! !
:=4!G%DD.&#,!@+((!A.!B.(C60(!6%$!D.!#%!6+&7! 2! :! ;! <! =!
-,,+'&D.&#,!$.(.9-&#!#%!D)!+&#.$.,#!
! ! ! ! ! !
115
! "#$%&'()! *+,-'$..! /.0#$-(! 1'$..! "#$%&'()!
*+,-'$..! 1'$..!
234!5%66.&#,!7+((!8.!9.(:;0(!;%$!6.!#%!;+&<! =! 2! >! ?! @!
:.%:(.!7+#9!,+6+(-$!+&#.$.,#,!-,!6.!
! ! ! ! ! !
2A4!5%66.&#,!7+((!8.!9.(:;0(!;%$!6.!#%!;+&<! =! 2! >! ?! @!
6%$.!-8%0#!:.%:(.!B!-($.-<)!C&%7!
! ! ! ! ! !
2D4!5%66.&#,!7+((!8.!9.(:;0(!;%$!6.!#%! =! 2! >! ?! @!
+&#.$-E#!7+#9!%#9.$!:.%:(.!%&!51F,:-E.!
! ! ! ! ! !
2G4!5%66.&#,!7+((!8.!9.(:;0(!+&!,9-$+&'!6)! =! 2! >! ?! @!
-,,+'&6.&#,!7+#9!%#9.$!0,.$,!%;!51F,:-E.!
! ! ! ! ! !
>H4!5%66.&#,!7+((!8.!9.(:;0(!+&!7%$C+&'!7+#9! =! 2! >! ?! @!
0,.$,!%&!51F,:-E.!#%!$.-E9!,%6.!E%66%&!
'%-(!!
! ! ! ! ! !
!"#$%&'( ! ! ! ! !
>=4!I-#+&',!7+((!8.!9.(:;0(!;%$!6.!#%!;+&<! =! 2! >! ?! @!
-,,+'&6.&#,!$.(.J-&#!#%!6)!+&#.$.,#!
! ! ! ! ! !
>24!I-#+&',!7+((!8.!9.(:;0(!;%$!6.!#%!;+&<! =! 2! >! ?! @!
:.%:(.!7+#9!,+6+(-$!+&#.$.,#,!-,!6.!
! ! ! ! ! !
>>4!I-#+&',!7+((!8.!9.(:;0(!;%$!6.!#%!;+&<! =! 2! >! ?! @!
6%$.!-8%0#!:.%:(.!B!-($.-<)!C&%7!
! ! ! ! ! !
>?4!I-#+&',!7+((!8.!9.(:;0(!;%$!6.!#%!+&#.$-E#! =! 2! >! ?! @!
7+#9!%#9.$!:.%:(.!%&!51F,:-E.!
116
! "#$%&'()! *+,-'$..! /.0#$-(! 1'$..! "#$%&'()!
*+,-'$..! 1'$..!
! ! ! ! ! !
234!5-#+&',!6+((!7.!8.(9:0(!+&!,8-$+&'!;)! ?! @! 2! A! 3!
-,,+'&;.&#,!6+#8!%#8.$!0,.$,!%:!<1=,9->.!
! ! ! ! ! !
2B4!5-#+&',!6+((!7.!8.(9:0(!+&!6%$C+&'!6+#8! ?! @! 2! A! 3!
0,.$,!%&!<1=,9->.!#%!$.->8!,%;.!>%;;%&!
'%-(!!
! ! ! ! ! !
!"#$%&$'()*#"% ! ! ! ! !
2D4!E!:%0&F!0,.$!9$%:+(.,!.-,)!#%!0&F.$,#-&F! ?! @! 2! A! 3!
! ! ! ! ! !
2G4!H,.$!I$%:+(.,!6+((!7.!8.(9:0(!:%$!;.!#%!:+&F! ?! @! 2! A! 3!
-,,+'&;.&#,!$.(.J-&#!#%!;)!+&#.$.,#!
! ! ! ! ! !
2K4!H,.$!I$%:+(.,!6+((!7.!8.(9:0(!:%$!;.!#%!:+&F! ?! @! 2! A! 3!
9.%9(.!6+#8!,+;+(-$!+&#.$.,#,!-,!;.!
! ! ! ! ! !
AL4!H,.$!I$%:+(.,!6+((!7.!8.(9:0(!:%$!;.!#%!:+&F! ?! @! 2! A! 3!
;%$.!-7%0#!9.%9(.!E!-($.-F)!C&%6!
! ! ! ! ! !
A?4!H,.$!I$%:+(.,!6+((!7.!8.(9:0(!:%$!;.!#%! ?! @! 2! A! 3!
+&#.$->#!6+#8!%#8.$!9.%9(.!%&!<1=,9->.!
! ! ! ! ! !
A@4!H,.$!I$%:+(.,!6+((!7.!8.(9:0(!+&!,8-$+&'!;)! ?! @! 2! A! 3!
-,,+'&;.&#,!6+#8!%#8.$!0,.$,!%:!<1=,9->.!
! ! ! ! ! !
117
! "#$%&'()! *+,-'$..! /.0#$-(! 1'$..! "#$%&'()!
*+,-'$..! 1'$..!
234!5,.$!6$%7+(.,!8+((!9.!:.(;70(!+&!8%$<+&'! A! B! 3! 2! C!
8+#:!0,.$,!%&!=1>,;-?.!#%!$.-?:!,%@.!
?%@@%&!'%-(!!
! ! ! ! ! !
!"#$%"&'()*(+,& ! ! ! ! !
224!D!7%0&E!#:.!;.%;(.!;-'.!.-,)!#%! A! B! 3! 2! C!
0&E.$,#-&E!
! ! ! ! ! !
2C4!6.%;(.!(+,#+&'!8+((!9.!:.(;70(!7%$!@.!#%! A! B! 3! 2! C!
7+&E!-,,+'&@.&#,!$.(.F-&#!#%!@)!+&#.$.,#!
! ! ! ! ! !
2G4!6.%;(.!(+,#+&'!8+((!9.!:.(;70(!7%$!@.!#%! A! B! 3! 2! C!
7+&E!;.%;(.!8+#:!,+@+(-$!+&#.$.,#,!-,!@.!
! ! ! ! ! !
2H4!6.%;(.!(+,#+&'!8+((!9.!:.(;70(!7%$!@.!#%! A! B! 3! 2! C!
7+&E!@%$.!-9%0#!;.%;(.!D!-($.-E)!<&%8!
! ! ! ! ! !
2I4!6.%;(.!(+,#+&'!8+((!9.!:.(;70(!7%$!@.!#%! A! B! 3! 2! C!
+&#.$-?#!8+#:!%#:.$!;.%;(.!%&!=1>,;-?.!
! ! ! ! ! !
2J4!6.%;(.!(+,#+&'!8+((!9.!:.(;70(!+&!,:-$+&'!@)! A! B! 3! 2! C!
-,,+'&@.&#,!8+#:!%#:.$!0,.$,!%7!=1>,;-?.!
! ! ! ! ! !
-))(,+."+*&/0(*1(+1$%23"& ! ! ! ! !
CK4!D!7%0&E!+#!.-,)!#%!0&E.$,#-&E!:%8!#%!.E+#! A! B! 3! 2! C!
#:.!E+77.$.&#!;$%;.$#+.,!%7!-&!-,,+'&@.&#!
! ! ! ! ! !
118
! "#$%&'()! *+,-'$..! /.0#$-(! 1'$..! "#$%&'()!
*+,-'$..! 1'$..!
234!1,,+'&5.&#!.6+#+&'!7.-#0$.,!8+((!9.! 3! =! >! ?! 2!
:.(;70(!7%$!5.!#%!7+&6!-,,+'&5.&#,!$.(.<-&#!#%!
5)!+&#.$.,#!
! ! ! ! ! !
2=4!1,,+'&5.&#!.6+#+&'!7.-#0$.,!8+((!9.! 3! =! >! ?! 2!
:.(;70(!7%$!5.!#%!7+&6!;.%;(.!8+#:!,+5+(-$!
+&#.$.,#,!-,!5.!
! ! ! ! ! !
2>4!1,,+'&5.&#!.6+#+&'!7.-#0$.,!8+((!9.! 3! =! >! ?! 2!
:.(;70(!7%$!5.!#%!7+&6!5%$.!-9%0#!;.%;(.!@!
-($.-6)!A&%8!
! ! ! ! ! !
2?4!1,,+'&5.&#!.6+#+&'!7.-#0$.,!8+((!9.! 3! =! >! ?! 2!
:.(;70(!7%$!5.!#%!+&#.$-B#!8+#:!%#:.$!;.%;(.!
%&!C1D,;-B.!
! ! ! ! ! !
224!1,,+'&5.&#!.6+#+&'!7.-#0$.,!8+((!9.! 3! =! >! ?! 2!
:.(;70(!+&!,:-$+&'!5)!-,,+'&5.&#,!8+#:!%#:.$!
0,.$,!%7!C1D,;-B.!
!
!"#$%&'()'$*'+,-.+$(+/$.,0'1$2'+'3-,)$.3$3'(,41')$.3$567)8(9'$2'&.:#$;.4$9(+$(<<$3'(,41')$,0(,$/.4$<.+=,$)''$
+&!#:.!(+,#4!
!
!
!"#$%&"' (")"*+$,'
D-''+&'! !
E-#.,#!1B#+<+#)! !
C%55.&#+&'! !
F-#+&',! !
G,.$!H$%7+(.,! !
119
!"#$%&"' (")"*+$,'
!"#$%"&'()*(+,& &
-))(,+."+*&"/(*0(+0 &
$%12"&
!"#$%&'()*$+,$'+--'.$*/01!
& &
& &
& &
&
&
345&-6"&7#8&2#+2"6+"/&19#8*&1+7&:17)&*;1*&8)"6)&.(,;*&198)"&*;"&<"1*86")&$6#=(/"/&#+&>-?)$12"@&-+7&
1//(*(#+1%&2#.."+*)&7#8&;1="&#+&$16*(28%16&<"1*86")@&!%"1)"&1//&*;".&9"%#:5&
&
&
!"#$%&"' -.//")$,'
?1,,(+,& &
-2*(=(*7&A*6"1.)& &
>#.."+*(+,& &
B1*(+,)& &
C)"6&!6#<(%")& &
!"#$%"&'()*(+,& &
-))(,+."+*&"/(*0(+0 &
$%12"&
!"#$%&'()*$+,$'+--'.$*/01!
& &
& &
& &
&
&
120
!"#$%&'()'$(**$(+,$-./'0$1-22'+.)$-+$345)6(1'$/'0'#$789:'$109.9;<'$-=$='(.<0')>$(**9.9-+(&$='(.<0')$./(.$,-<$
./9+:$?9&&$@'$<)'=<&>$(+*$)-$-+#A$$
$
121

Sameer Ahuja Thesis

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Sameer Ahuja Thesis

Uploaded by

Copyright:

Available Formats

A Tale of Two Sites: An Explorative Study of the Design and

Evaluation of Social Network Sites

Thesis submitted to the Faculty of the

Dr. Manuel A. Pérez-Quiñones, Chair

July 17, 2009

Keywords: Social Network Software, Usability Evaluation, Social Affordance, Design

1.3 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.3.1 Why Evaluate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.4 Overview of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.1 Social Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.1.2 Activity Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.1.3 User Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.1.5 Ratings and Votes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.2 Social Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.2.1 Social Browsing and Search . . . . . . . . . . . . . . . . . . . . . . . 16

2.2.3 Sharing Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.3 Local Deliberation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.3.1 Low Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.3.2 Automated relevance algorithms . . . . . . . . . . . . . . . . . . . . . 19

2.3.3 Citizen Participation . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.3.4 Centralization of content . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.4 Computer Science Assignment Repositories . . . . . . . . . . . . . . . . . . . 20

3.1.2 System Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.2.2 System Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

3.3 Social Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3.3.2 Activity Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

3.3.5 Ratings and Votes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

4.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

4.2.1 Selection of Participants . . . . . . . . . . . . . . . . . . . . . . . . . 50

4.2.2 Usability Lab Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.2.3 Evaluation Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.3 Benchmark Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.4 Perceived Social Affordance . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.4.1 Simulated Community . . . . . . . . . . . . . . . . . . . . . . . . . . 56

4.4.2 Mapping social features to actions . . . . . . . . . . . . . . . . . . . . 56

5.1 Overall Usability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

5.2.2 Activity Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

5.2.5 Ratings and Votes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

6 Discussion and Conclusions 70

6.1.1 Articulation of connections . . . . . . . . . . . . . . . . . . . . . . . . 70

6.1.2 Activity Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

6.1.3 Task Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

6.2 Evaluation Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

6.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

B Informed Consent Forms 87

E CATspace Tasks 101

F Colloki Post-Evaluation Questionnaire 104

G CATspace Post-Evaluation Questionnaire 112

3.1 A sample topic page in Colloki . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.2 Colloki Data Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.3 CATspace Homepage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

3.4 Representation of information that the different CATspace views display. . . 34

3.5 Object model and basic design of CATspace . . . . . . . . . . . . . . . . . . 37

3.6 CATspace data model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

3.7 Interaction between CATspace and Facebook . . . . . . . . . . . . . . . . . . 40

3.8 A user profile on Colloki . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

3.9 A user profile on CATspace . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.1 Participant responses for questions on overall usability . . . . . . . . . . . . 59

5.2 Participant responses for questions on tagging . . . . . . . . . . . . . . . . . 61

5.3 Participant responses for questions on activity streams . . . . . . . . . . . . 64

5.4 Participant responses for questions on user profiles . . . . . . . . . . . . . . . 66

5.5 Participant responses for questions on comments . . . . . . . . . . . . . . . . 68

5.6 Participant responses for questions on votes/ratings . . . . . . . . . . . . . . 69