Anger Management:
Using Sentiment Analysis to Manage Online Communities
Sara Owsley Sood
Pomona College
185 East Sixth Street
Claremont, CA 91711
sara.owsleysood@pomona.edu
ABSTRACT
As online communities grow and user generated content
increases, the need for community management also rises.
Community management has three main purposes: to create a
positive experience for existing participants, to encourage
appropriate, socio-normative behaviors, and to encourage
potential participants to make contributions. Research indicates
that the quality of content a potential participant sees on a site is
highly influential; off-topic, negative comments are a particularly
strong boundary to participation. A problem for community
managers, therefore, is the detection and elimination of such
undesirable content. As a community grows this task becomes
more daunting. In this paper, we introduce an approach to
automatic detection of inappropriate negative user contributions.
We apply sentiment analysis techniques to the task of classifying
short comments about news stories, a challenge due to the lack of
context in the short comments. In combination with relevance
detection techniques, this work will inform a hybrid community
management system by automatically flagging off-topic and
hostile comments.
Keywords
online communities, sentiment analysis, comment threads, user
generated content, emotion, negativity, community management
1. INTRODUCTION
User generated content is the essence of the social web. An
everyday issue facing technology companies and application
developers is how to incent people to actively participate. One
barrier to participation is the presence of negative, that ishostile,
angrycontent on a site; such content can drive existing
participants away and deter newcomers, but also signals for those
remaining that negative contributions are tolerated. There is,
however, an interesting tension between encouraging debate and
moderating negativity; negative content may discourage
participation, but a lack of negative content may also indicate lack
of engaged debate. Completely sanitized sites may not provide an
interesting forum and thus may not attract repeat visitors. A
central task for the community, therefore, is to be able to identify
and possibly eliminate negative content as it is produced, to model
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
Conference10, Month 12, 2010, City, State, Country.
Copyright 2010 ACM 1-58113-000-0/00/0010$10.00.
Elizabeth F. Churchill
Yahoo! Research
4301 Great America Parkway
Santa Clara, CA 95054
echu@yahoo-inc.com
and incent appropriate behaviors while disincenting disruptive
behaviors, and to identify repeat offenders [6][7][8]. As sites
become more popular, the scale of this task obviously grows,
requiring that community managers and moderators scan vast
amounts of content to locate inappropriate contributions and
behaviors. Studies of griefers and trolls (people who engage in
transgressive behaviors including posting general negative meta-
comments and targeted bully like insults as a form of sport)
suggest these behaviors are a serious problem for online sites
(e.g., see [4]). From this we believe it is worthwhile to distinguish
off-topic negative comments from on-topic negative comments
that, while negative, are offered in the spirit of debate. We also
propose that there is value in distinguishing between expression of
sadness and of anger, anger having far more potential for social
discomfort and disruption than sadness.
Recent proposals have been made to automate the detection of
inappropriate content, with varying degrees of success (e.g., [9]).
Our approach is to combine relevance analyses for detecting on-
and off-topic comments with sentiment detection methods. The
latter will help us determine the emotional valence of content, that
is, how positive/negative it is - we are focused on breaking
negativity down further to identify specifically angry, off-topic
comments, as we feel surfacing these will be the greatest benefit
in the community management process. In the next sections, we
present a description and initial test of our approach.
2. COMMENTING ON THE NEWS
2.1 Characterizing the data
Our analysis focuses on a news-story commenting site. On sites
like these, community members post stories of interest. Selected
stories tend to focus on politics, entertainment and sports. Content
popularity is determined through the votes of other readers, and
this in turn determines presentation format and duration; very
popular postings are promoted to site front pages and are visible
longer, gaining reputation points for the submitter/poster. Such
sites also allow people to make comments on posted content,
providing a space for conversation around postings. It is these
comment threads that are of interest to us; these provide new
readers with a feel for the public response to the posting, but also
to the site in general.
Our data set consists of comments from 168,095 distinct threads
(each based on a news story) from a popular news-story
recommendation and comment site. Our dataset consisted of
782,934 comments in total, an average of 4.7 comments per
thread (a standard deviation of 27.7). As seen in Figure 1, the
most common (mode) number of comments per story is 1, with a
median of 2. The comments themselves vary greatly in length (as
shown in Figure 2). The mean length is 45.7 words (with a
standard deviation of 40.5), the mode length is 8 words, and the
median length is 33 words.
Figure 1: The distribution of comments among threads in the
data set.
Figure 2: The distribution of comment lengths (in words) in
the data set.
2.2 Example data
To illustrate the kinds of comments that are posted, and also to
introduce our analysis method, below we offer an excerpt from a
longer comment thread on a story from a Philadelphia-based
online news site from October 20092. The news item relates to a
development in an ongoing public spat between Sarah Palin, the
former running mate of the defeated Republican contender in the
US Presidential Elections of 2008, John McCain, and David
Letterman, a famous talk show host. In this particular story, Palin
was reported as smiling at the revelation that David Letterman
had admitted having extra-marital affairs. The headline is shown
in Figure 3. We selected this example because it illustrates several
comment types and varying viewpoints. Following each comment
is an automated mood classification (in parenthesis) using a
system that we describe in section 3.2. :
3
C1 : At least he apologized; Palin accepted his apology so Ill follow
her lead. She was right about Letterman, which we all knew and
2
http://www.nbcphiladelphia.com/news/politics/Number-One-Reason-Sarah-Palin-
Is-Smiling-Today.html
3 ethics problems etc. David is probably a nicer person than Sarah &
We have replaced user IDs with C for commenter, with the number denoting the
specific commenter. We have used three dots to denote where additional
comments were made, that we felt were not useful for our discussion. These were
usually off-topic exclamations or spam.
Figure 3: Story posted on a community comment
news site. The story focuses on Sarah Palins
response to David Lettermans public confession
about having affairs. This story generated many
emotional comments, with varying degrees of
relevance to the story.
hopefully America realizes she has been right about OH so much
more. (sad)
C2: Sarah Palin : Told ya so! GO SARAH ! (happy)
C3: You go, girl, were behind you 100%. Yes, thats right, blue
dogs are Palin Democrats! (happy)
C4: The number 1 reason Sarah is smiling today, is because she
just got a fat check for $7,000,000 bucks. Who wouldnt be smiling?
Notice too, its listed under the title, ENTERTAINMENT. Thats the
most truthful thing about this story. Obviously, she coughed up a
substantial amount of the millions to get Todd to keep quiet about
their impending divorce, by buying him a new boat and sending off
into the Bering Sea to do some fishing. What he doesn't know
yet is, the hull of the boat is made of bamboo and paper machete.
(angry)
C5: The family that quits together splits apart. (sad)
C6: Man Palin is a has been stale news, good for small town hicks
selling news to those like them. To them a pig farting is big news.
(happy)
C7: STOP OFFENDING MY FELLOW AMERICANS. (angry)
C8: Say what you will. Palin rooked her naysayers. You can call
her book whatever you can to dilute her success but, WHOS
LAUGHING NOW! The Clowns and Comics are presently in
conference crying and consoling each other with their big towels
wiping off their makeup and drying their big old shoes.
Waaaaahhhhhh! (happy)
C9: I think the real reason Sarah is smiling is that so many people
want to read her book! It is not only the money (everyone wants
paid for their work) but the fact that Sarah can get the truth out to
the Americans! Sarah does not like corruption or deceit! I want to
know the truth too! (angry)
C10: The Palin family sucks, always have and always will. There
are a few sex scandals in Sarah & Todds past. Not to mention
Todd put together. She can smile at Daves misfortune while
conveniently forgetting her own sins. (sad)
The thread begins with three relatively positive comments in
support of Sarah Palin, though the first is sympathetic to David
Letterman. Peppered throughout the thread are various angry
comments including comments both in support of Palin (e.g. C9)
and angry conversational comments (e.g. C7). Though there is
much variety in the sentiment conveyed in the posts, they clearly
start out rather positive and become more negative over time. This
is a pattern we observed across the entire dataset, as will be
discussed below. To set the groundwork for these results, in the
next section, we will outline in more detail our theoretical
framework for the automated analysis through which we aim to
surface relevance and sentiment, a combination of which will be
used to inform community managers as to inappropriate content in
the end system.
3. THORETICAL FRAMEWORK AND
METHOD
3.1 Determining Relevance
To determine whether or not each comment is on-topic (relevant
to the focal story), we employ standard information retrieval
techniques including tfidf [14]. While it was originally created for
early search engines as a method to evaluate the relevance of a
document to a query, others have recently used this technique as a
way to form a content query from a document to be passed of to a
search engine to retrieve similar documents [3]. In this way, it
can be viewed as a document similarity metric.
The tfidf document representation technique treats each document
as a bag of words, where words have varying degrees of
importance in the document. The importance of a word is boosted
by the frequency with which it occurs in the document (term
frequency, TF). The importance of a word is inversely
proportional to the frequency with which it occurs in documents
across a corpus (document frequency, DF). In our preliminary
system, a corpus of news stories from Reuters was used to
compute document frequency values in our system [13].
Our system creates a tfidf representation of the focal story. It then
treats each comment as a query and measures the relevance of the
focal story to that query (comment). The relevance score is
simply the sum of the tfidf values for each of the query terms in
the focal story. If the relevance score is above a similarity
threshold, the comment is considered to be on topic. The real-
valued outputs of such comparisons allow us to evaluate
comments by degrees of similarity to the original focal story, in
addition to simply on-topic or off-topic.
3.2 Sentiment Analysis
Since 2002, many researchers have focused their efforts on the
task of automatically analyzing the sentiment of a document (how
positive/negative it is) as expressed by its author by analyzing
words that are used in the text. For example, words such as
desirable, great, better are associated with positive However, there are a number of more complex models of emotion
sentiments, and words such as awful, terribly, dangerous or sentiment that have been used when classifying human
have been associated with negative sentiments.
Sentiment or opinion detection applies to any database or stream
of textual information, especially user-generated content, such as
blogs, web pages, bulletin boards, emails, chat rooms, and so on.
The process for detecting attitudes and sentiments requires three
basic steps:
1. Collecting a training corpus of texts, often human annotated as
to their sentiment
2. Building a set of patterns associated with positive, negative and
neutral sentiments.
3. Training a statistical machine learning system possibly
augmented with a suite of interpretation rules that can classify
new texts into the desired categories.
Much of this work began with, and is still utilizing movie and
product reviews as labeled training data (where the associated star
rating for a review serves as the label). Many have used this data
in supervised machine learning systems [12] while others have
taken an unsupervised approach to building such systems,
leveraging relationships between the words in the target document
in order to calculate an overall sentiment score [17].
Given that such systems exist and are quite accurate, some as high
as 90% [1], it seems as though utilizing these systems on our data
(user comments on news stories) is a trivial task. However, it is
well known that sentiment analysis is a domain specific problem.
That is, words used in a positive context to describe cars are not
necessarily the same words used in a positive way to describe
movies. In fact, many words have opposing emotional
connotations across domains. For example, a cold beverage is
good while a cold politician is bad [10]. Further, a text may say
that a policy is not at all desirable (negative sentiment), or a
product is terribly good (positive sentiment); detecting the
underlying sentiment behind these kinds of statements is hard if
simple lexicons are used. This implies that, in order to build an
accurate sentiment analysis system, you must have labeled
training data from within the same domain. As sufficient data sets
do not exist in all domains, many have made efforts toward
building systems to customize sentiment analysis systems to new
domains without a large amount of labeled training data in the
target domain [1][15].
The system that embodied our past approach to adapting to new
domains without labeled training data was called Reasoning
Through Search. This system characterized a target document by
a sentiment query containing the words in the document with
the highest sentiment magnitude (extracted from a statistical
model). This query was then posted to a case base of labeled
movie and product reviews to retrieve (emotionally) related
documents. The labels of the retrieved documents are then
combined with knowledge of the potential domain of the target
document in order to compute an overall valence intensity score
a score ranging from -2 to +2 where -2 is most negative and +2 is
most positive [15].
Most work in sentiment analysis has focused on building systems
that simply indicate whether a document is positive, negative or
neutral. Some have built systems to judge valence on multipoint
scales [11], but little work has moved beyond the dimension of
valence (a measure of the authors sentiment toward a topic how
positive/negative they are).
behavior. The simplest of these is the VAD or PAD Model, which
characterize emotion on three dimensions representative of
valence (or pleasure), intensity and dominance [2]. The most
famous is Ekmans six emotion model, which lays out the
following as the 6 basic emotions that human beings experience:
happiness, sadness, anger, surprise, disgust, fear [4].
 Happy Sad Angry
Energetic Confused Aggravated
Bouncy Crappy Angry
Happy Crushed Bitchy
Hyper Depressed Enraged
Cheerful Distressed Infuriated
Ecstatic Envious Irate
Excited Gloomy Pissed off
Jubilant Guilty
Giddy Intimidated
Giggly Jealous
Lonely
Rejected
Sad
Scared
Table 1: Three consistent 'mood clusters' that emerged from
k-means clustering (Sood, et al 2009).
Our past efforts have included moving beyond the valence
dimension and classifying documents based on the general mood
of the author, using a dataset of blog posts (labeled with the
authors stated mood) from LiveJournal as training data [16].
While the LiveJournal dataset was ideal as blog posts were
labeled with the user stated mood, it was less than ideal in that the
author could select from one of 130 possible moods, or write in
their own. Such a selection of classes was not only too large for a
supervised learning system, but it was not evident that there was a
statistical distinction in the use of the mood labels excited and
happy. Additionally, some authors may choose a small number
of moods to rotate between, making the dataset skewed by author
bias. In an attempt to make this dataset more meaningful, we
applied k-means clustering, treating the mood labels themselves
as data points where the feature vector for that data point
summarized all of the posts labeled with that mood. Through
several clusterings, we found that mood clusters representative
of happy, angry, and sad emotions emerged consistently (see
Table 1 for the contents of these clusters). While clearly not
aligning to emotions as characterized by researchers, these were
expressions of emotions as people use them. The original goal of
our system was to classify mood as one of Ekmans six
emotions [4], however, the data from the happy, angry and sad
mood clusters were best utilized in our final supervised learning
system [16]. It is noted that requiring that a document fall into one
of these classes (happy, sad or angry) is limiting (e.g. - one could
be slightly angry and very sad at the same time), however,
approaching this as a classification task greatly reduces its
complexity.
Given our data set and task at hand, a few challenges arise. First,
we have a large dataset of user comments and their associated
new stories, but no sentiment labels for this data. Simply using
systems trained on other domains will lead to some inaccuracies.
Second, the comments themselves are rather short and often
conversational. This dataset is quite different from the self-
contained longer documents given in movie and product reviews.
This creates two interesting sentiment analysis problems. Our
preliminary work, presented here, towards sentiment analysis with
this dataset has relied on our past efforts (described above)
systems that classify text by sentiment (the Reasoning Through
Search system) and also by mood (happy, sad, or angry). Future
work will utilize our vast (unlabeled) data set in training a domain
focused sentiment analysis system.
4. RESULTS
4.1 On relevance
Table 2 shows several comments corresponding to the news story
shown in Figure 3. Although there were many comments in the
thread for this story, we report a sample here to illustrate our
technique. Overall the comments on the story skewed angry,
which was one reason why we have selected it for illustrative
purposes in this paper. In Table 2, we present a relevance score
conveying how similar a comment was to the news story itself;
that is, how on-topic the comment was. As noted above, the story,
titled Number One Reason Sarah Palin Is Smiling Today, is
about Lettermans admission of infidelity, reporting multiple
affairs some of which were with colleagues. Table 2 illustrates
that the tfidf relevance scores clearly reveal which comments are
on-topic; the top three comments in the table show relatively high
tfidf scores while low scores are registered for posts that are
clearly off-topic or are spam.
Comment TFIDF
Relevance
Score
The Palin family sucks, always have and always will. There 5.239
are a few sex scandals in Sarah & Todds past. Not to
mention ethics problems etc. David is probably a nicer
person than Sarah & Todd put together. She can smile at
Daves misfortune while conveniently forgetting her own
sins.
Say what you will. Palin rooked her naysayers. You can 3.754
call her book whatever you can to dilute her success but,
WHOS LAUGHING NOW! The Clowns and Comics are
presently in conference crying and consoling each other
with their big towels wiping off their makeup and drying their
big old shoes. Waaaaahhhhhh!
At least he apologized; Palin accepted his apology so Ill 3.717
follow her lead. She was right about Letterman, which we
all knew--and hopefully America realizes she has been right
about OH so much more.
ClassyMingle.com is the best and largest online personals 0.359
site dedicated to men and women seeking a higher caliber
online dating experience.
Whyyyyyyyyyyyyyyy so many people are interested in an 0.696
ageless relationship. young girls want to have fun with 40+
man and young guys want to have fun with 40+ women.
There are many sites focusing on this kind of relationships
such as http://www.Seekingsugar.com !
Who cares!!! My boyfriend thinks the same with me. He is 0.351
eight years older than me, lol. We met online at
Agelessmatch.com a nice and free place for Younger
Women and Older Men, or Older Women and Younger Men,
to interact with each other. Maybe you wanna check out or
tell your friends.
Table 2: Comments and their tfidf relevance scores. High
numbers are more relevant; low numbers less relevant. Very
low relevance often indicates spam.
While these tfidf scores are very helpful in automatically detecting
whether a post is or is not on topic, we are also interested in
detecting the emotional or affective trajectory of a comment. This
is useful when used in combination with the tfidf measures of
relevance because it allows us to determine whether a post is
somewhat off-topic (median tfidf score), yet of a positive
emotional type, which may indicate a conversational comment. As
noted in related work, phatic statements and conversational
comments are often the glue that moves a web site from being
informational to being social [8]. Ideally in a community news
comment site, we would want both of these elements.
4.2 On Sentiment
As noted in Section 3, where we outline our methods, our
approach is to break sentiment down into positive and negative
valence and additionally into broad categories: happy, sad and
angry. In Figure 4, we show how comments that are illustrated in
Table 2 fare when being classified by the mood classifier.
Figure 4 Comments classified by the overall mood of the
author. Highlighted features are those that contributed the
greatest to the overall classification (green represents happy,
blue represents sad and red represents angry).
In addition to classifying comments by mood, we also used the
Reasoning Through Search system to characterize each
comments valence and intensity (on a scale from -2 to +2). Over
the entire corpus of comments, we found that 54.5% of the
comments were negative, 2.7% were neutral, and 42.8% were
positive. The average valence of comments in the corpus was
-0.211. This automated analysis indicates that the comments tend
slightly negative overall.
An interesting trend emerged as we examined valence/intensity
changes within individual threads. We found that the most
commonly occurring trend (occurring in 54% of the threads) was
for a thread to start out positive and end negative (as seen in an
example thread in Figure 5), while only 43% made a negative to
positive change and 3% stayed roughly the same throughout the
course of the thread. This emphasizes our need to detect
negativity, as it seems that these comments bring the discussion
thread to an end, perhaps creating a boundary to participation.
As these examples illustrate, the sentiment classification
algorithms fared well in classifying the comments. We conducted
a sample test to compare our own performance as human
classifiers with that of the classification algorithms. We took 20
comments selected at random from our whole dataset of 168,095
threads and 782,934 comments, and conducted our own manual
classification of these comments. For each comment, we (each of
the authors) answered two questions 1) Is this comment positive
or negative? 2) Is this comment happy, sad or angry? We worked
alone and then compared our classifications with that of the two
automated sentiment classifiers (the RTS valence and mood
classification systems).
For question #1, our answers agreed for 95% of the documents
(only disagreeing on 1 comment out of 20). When compared to
the RTS classified valence of each comment, each of us agreed
with the system output for 75% of the comments. For question
#2, our answers agreed for 85% of the comments. When
compared to the mood classification of each comment, one author
agreed with the system output for 80% of the comments and the
other author agreed 65% of the time. These results are
encouraging as the systems (both trained on data from other
domains), performed well on this dataset where the random
baseline performance would be 50% for question #1 and 33% for
question #2. High agreement scores between the manual
classifications were encouraging. There is room for improvement
in this task, perhaps through more domain specific approaches.
Figure 5: Graphing comments in a thread by valence:
Conversations in general start positive and end negative. This
is true of the comments on the story in this illustration.
5. DISCUSSION AND FUTURE WORK
In this paper we have outlined our approach surfacing negative
comments in online comment forums. We were impressed with
the relative success of the relevance plus sentiment detection
system.
Our approach reveals not just negative and positive valence, but
offers a richer palette of affect, classifying comments as being
happy, sad or angry. We believe that introducing this additional
nuance offers greater insight into both the overall sentiment felt
by an aggregate group but also can help us identify inappropriate
affect levels (by establishing a level of anger scale). We
conjecture that a post which is off-topic and very angry is a
clearly salient for further investigation by community and site
managers. Once conversational and trial posts (e.g., hi) are
taken into account, we find also that highly off-topic neutral posts
are often spam. Our technique will also allow us to identify people
who are consistently posters of angry content across different
stories.
Given that our mood classification system was trained on
LiveJournal blog posts, it is not surprising that there were some
detection errors. In future work we will train our sentiment
classifier with a larger dataset of comment threads taken from the
site we are studying, utilizing this unlabeled data in an
Expectation Maximization approach. This will also address the
challenge of how well a sentiment detection system can perform
when trained on short snippets or comments rather than blocks of
prose.
We have also begun a hand-coded classification of comments
according to an expanded notion of relevance: on-topic relevance
is where a comment relates to the original text, whereas
conversational relevance denotes when a comment refers to a
previous posted comment by another community member. In the
latter case, we have found through our analyses that
conversational relevance can be statements that are directed at
something someone said, or at the person themselves. Negative,
off-topic, conversational relevance, where there is no relevance to
the original text, is usually an insult directed at a personthe
content refers to a community member themselves (e.g., by name,
using You are) and is combined with known insulting terms.
We are also interested in whether the emotional valence of the
original text is correlated with the emotional valence of the overall
comment thread. We are validating our model by hand before
investigating what kinds of linguistic disambiguation will need to
be used in combination with our existing sentiment detection
model.
In addition, we are intending to use our method to model
emotional trajectories through comment threads. Specifically we
wish to address whether negative comments (crossed by whether
they are on or off topic) have an impact on comments that follow:
does negativity beget more negativity? Do conversations go
south? And if they do, what are the characteristics of a
conversation that escalates versus one that does not? Through this
means we intend to address the issue of whether undesirable
behavior does or does not model undesirable behavior in others,
and if so what are effective, in thread, remediation strategies
which may be better than simple deletion. We believe these local
strategies over content are the way to effective community
management. With tools that help surface where non socio-
normative behaviors are occurring we can support human
community managers work more effectively by automatically
finding and filtering potential violations.
There are of course also open questions, especially when dealing
with people who are regular community visitors. For example, an
open question is about how a single person behaves over time:
that is, what is the history of an authors sentiment about a topic
over time? What is their authority in the social group, and from
that what is their influence? Clearly some peoples negative
comments may have more weight than others. Do we see the
emergence of groups who all share the same sentiment on certain
topics? These questions represent a valuable, but fine-grained and
socially oriented research program. We believe our approach, to
combine relevance, affect/sentiment and descriptions of posting
patterns is a good starting point.
6. ACKNOWLEDGMENTS
We thank our colleagues at Yahoo! for their help with these
analyses.
7. REFERENCES
[1] Aue, A. and M. Gamon. Customizing sentiment classifiers to
new domains: a case study. In Proceedings of RANLP, 2005.
[2] Bradley, M.M. and Lang, P.J. Affective norms for English
words (ANEW): Stimuli, instruction manual and affective
ratings. Technical Report C-1, Center for Research in
Psychophysiology, University of Florida, Gainesville,
Florida, 1999.
[3] Budzik, J, Hammond, K.J., and Birnbaum, L. Information
access in context. Knowledge-Based Systems. 14(1-2): 37-53
(2001)
[4] Chesney, T,. Coyne, I., Logan, B., and Madden. N. Griefing
in Virtual Worlds: Causes, Casualties and Coping Strategies,
Information Systems Journal, 19, 6, 525-548, 2009
[5] Ekman, P. Emotions Revealed: Recognizing Faces and
Feelings to Improve Communication and Emotional Life,
Henry Holt and Company, New York, NY: 2003.
[6] Lampe, C. and Resnick, P. (2004) Slash(dot) and Burn:
Distributed Moderation in a Large Online Conversation
Space. In Proc. CHI 2004, pp 543-550
[7] Lampe, C. and Johnson, E. (2005) Follow the (Slash)dot:
Effects of Feedback on New Members in an Online
Community. In Proc Group05, pp 11-20.
[8] Gazan, R. When Online Communities Become Self-Aware. In
Proc HICSS09 (HICSS-42), pp 1-10, ACM Press.
[9] Lou, J.K., Chen. K.T. and Lei, C.L. A collusion resistant
automation scheme for social moderation systems. In Proc.
IEEE Conference on Consumer Communications and
Networking, p 571-575.
[10] Owsley, S, Sood, S and Hammond, K.J.. Domain specific
affective classification of documents. In Proceedings of the
AAAI Spring Symposium on Computational Analysis of
Weblogs., pages 181-183, 2006.
[11] Pang, B. and Lee, L.. Seeing stars: exploiting class
relationships for sentiment categorization with respect to
rating scales. In Proceedings of ACL, pages 115-124, 2005.
[12] Pang, B., Lee, L. and Vaithyanathan, S.. Thumbs up?
sentiment classification using machine learning techniques.
In Proceedings of EMNLP, pages 79-86, 2002.
[13] Reuters-21578 text categorization test collection
[14] Salton, G. and Buckley, C. (1988). Term-weighting
approaches in automatic text retrieval. Information
Processing and Management. 24(5): 513 to 523.
[15] Sood, S., Owsley, S. Hammond, K.J. and Birnbaum, L..
Reasoning Through Search: A Novel Approach to Sentiment
Classification. Northwestern University Tech Report
Number NWU-EECS-07-05, 2007.
[16] Sood, S..O. and Vasserman, L.. ESSE: Exploring Mood on
the Web. International Conference on Weblogs and Social
Media Data, 2009.
[17] Turney, P.D. Thumbs up or thumbs down? Semantic
orientation applied to unsupervised classification of reviews.
In ACL, pages 417{424, 2002.
[18] Wans, N., El-Saban, M., Ashour, H. and Ammar W. (2008).
Automatic Scoring of Online Discussion Posts. In
WICOW08, pp 19-25. ACM Press.
[19] Weimer, M., Gurevych, I. and Muhlhauser, M. (2007)
Automatically Assessing the Post Quality in Online
Discussions of Software. In Proc. ACL 2007, Demos and
Posters, pp 125-128. Association for Computational
Linguistics.
[20] Wensel, A. and Sood, S.O. VIBES: Visualizing Changing
Emotional States in Personal Stories. ACM MultiMedia
Workshop on Story Representation, Mechanism and Context,
2008.