Abstract
This study offers a systematic comparison of automated content analysis tools. The ability of different lexicons to correctly identify affective tone (e.g., positive vs. negative) is assessed in different social media environments. Our comparisons examine the reliability and validity of publicly available, off-the-shelf classifiers. We use datasets from a range of online sources that vary in the diversity and formality of the language used, and we apply different classifiers to extract information about the affective tone in these datasets. We first measure agreement (reliability test) and then compare their classifications with the benchmark of human coding (validity test). Our analyses show that validity and reliability vary with the formality and diversity of the text; we also show that ready-to-use methods leave much space for improvement when analyzing domain-specific content and that a machine-learning approach offers more accurate predictions across communication domains.
Keywords
Social media has radically transformed the information environment that we inhabit. We constantly feed information to online networks that, as a result, pulsate restlessly with news and public reactions to trending topics. On the aggregate, social media allows opinions to take shape as events evolve, generating near to real-time indicators of salient issues and how they are framed by the public; because of this, online communication offers the potential to advance our theoretical understanding of public opinion by capturing the trends and dynamics that are often missed in opinion polls and surveys. To realize this potential, however, automated content analysis techniques are necessary to parse large amounts of data in an efficient and effective manner, offering a timely and precise account of what communication through social media really reflects.
Opinion mining, also known as sentiment analysis, deals with the computational treatment of expressions of opinion, sentiment, emotion, beliefs, and speculation in written text (Liu 2012; Pang and Lee 2008). A subdiscipline within the fields of natural language processing (NLP) and computational linguistics, sentiment analysis has become a popular tool in social media research to automatically detect, extract, and analyze affective content within written communication. There are two main approaches for automated sentiment classification (Liu 2012): unsupervised learning, in particular lexicon-based methods that use a predefined dictionary of words and phrases to classify text as positive, neutral, or negative; and supervised learning, which requires training the classifier with corpuses of text that have known responses to it (e.g., online movie reviews and their associated rankings) to build predictive models for unlabeled text (e.g., reviews without a ranking). Supervised learning is more costly to implement than employing off-the-shelf dictionaries, but it offers a tailored characterization of domain-specific text. Here we focus on unsupervised methods, very popular for their simplicity, to compare their performance with a supervised approach and with human classification.
Prior Work and Definitions
The lexicon-based approach is a commonly used method when conducting automated content analysis of online communication (Bollen, Pepe, and Mao 2011; Dodds and Danforth 2009; Dodds et al. 2011; Golder and Macy 2011; González-Bailón, Banchs, and Kaltenbrunner 2012; Thelwall, Buckley, and Paltoglou 2011; Young and Soroka 2012). The popularity of annotated dictionaries results from their ability to offer a straightforward method to classify documents: they simply need a word frequency count and some logic to weight and aggregate the sentiment scores of the words that are included in the dictionary. However, prior studies offer no systematic comparison of different lexicons and an assessment of the conditions under which they perform best. This article aims to offer that comparison with two goals: first, to provide guidelines to choose the most appropriate method given differences in data sources, especially in online, user-generated content; and second, to compare the performance of these unsupervised methods to the more sophisticated approach of supervised learning.
We argue that two characteristics of the text under analysis are of particular importance to assess the performance of automated content tools. The first is the formality of the language used; the second is the diversity of the information being communicated. These two characteristics allow us to differentiate corpuses in line with their linguistic properties—which, prior research suggests, determine the success of different automated methods to yield accurate classifications. Research in NLP has consolidated a range of numerical statistics that reflect the importance of certain words in a document or corpus; most of these statistics are related to the frequency of terms and their relative weight (Manning, Raghavan, and Schütze 2008; Salton and Buckley 1988; Sparck Jones 1972). Inverse document frequency, for instance, captures the extent to which there are rare terms in a document, that is, terms that are less frequently used compared to other words in the same corpus. In addition to reducing the importance of terms that are too common and thus not very helpful to characterize the text under analysis, this statistic can also help to describe the register of the language used.
Prior research has also tried to assess topic diversity in texts deriving probabilities from word frequencies. A prominent technique in topic modeling is the Latent Dirichlet Allocation or LDA (Blei, Ng, and Jordan 2003), which applies probabilistic methods to discover themes underlying the structure of written expression. This approach has been used to find trending topics in scientific publications (Griffiths and Steyvers 2004) and also newspaper coverage of government policies (DiMaggio, Nag, and Blei 2013). More specifically for the purposes of this study, topic modeling has been applied to identify information diversity in text documents (Bache, Newman, and Smyth 2013). In the context of social media, measuring diversity can help to characterize the type of communication these technologies afford or encourage: the language used to post comments on a news website such as the BBC, for instance, is likely to differ from the language used to communicate in micro-blogging sites such as Twitter; the range of topics is likely to vary as well.
We aim to characterize linguistic heterogeneity using measures of formality and diversity that are comparable across platforms. Even though we focus on one specific set of techniques that extract opinions in the form of sentiment scores, we also make use of the relative frequency of words and the distribution of topics to characterize the text that we analyze. Hence, we draw a comparative dimension that can help us to assess the different performance of sentiment dictionaries. This line of work, like any specialized literature, operates on the basis of technical jargon. There are many excellent introductions to machine learning and opinion mining (Liu 2012; Manning, Raghavan, and Schütze 2008); for the purposes of this article, a few basic definitions will suffice. The analysis of text requires processing, cleaning, and standardizing language prior to its analysis. One of the steps involved is tokenization, which refers to the process of breaking a text document into meaningful units that are usually words. Stemming is another step; it involves reducing inflected or derived words to their stem or root form. Once these preprocessing steps are done, word frequencies and their relative probability in a document can be derived. The following section gives more details on the methods we use to analyze our data.
Data and Methods
We use five datasets (see Table 1) that track communication in a range of online platforms: a news discussion forum maintained by the BBC; the micro-blogging site Twitter; the news aggregator Digg; the video-sharing website YouTube; and a sample of politically oriented blogs. We chose these datasets because they all track user-generated content but vary significantly in their subject domain. The BBC news forum, for instance, includes discussions around salient news in the UK, and it was actively moderated by BBC staff. The Twitter sample, on the other hand, contains communication that was not moderated and encompasses a wider range of public interests; random messages were retrieved from public streams using the platform’s API. Digg discussions are also wide-ranging but focus on news and current affairs, and users self-moderate many discussions. The YouTube comments are linked to the contents of videos, although they are not necessarily related to that visual content. Finally, the blog data derive from a study that focused on online debates around health care, so the content is likely to be more specialized and restricted than in the other datasets.
Descriptive Statistics
The text for each dataset was tokenized (i.e., words were extracted) using a standard library (McCallum 2002). The number of unique terms (or tokens) gives us a first approximation of the size of the vocabulary employed in each context and the richness of the language used. According to this measure, the Twitter dataset contains the more heterogeneous language, followed by YouTube. At the same time, the average number of unique terms by document (i.e., posts or comments) indicates that the BBC forum and political blogs are the most elaborate in terms of communication length (see Table 1). Table 2 shows examples of the type of messages or posts contained in each of these datasets.
Examples of Text Analyzed within Each Dataset
We summarize differences in the language used across these datasets using two features: formality and diversity. We measure formality as the percentage of rare terms that appear less than two times in the collection of documents; as explained in the previous section, using word frequency to characterize text is a standard heuristic in NLP. Our logic is that rare terms are likely to be associated with colloquial language; a higher percentage would be indicative of a more informal type of communication, as the examples in Table 2 for Twitter and YouTube show. We measure diversity, on the other hand, using the topic modeler approach (Bache, Newman, and Smyth 2013). First, we extracted the count-matrix representation for each dataset with D documents, where each row represents a document d (1 ≤ d ≤ D) and each column j represents a topic category (1 ≤ j ≤ T). Each cell (d, j) in this matrix indicates how many elements in document d (i.e., word tokens) belong to category j (i.e., topics). Then, we identified the topics using the LDA method with Gibbs sampling (Blei, Ng, and Jordan 2003; DiMaggio, Nag, and Blei 2013). This method requires specifying in advance the number of topics to be identified; we set this parameter as T = 100, which in the past has been shown to generally produce coherent topics (Hong and Davison 2010; Rosen-Zvi et al. 2004). We expect broadly similar results with other values of
where
and
Using these measures of formality and content diversity, we ranked the datasets in decreasing order as summarized in Figure 1. According to this ordering, the two most formal datasets (blogs and the BBC forum) are the least diverse; that is, they contain more specialized communication. Overall, these two measures exhibit a negative association: when formality is higher, diversity is lower.
Linguistic Differences across Datasets
To classify the documents contained in these datasets (i.e., posts and comments) as positive or negative, we used several annotated lexicons. In particular, we used five different dictionaries: LexiconBased, SentiStrength, ANEW, labMT, and Lexicoder (see Table 3). All these dictionaries operate on the basis of lists of words that are labeled as positive or negative.
List of Lexicon-Based Classifiers Used in the Analyses
The first dictionary, LexiconBased, is an extension of the dictionary included in the linguistic inquiry and word count (LIWC) software (Pennebaker, Booth, and Francis 2001). The extension incorporates modules for negation detection and a set of other linguistic and stylistics modifiers that capture the intensity of emotions, for instance, capitalization or emoticons. This popular lexicon has been used in many studies, including the analysis of mood trends in large-scale communication data collected from Twitter (Golder and Macy 2011). The second list, SentiStrength, contains nearly three times the amount of words; it draws from LIWC (Pennebaker, Booth, and Francis 2001) and the general inquirer (GI) lexicon (Stone et al. 1966), and it also contains modules to account for negation, idioms, emoticons, and so on. This lexicon was explicitly designed to analyze social media data (Thelwall et al. 2010). These two lists apply stemming, which means that all inflected words are mapped to their common root and receive the same score.
The Affective Norms for English Language Words dictionary (ANEW) was manually labeled by human subjects as part of a psychological study (Bradley and Lang 1999). This list contains ratings of words on a 9-point scale in three dimensions: valence, arousal, and dominance. The valence dimension has been used to assess levels of happiness in large-scale written expression (Dodds and Danforth 2009) and in connection with long-term shifts in public opinion (González-Bailón, Banchs, and Kaltenbrunner 2012). The labMT dictionary is a tenfold extension of the ANEW list (Dodds et al. 2011). Valence scores were collected using the crowdsourcing platform Mechanical Turk (Kittur, Chi, and Suh 2008), where each word obtained fifty different evaluations. Finally, the Lexicoder Sentiment Dictionary (Young and Soroka 2012) was developed for the automatic coding of political text drawn from news articles covering a range of policy issues. Two of the five dictionaries (ANEW and labMT) do not apply stemming—whether this has implications for their accurate classification of text is one of the aspects we consider.
We align these dictionaries by reducing them all to a ternary classification, with categories for negative, positive, and neutral content. For the SentiStrength, LexiconBased, and Lexicoder methods, documents are classified as positive if they have a majority of positive words; negative if they have a majority of negative words; and neutral if they have an equal number of positive and negative words. For ANEW and labMT, the conversion is as follows: documents with a score of 4 or lower are classified as “negative”; documents with a score of 6 or higher are classified as “positive”; and documents with a score between 4 and 6 are classified as “neutral.”
We compare the five classifiers using two criteria. The first assesses the extent to which there is agreement in document classification for each of the five datasets. We use the Spearman’s rho correlation coefficient to determine to what extent the five different dictionaries yield similar results when classifying documents. We also calculate the Krippendorff’s alpha for each of the five datasets; this statistic is commonly used to assess agreement and intercoder reliability in traditional content analysis performed by trained coders (Krippendorff 2013). Applied to our data, the calculation of this statistic assumes that each dictionary can be treated as an independent rater. Since this measure of reliability was developed to measure agreement between human coders, the usual rule of thumb when assessing its strength (usually α > .8) does not necessarily apply when assessing automated methods; however, it still offers an interesting benchmark for comparison.
The second test assesses the validity of automated classification by comparing it with human classification. Each of the datasets summarized in Table 1 has associated classifications performed by human coders. Five annotators coded independently at least 1,000 posts, assessing the affective content of the text through a web interface designed specifically for this purpose. The instructions for the coding read: “Please read the text in the box below and rate it in terms of negative and positive sentiments using the scales below. You should enter a rating for both positive and negative sentiments. Possible values range from 1 (no emotion expressed) to 5 (strong emotion). If you would like to see examples of data that have already been classified, click here.” A randomly selected message was then shown, followed by the 5-point scale for positive and negative emotions. This web-based annotation application, including a discussion of the resulting classification, is described in Paltoglou, Thelwall, and Buckely (2010). We used this human-generated classification to assess the performance of the five lexicons, which we measure in terms of agreement—that is, as the percentage of posts that were classified in the same way by human coders and lexicons. In addition, we also compared the performance of the five dictionaries with a supervised algorithm trained with the data and, therefore, better attuned to the specificities of each corpus.
Findings
Reliability
Figure 2 summarizes the extent to which the five dictionaries agreed in their classification of positive, negative, and neutral content. The panels show the correlation matrices for each dataset. The highest correlation coefficients do not go beyond .60, which means that overall the five methods yield very different classifications of the same documents: the assessment of the tone of messages varies drastically depending on which lexicon is employed to parse and analyze the text. These high levels of disagreement are confirmed by the Krippendorff’s alpha, which varies between .10 and .21, signaling a much lower agreement than would be acceptable for human coders.
Level of Agreement of Lexicons in their Classification of Content across Social Media Platforms
The LexiconBased, SentiStrength, and Lexicoder dictionaries provide the most consistent classifications, even though the number of words they contain differs significantly. Most surprisingly, the similarity levels between ANEW and its tenfold extension labMT are strikingly low. The reason lies in how the labMT dictionary was constructed: first, four different large-scale text sources—Twitter, Google Books, music lyrics, and the New York Times—were parsed to select the 5,000 most frequent words in each corpus; once merged, these ranks resulted in the final list of 10,222 words, which spans multiple languages, verb conjugations, slang, and stop-words like “the” and “of” (Dodds et al. 2011). The crucial aspect of this dictionary is that all these stop-words obtained a valence score through fifty independent mechanical turk evaluations; overall they averaged around five, which the authors interpret as a sign that the method is sensible (Dodds et al. 2011, 5). However, our results suggest that the presence of stop-words in the lexicon dampen the effect of actual affective words, giving an overall evaluation that is misleadingly objective or neutral. For instance, the expression “I am so happy” would receive a score of 8.21 according to the ANEW dictionary (which only has an annotation for the word “happy”); but it would receive a score of 6.17 according to labMT, which has annotations for “I” (5.92), “am” (5.38), “so” (5.08), and “happy” (8.3). In terms of capturing affective content, only “happy” is arguably relevant, but the labMT does not discriminate syntactic meaning and weights all words equally.
Validity
The previous section assessed agreement across the four dictionaries; here we consider their agreement with human coders, which we use as our benchmark for validity (the equivalent of accuracy tests in NLP). For the BBC, Twitter, Digg, and YouTube datasets, human coders classified the documents as “positive,” “negative,” and “neutral”; for the blogs dataset, the coders used only binary categories (positive, negative). To assess the significance of the lexicons’ performance, we estimate random accuracy as the ratio 1/C where C is the number of available classes (e.g., 1/3 or 33 percent for ternary classification; 1/2 or 50 percent for binary classification). This is the accuracy level we would obtain if we selected classes randomly, so the larger the departure from this random baseline, the better the performance.
In addition, we also employed a machine-learning algorithm that applies an estimation method comparable to logistic regression to infer the probability that a document belongs to a category (positive, negative, or neutral) if it contains a certain word count distribution (Nigam, Lafferty, and McCallum 1999). We trained this algorithm using a tenfold cross-validation, which involves partitioning the datasets into ten subsamples that are then iteratively used for training and validation. Figure 3 summarizes the accuracy percentages of this supervised learning approach (orange, darker bars) alongside the five lexicon-based methods (green, lighter bars).
Lexicons’ Accuracy in Document Classification Compared to Machine-Learning Approach
Classification accuracies vary substantially across datasets: the worst overall performance is associated with the most formal, less diverse corpuses (BBC and blogs); in fact, none of the dictionaries perform better than random for blogs. This is not surprising given that this dataset tracks health care discussions, which are more specialized and exhibit characteristic features that are unlikely to be captured by general-purpose lexicons. The supervised classifier trained with subsets of the data gives an overall better performance because it is more aligned with the specificities of the language used in a given communication domain. However, its superiority compared to three of the dictionaries (LexiconBased, Lexicoder and, mostly, SentiStrength) is small for the less formal, more diverse datasets: YouTube, Twitter, and Digg. Again, the labMT classifier is the worst in terms of validity, despite its impressive ten-thousand-word lexicon: the spurious relevance it gives to common words that have no real affective load results in largely misleading classifications. According to our analysis, this dictionary is (slightly) better than random for only two of the most informal types of communication.
Discussion
Social media opens new windows to public opinion and the way in which we react to news, debates, or policies. However, analyzing the ever-increasing, always-fluctuating flow of online talk requires scalable methods that can parse those contents effectively to extract an informative signal. Lexicon-based methods for sentiment classification have become prevalent because of their straightforward application, but their comparative performance across domains or vis-à-vis other methods usually goes untested. Here we have examined five publicly available dictionaries to assess their reliability (agreement) and validity (accuracy) compared to both human annotations and more sophisticated, supervised machine-learning methods.
Our findings suggest that the reliability and validity of different methods are a function of the formality and diversity of the text analyzed. Unsupervised (lexicon-based) methods perform comparably to a supervised, data-trained approach when the text under analysis is diverse and informal; their performance is poorer for less diverse, domain-specific content. We also showed that agreement across the four dictionaries considered is generally low. Disagreement derives from two main factors: the scales used to quantify the emotional score of words, and the list of words with annotations. We have shown that some dictionaries are not well suited for ternary classification (i.e., positive, negative, and neutral). In particular, the labMT dictionary contains so many common words with neutral scores that they end up dampening the contribution to average valence of more meaningful words. The consequences of this indiscriminate approach are more visible for specialized and formal text.
Automated content analysis tools offer a trade-off between feasibility and accuracy: lexicon-based approaches are ready to use and fast to implement, and for this reason they offer an efficient choice when human annotations to train classifiers are unavailable. The validity of their measurement is likely to suffer, however, if the content analyzed relates to specific domains that are not well represented by general purpose dictionaries. The size of the lexicon is less relevant than the correct annotations of the appropriate words, and, when human coding is available, a machine-learning approach can help to scale up the analyses and improve the predictions. For this reason, and on the basis of our findings, we suggest that future research efforts concentrate on supervised approaches and on building training datasets that can be used to improve learning algorithms and increase their accuracy performance. We believe this is an exciting area of research in need of much refinement but with the potential to improve our measurement tools and contribute new evidence to develop theories of opinion formation.
Online communication evolves at a fast pace. Social research has to embrace more efficient and scalable methods if it is to use online data to improve current models and theories. Our aim with this work was to provide a benchmark to assess the reliability and accuracy of some of the publicly available methods for automated content analysis. In the process, we hope to have provided some guidance to determine research choices in future studies. Automated content analysis is a relatively new area of research in the social sciences. We hope this contribution encourages more work in this relevant domain.
Footnotes
Sandra González-Bailón is an assistant professor at the Annenberg School for Communication, University of Pennsylvania. Her research interests lie at the intersection of network science, data mining, computational tools, and political communication.
Georgios Paltoglou is a senior lecturer at the School of Mathematics and Computer Science at the University of Wolverhampton. His research interests include opinion mining, machine learning, and information retrieval.