Multi-thread hierarchical deep model for context-aware sentiment analysis

Abstract

Real-time messaging and opinion sharing in social media websites have made them valuable sources of different kinds of information. This source provides the opportunity for doing different kinds of analysis. Sentiment analysis as one of the most important of these analyses gains increasing interests. However, the research in this field is still facing challenges. The mainstream of the sentiment analysis research on social media websites and microblogs just exploits the textual content of the posts. This makes the analysis hard because microblog posts are short and noisy. However, they have lots of contexts which can be exploited for sentiment analysis. In order to use the context as an auxiliary source, some recent papers use reply/retweet to model the context of the target post. We claim that multiple sequential contexts can be used jointly in a unified model. In this article, we propose a context-aware multi-thread hierarchical long short-term memory (MHLSTM) that jointly models different kinds of contexts, such as tweep, hashtag and reply besides the content of the target post. Experimental evaluations on a real-world Twitter data set demonstrate that our proposed model can outperform some strong baseline models by 28.39% in terms of relative error reduction.

Keywords

Context-aware long short-term memory multi-thread sentiment analysis social media Twitter

1. Introduction

Social media websites have become very popular for sharing ideas and opinions about everything. In the social media, people share their opinions about different topics, discuss daily issues and express their sentiments about products they use [1]. In 2019, 2.95 billion users used social media websites, and in Twitter as one of the most famous social networks, people published 511,200 tweets each minute in July 2019.^1,2 In fact, the growth of the Internet has enabled opinion sharing on a wide range of topics. This huge amount of up-to-date data has brought opportunities for performing different kinds of analysis, including sentiment analysis about hot topics [2,3].

Sentiment analysis also called opinion mining [4] analyses opinions, sentiments, appraisals, attitudes and emotions in different types of data including video, audio and mostly text [5]. The ultimate goal of this analysis is to understand the emotional orientation towards entities including products, organisations, individuals, events, issues or topics [4]. Furthermore, it is sometimes more useful to extract the sentiment towards specific aspects of an entity, such as the weight of a laptop. In order to infer sentiment polarity about a unit, researchers have used predefined lexicons, machine learning or hybrid methods [6 –8]. Results of the sentiment analysis can benefit different groups: companies become aware of the costumer’s sentiment about their products and brands [1,9], governments can be informed of people’s opinion about their policies [10] and potential buyers can decide more efficiently, based on the opinions expressed by other people who have already used those products [11].

As the usage of social media increases, analysing its data gains more attention, too. Especially, growing attention to social media data has been reported in the field of sentiment analysis [4,6,12]. However, sentiment analysis in social media web sites such as Twitter [13] has some special challenges, including limitation of tweet length [6,8,14] and the dependency of tweets’ semantic to the other related tweets such as reply tweets [12,15]. These challenges make it hard to infer the sentiment using the content of the tweet itself (dubbed as the target tweet). In response to these challenges, some researchers suggested exploiting contexts of the target tweet. These contexts includes the tweets of the friends of the user who wrote the target tweet (dubbed as the tweep) [10,16], related tweets through conversations [12,14] or tweets of the similar tweeps to the tweep of the target tweet [6]. In this line of research, different methods have been suggested. Some recent papers suggested using recurrent neural networks (RNNs) to model the retweet/replies thread of related tweets, because tweets are published sequentially [12,15]. However, there are other kinds of useful sequential contexts that can be useful in determining the sentiment of the target tweet.

Sentiment consistency indicates that the tweets posted by a person tend to have similar sentiment [6,17]. Note that the most recent tweets of a tweep are more related to the target tweet than earlier ones. For example, consider this target tweet and its tweep context:

Target tweet:

‘I need to learn my lesson. I can’t be happy, I can’t be content. I don’t deserve to be happy, I don’t deserve to be content. I deserve nothing and I will always have nothing. I wish things were different but I don’t believe things will ever change. #Depression’.

Tweep context:

‘Mental illness runs in my family on my mother’s side. She has had issues as well as her mother and father and her grandmother also had issues. I was dealt a losing hand from the beginning. My mental health has degraded over the last 5 years to the point that I can’t function’.

‘Just when you think you are having kind of an ok day something comes along and ruins you and makes things worse than they have been. Drinking doesn’t help the situation, but it at least it makes you feel a little better. #Depression #bills #Sadness #sad’.

‘I have brief moments where I feel better about life in general. I wouldn’t call it happiness, but more moments of contentment. I wish it lasted longer than the fleeting time it happens. #mentalillness #mentalhealth #MentalHealthAwareness #sadness’.

‘I miss days when things were simple. It was only a few years ago. Looking back on it and I feel like I was happier then. Now there is nothing but stress and worry and dread and hopelessness. #sad #Depression #thoughts #mentalhealth #mentalillness’.

‘When your mental health is a full time job, there’s no vacation’.

Due to the existence of some positive words including happy, content and deserve, the textual models such as support-vector machine (SVM) and long short-term memory (LSTM) fail in analysing the sentiment of the target tweet (positive instead of negative). However, our proposed model which effectively utilises tweep context can infer a negative state about the tweep of the target tweet and make better estimation about the target tweet.

Furthermore, tweets with common hashtags tend to have similar sentiments. This is because that people have similar sentiments towards topics [6], which provide another context for the target tweet. In this article, we propose a multi-thread hierarchical long short-term memory (MHLSTM) network that uses different context types of the target tweet in order to determine its sentiment polarity. The proposed architecture is hierarchical in the sense that it considers word-based modelling of tweets’ content in one level and the tweet-based modelling of the thread at a higher level. We evaluate our proposed model using a real-world data set and show that it can outperform some strong baselines and state-of-the-art models. The main contributions of this article are as follows:

We propose MHLSTM that can jointly model the arbitrary number of different sequential contexts, such as previous tweets that the target tweet replies to. This model considers the order of tokens in the tweet and the order of context tweets in each of the threads in a hierarchical structure.

We model the tweets of a person as time-ordered sequence ends with the target tweet, and utilise his or her previous tweets to enrich the content of the target tweet.

We model tweets which have a common hashtag with the target tweet as a time-ordered sequence, and utilise previous tweets to enrich the target tweet representation.

The rest of this article is organised as follows. In section 2, we provide a literature review of the social media sentiment analysis and the use of deep learning methods in the field. Each subsection in section 2 ends with the contextualised models. Section 3 introduces the baselines and the proposed model. In section 4, we present our experiments. Finally, this article ends in section 5 with conclusion and future works.

2. Related work

2.1. Shallow learning in social media sentiment analysis

The surge of social media websites and the simplicity of their usage provide us a unique opinionated information source for different types of analysis, including sentiment analysis. The mainstream of research in sentiment analysis on social media concentrated on the content of the posts [6]. In this line, researchers have used different pre-processing techniques, features and classifiers. Kouloumpis et al. [18] used n-grams, lexicons, part-of-speech and micro-blogging features such as presence of emoticons and abbreviations as features. Go et al. [9] used different classification methods including Naive Bayes, maximum entropy and SVM on the text of tweets. Basile and Novielli [19] used SVM with features such as bigrams, number of hashtags, lexicon and embeddings of tweet words. Angiani et al. [20] investigated the impact of different pre-processing techniques on the Twitter sentiment analysis [21]. Their results showed that among different pre-processing methods, stemming is the most influential for improving Twitter sentiment classification task.

As posts in social media are not isolated, the meaning of a typical post is not completely understood without its context [8]. Therefore, another line of research used different kinds of related posts as contexts. Hu et al. [14] proposed sociological approach to handling noisy and short texts (SANT), a system that jointly minimises the difference between textual prediction and labels, as well as the difference between context-based predictions and labels. They used tweep and friends as contexts. Mi et al. [22] proposed the microblog sentiment analysis using user similarity and social relations (MSA-USSR) method which used the tweep similarity and interaction-based social relations and achieved better performance than SANT. Vanzo et al. [23] modelled the sentiment classification problem as a sequential classification task over thread of tweets and used SVM^hmm as their classifier. Zou et al. [6] proposed a method based on the sentiment consistency [17] and emotional contagion. Emotional contagion [24] indicates that similar people tend to have similar opinions. So, they combined social and topical contexts to improve sentiment classification.

2.2. Deep learning in social media sentiment analysis

In recent years, deep learning methods have reached remarkable results in many natural language processing tasks, including sentiment analysis [8,12]. Based on the idea that general embeddings (such as word2vec) of some opposite words such as ‘good’ and ‘bad’ are similar because of their similar contexts, some works proposed other word embedding techniques which are capable of capturing the words sentimental direction, too [25 –27]. Wang et al. [28] used long short-term memory (LSTM) for the Twitter sentiment classification and showed this model has better understanding of negative compositions. Since the initial word representations greatly affect sentiment classification, Rouvier and Favre [29] combined lexical, part-of-speech and sentiment embeddings to initialise input representations. They used a convolutional neural network (CNN) on top of these representations. Based on the same idea, Severyn and Moschitti [30] proposed to pre-train embeddings in two stages. First, they trained embeddings on 50 million tweets; then, to enrich representations with sentimental clues, they fine-tuned them with 10 million tweets that were labelled in distant supervision manner. Also, Cliche [31] used millions of tweets to train word embeddings and then refined them using distantly supervised tweets. He used these embeddings in the bidirectional long short-term memory (BiLSTM) and CNN networks. Majumder et al. [2] utilised correlation between sentiment classification and sarcasm detection proposed a multitask system. This model jointly predicts the sentiment of the tweet and the sarcasm.

Besides using the contents of the target tweet, a few researchers have examined different types of contexts to improve sentiment classification results of deep learning methods; however, these research works are in early stage and further works are still required. In this line, Croce et al. [32] proposed a system called the Unitor that injected sentiment information using polarity lexicons to CNN and used related tweets based on conversation or hashtag instead of arbitrary tweets in order to pre-train network. Since Twitter provides the features of reposting a tweet, called retweeting [33], and replying to a tweet, Huang et al. [12] used reply/retweet as the context and proposed the hierarchical long short-term memory (HLSTM) for tweet classification. In the first level of this model, a representation of each tweet was formed, and the representation of the thread was obtained in the second level. Feng et al. [15] proposed a similar system called context attention–based long short-term memory (CA-LSTM), but instead of using the last state of the LSTM, they equipped the model with attention mechanism in each level of the model. The attention mechanism made a weighted average of the all LSTM states resulting in a better representation. The last two methods proposed to model just one context in a sequential manner, but ignored the other related sequential contexts.

3. Baselines and the proposed method

In this section, we introduce the architecture of our proposed method and some baselines and the state-of-the-art models.

3.1. SVM

First, we train different types of linear SVM including SVM(1), SVM(1–3), context-aware support-vector machine (CASVM) and SVM(GloVe). The first three models use the bag-of-words (BOWs) representations of the target tweet’s text as features. This is a popular baseline for text classification [4,34]. SVM(1) uses only unigrams as vocabulary while SVM(1–3) uses unigrams, bigrams and trigrams in its vocabulary. In order to test the ability of SVM to benefit from contexts, in CASVM, we concatenate the target tweets’ tokens with its contexts’ tokens and feed them to SVM(1–3). We train different variants of this model including CASVM(user context), CASVM(hashtag context), CASVM(reply context) and CASVM(all contexts). Finally, SVM(GloVe) uses the average of the GloVe word embeddings [35] of the target tweet’s tokens as tweet’s representation.

3.2. LSTM

RNN is a special kind of the neural network models adapted to sequential data such as texts. These models (RNN and its variants) have proved their efficiency in sentiment analysis task [34]. In each time step, the RNN linearly combines input features of that time step and the current state of the system. It then applies a non-linearity to the result [36]

h_{t} = g (W h_{t - 1} + U x_{t} + b)

(1)

where $b$ is the bias term, $W$ and $U$ are the weight matrices, $h_{t}$ is the model state at the time step $t$ , $x_{t}$ is the input in time step $t$ , and g is a non-linearity function which usually is hyperbolic tangent (tanh). A problem with vanilla RNN is that increasing the number of time steps leads to vanishing or exploding gradients. This prohibits the network from learning long-term dependencies [15]. LSTM [37] is a kind of RNN that can learn long-term dependency connections by a complicated unit. In each time step, LSTM decides which information should be forgotten and which new information should be retained. We train an LSTM model as another baseline.

3.3. BiLSTM

While RNN considers only information from the previous states, bidirectional recurrent neural network (BRNN) [38] considers information from next states, too. In fact, BRNN combines two RNNs in different directions in order to use forward and backward information

\begin{array}{l} {\vec{h}}_{t} = σ (W^{\vec{h} \vec{h}} h_{t - 1} + W^{x \vec{h}} x_{t} + b^{\vec{h}}) \\ {\overset{\leftarrow}{h}}_{t} = σ (W^{\overset{\leftarrow}{h} \overset{\leftarrow}{h}} h_{t - 1} + W^{x \overset{\leftarrow}{h}} x_{t} + b^{\overset{\leftarrow}{h}}) \end{array}

(2)

where ${\vec{h}}_{t}$ and ${\overset{\leftarrow}{h}}_{t}$ are the forward and backward hidden states in time step t, respectively, and $W^{\vec{h} \vec{h}}$ , $W^{x \vec{h}}$ , $W^{\overset{\leftarrow}{h} \overset{\leftarrow}{h}}$ and $W^{x \overset{\leftarrow}{h}}$ are the weight matrices. Replacing RNN cell with LSTM cell results in BiLSTM. In our experiments, we concatenate the last two outputs of the model before passing them to the dense layer.

3.4. HLSTM and CA-LSTM

Text data exhibit sequential structure at multiple levels including characters, words, sentences, tweets and related tweets. These levels constitute a natural hierarchy for representing the meaning of different fragments of text [39,40]. Consider a thread of tweets in which each tweet has relation with previous ones, and each word in each tweet is related to other words in the same tweet. To model this phenomenon, Huang et al. [12] proposed HLSTM that uses a word-level LSTM which receives the words as input, and represents the tweet by the last state of the first-level LSTM. It then uses that representation as input to the second-level LSTM. In this line, Feng et al. [15] showed that the attention mechanism could improve the model performance, so they incorporated attention to both levels of the model. After obtaining the tweet representations, the model feeds each tweet representation to the second-level LSTM that produces an overall representation of the thread. The output of this level then is used for classification.

3.5. Proposed model: MHLSTM

There are some kinds of related tweets to the target tweet. These data can potentially be used to improve the quality of sentiment analysis. Furthermore, since each tweet has a publication time, related tweets constitute a sequence. As we mentioned earlier, both HLSTM and CA-LSTM represent each tweet based on one context. However, there are other related contexts which can be utilised in a similar way, such as the past tweets of the tweep who tweeted the target tweet. It has been shown that the tweets of the same tweep are more probably to have similar sentiments than random tweets [14]. Also, sentiment towards the same topic can be consistent to some degree [12]; for example, during some disasters or pandemics such as Corona virus outbreak, it is quite possible to see more negative tweets about that event. So, we propose multi-thread context-aware LSTM that can utilise different kinds of related sequences. Our proposed system contains two levels of LSTM in each sequence which model tweets and threads, respectively. The system then combines representations of different threads. Figure 1 shows the overall architecture of the proposed model. The architecture is structured by different threads. In each of these threads, we have a sequence of tweets ordered by time that ends with the target tweet. Inputs to the model are tweet tokens. There is a word-level LSTM for each thread

\begin{matrix} f_{t} = σ (W_{f} x_{t} + U_{f} h_{t - 1} + b_{f}) \\ i_{t} = σ (W_{i} x_{t} + U_{i} h_{t - 1} + b_{i}) \\ {\tilde{c}}_{t} = \tanh (W_{n} x_{t} + U_{n} h_{t - 1}) \\ c_{t} = f_{t} * c_{t - 1} + i_{t} * {\tilde{c}}_{t} \\ o_{t} = σ (W_{o} x_{t} + U_{o} h_{t - 1} + b_{o}) \\ h_{t} = o_{t} * \tanh (c_{t}) \end{matrix}

(3)

where $x_{t}$ is the input token initialised by a d-dimensional pre-trained word embedding or initialised randomly (we report both results), and it is fine-tuned during training. $σ (.)$ is the sigmoid activation function. $W_{f}$ , $W_{i}$ , $W_{n}$ and $W_{o} \in R^{h \times d}$ and $U_{f}$ , $U_{i}$ , $U_{n}$ and $U_{o} \in R^{h \times d}$ are the weight matrices, and $b_{f}$ , $b_{i}$ and $b_{o} \in R^{h}$ are the bias terms. $f_{t}$ , $i_{t}$ and $o_{t}$ are the forget, input and output gates, respectively. The variable ${\tilde{c}}_{t}$ is the candidate cell state, $c_{t}$ is the new cell state, and $h_{t}$ is the output of LSTM at time step $t$ (representation of token $t$ ). The * operator represents element-wise multiplication. For more information about LSTM, the reader can refer to Zhang [41] and Olah [42]. This word-level LSTM provides token embeddings for the attention mechanism and that makes a digest of the tweet

T = \sum_{t = 1}^{l} a_{t} h_{t}

(4)

where $T \in R^{h}$ is the representation of the tweet, $l$ is the length of the tweet (number of its tokens), and $a_{t}$ is the weight of the attention to the output of the state $t .$ In other words, $T$ is the weighted sum of tokens’ embeddings, which the model will use as the tweet representation. In equation (4), $a_{t}$ is computed as follows

a_{t} = \frac{\exp (f (a_{w}, h_{t}))}{\sum_{t = 1}^{l} \exp (f (a_{w}, h_{t}))}

(5)

where $a_{w} \in R^{a}$ is the attention vector, and $f (.)$ is a function that scores hidden states as follows

f (a_{w}, h_{t}) = a_{w}^{T} \tanh (W_{w} h_{t} + b_{w})

(6)

where $W_{w} \in R^{a \times h}$ and $b_{w} \in R^{a}$ are the weight matrix and the bias vector, respectively. In this stage, tweet representations $(Ts)$ are ready, and they are passed to the tweet-level LSTM. Tweet-level LSTM architecture is quite similar to equation (3) and produces states which are representations of tweets. It should be noted that these representations are affected by related tweets. As we mentioned, there are three tweet-level LSTMs (according to the number of threads). In effect, the input to a tweet-level LSTM is tweets in a thread (their representations $(Ts)$ provided by equation (4)). Then, the attention mechanism (as stated above in the equations (4)–(6)) is applied on the states produced by this LSTM. The result is the representation of the thread. In the last step, the model combines representations of different threads and sends the result to the softmax

\begin{matrix} C = [c_{1}, \dots, c_{n}] \\ Softmax (F_{i}) = \frac{\exp (w_{i} C + b_{i})}{\sum_{j = 1}^{k} \exp (w_{j} C + b_{j})} \end{matrix}

(7)

where $C$ is the concatenation of all threads’ representations $(c_{i} s)$ . The vectors $w_{i}$ and $w_{j}$ are the ith and jth rows of the weight matrix, and $b_{i}$ and $b_{j}$ are the bias terms. We have three threads in our experiments, so n = 3.

4. Experimental results and discussion

4.1. Data and experimental setup

In order to evaluate the proposed model, we use Twitter. It has been reported that Twitter is the main social media data source for sentiment analysis [4] and context-aware sentiment analysis [8]. For annotating data, we use a popular annotation approach for Twitter sentiment analysis called distant supervision [9,25,43 –46], also known as indirect crowdsourcing [7]. In this line, researchers use sentimental hashtags and emoticons to label tweets as positive or negative. Mohammad [43] conducted two experiments which showed that the hashtag-based labels were consistent with the human annotations [7], and Janssens et al. [45] compared valence of hashtag labels and human labels and indicated a high degree of agreement between them. Based on the previous research works, we used #happy, #excellent, #happiness, #sadness, #sad and #frustrated to collect target tweets using the Twitter official application programming interface (API) in 14 April 2019. We extracted 22,437 target tweets, including 11,370 (~51%) positive and 11,067 (~0.49%) negative ones. The searched hashtags themselves were then removed in order to prevent data leakage. We then gathered three kinds of the contexts for each target tweet (up to five previous tweets). These include tweets that the target tweet is a reply to them, previous tweets that include the first hashtag of the target tweet, and the previous tweets of the tweep of the target tweet. Table 1 shows the statistics of the data set. It is clear that a high percentage of tweets has the tweep and the hashtag contexts, and about one-fifth of the tweets have the reply context.

Table 1.

Data set statistics.

Tweets	Number (%)	Average length
Target tweets	22,437	–
Positive samples	11,370 (~51%)	–
Negative samples	11,067 (~49%)	–
Reply context	4259 (~19%)	0.30
Hashtag context	16,223 (~72%)	3.51
User context	21952 (~98%)	4.84
Context	22,227 (~99%)	–

We implemented all of the models using the TensorFlow [47], except SVM. For SVM, we used the scikit-learn library [48]. The proposed model was implemented in two settings. First, word vectors were initialised randomly, and second, they were initialised with the pre-trained GloVe word embeddings [35], and weights of theses vectors were fine-tuned during training to better capture sentiment clues. We used GloVe instead of word2vec [49] because the results of a recent paper shows that the GloVe outperforms word2vec in Twitter sentiment analysis [50]. We choose the best hyperparameters for each model experimentally. Table 2 shows the hyperparameter values we used for our model. We used dropout [51] for regularisation with keep rate 0.5 on the output of the LSTMs in both levels, and we used the Adam optimisation method [52] as optimiser. We compared our proposed model with some baselines and state-of-the-art models including SVM, LSTM, BiLSTM, HLSTM and CA-LSTM.

Table 2.

Hyperparameters of the model.

Hyperparameter	Value
Max tweet length (token)	50
Learning rate	0.0005
Keep prob	0.5
Batch size	32
Epochs	2
Word embedding dimension	200
RNN size	200
Attention size	200

RNN: recurrent neural network.

5. Results

In this section, we present our experimental results and compare different models and approaches. In each of the following table, the best model is specified in bold. Table 3 shows the results for baselines which does not use neither context nor pre-trained word embeddings (dubbed as baselines). As a linear model, SVM performs quite well on our data set and makes a powerful baseline. Perhaps, it is the reason that SVM has been reported as the most frequent classification method in the sentiment analysis literature [4]. SVM(1–3) provides the best performance. This model which is equipped with unigrams, bigrams and trigrams performs better than SVM(1), probably because it can handle negation and sentimental phrases. We tested other feature combinations, such as unigrams to five-grams, but we did not get any improvement. As the second baseline, LSTM works slightly better than SVM(1), but not as well as SVM(1–3). This is because deep models are data-hungry which means they need a lot of data to learn well [8].

Table 3.

Experiment results for simple baselines.

Model	Precision	Recall	Macro F
SVM(1)	91.65	90.54	90.80 ± 0.22
SVM(1–3)	92.98	90.80	91.65 ± 0.24
LSTM	90.91	89.35	90.91 ± 0.31
BiLSTM	91.31	90.68	90.54 ± 0.19

SVM: support-vector machine; LSTM: long short-term memory; BiLSTM: bidirectional long short-term memory.

Table 4 shows the experimental results for context-aware models. For SVM(1–3), we concatenated all the context tweets to the target tweet and considered that as a single sample. This kind of context usage has not any positive impact on SVM, because it does not consider the position of tokens. Also, context tokens may add noise to the model. This result has been reported in the previous research works, too [15]. Among different contexts, hashtag has the most negative effect on SVM, because people may have different sentimental tendencies towards the same topic, and the SVM cannot handle this. The reply context has the least negative effect, probably because of the low fraction of the context (about 20%).

Table 4.

Experimental results for context-aware models.

Model	Precision	Recall	Macro F
CASVM(reply context)	90.92	92.07	91.32 ± 0.26
CASVM(hashtag context)	79.19	71.98	76.19 ± 0.43
CASVM(tweep context)	85.19	81.92	83.62 ± 0.59
CASVM(all contexts)	92.31	91.02	91.60 ± 0.15
HLSTM	92.43	90.14	91.19 ± 0.56
CA-LSTM	91.94	91.48	91.61 ± 0.09
MHLSTM	93.54	92.88	93.13 ± 0.22

CASVM: context-aware support-vector machine; HLSTM: hierarchical long short-term memory; CA-LSTM: context attention–based long short-term memory; MHLSTM: multi-thread hierarchical long short-term memory.

CA-LSTM [15] with two levels of attention on the reply thread makes an appropriate representation of the tweets and the reply thread, so it performs better than HLSTM [12]. It can be seen that our proposed model (MHLSTM) that utilises three kinds of the contexts performs quite well and decreases the error rate by 17.72% $((93.13 - 91.65) / (100 - 91.65))$ with respect to the most powerful baseline (SVM(1–3)). In effect, there is 8.35 unit of error based on the SVM(1–3), and our model reduces this by 1.48 unit.

In another experiment, we initialised word embeddings with pre-trained GloVe word vectors and fine-tuned them along with the other parameters. The reason for fine-tuning is that some words with opposite sentiments such as good and bad have similar general embeddings because of their similar contexts [25,26,50]. Table 5 shows the results of this experiment. Comparing this table with Table 4 reveals the benefit of initialising neural models with pre-trained vectors. This observation is in line with previous research [15]. SVM(GloVe) does not perform well, because averaging in low-dimensional space can lead to a lot of information loss as reported in previous research works, too [40,53]. BiLSTM performs better than LSTM as a powerful model for sentiment analysis [34]. CA-LSTM performs better than all the other previous models, because it utilises the reply context and efficiently combines this information with the target tweet representation. Finally, the proposed MHLSTM performs better than all methods and decreases the error rate by 28.39% $((94.78 - 92.71) / (100 - 92.71))$ with respect to the second most powerful model (CA-LSTM).

Table 5.

Experiment results for models with pre-trained embeddings.

Model	Precision	Recall	Macro F
SVM(GloVe)	86.00	84.87	85.33 ± 0.49
LSTM	93.98	89.44	91.44 ± 0.21
BiLSTM	92.33	91.12	91.53 ± 0.68
HLSTM	92.05	92.16	91.19 ± 0.30
CA-LSTM	92.85	93.23	92.71 ± 0.25
MHLSTM	95.54	94.14	94.78 ± 0.20

LSTM: long short-term memory; BiLSTM: bidirectional long short-term memory; HLSTM: hierarchical long short-term memory; CA-LSTM: context attention–based long short-term memory; MHLSTM: multi-thread hierarchical long short-term memory.

Figure 2 shows the final comparison of different models including baselines, state-of-the-art models and the proposed MHLSTM model. It is obvious that our proposed model performs significantly better than all the other models.

In this section, we study the difference between context length of the correctly predicted samples by MHLSTM and incorrectly predicted ones. Table 6 shows all of the context types and their combination. It is clear that the number of contexts is significantly different between correctly labelled samples and incorrectly ones.

Table 6.

Difference between correctly labelled and incorrectly labelled sample length by MHLSTM.

Context	Average context length in correctly labelled samples	Average context length inincorrectly labelled samples	p-value
All contexts	8.74	8.12	0.00
User context	4.86	4.74	0.01
Hashtag context	3.58	3.20	0.00
Reply context	0.30	0.20	0.03

6. Conclusion

In this article, we proposed the MHLSTM model which incorporates preceding tweets of the different context types by the attention weights for multi-thread context-aware sentiment classification. We modelled different kinds of the contexts as separate threads and utilised hierarchical LSTM to jointly model tweets and threads. Our proposed model captures the long-distance dependency in each context and classifies target tweets based on that. Experimental results on a real-world Twitter data set demonstrated that our model improves classification results compared with the other baselines and some of the state-of-the-art models. Future direction for works includes exploiting other kinds of sequential contexts of the target tweet, such as tweets of the tweep friends, inspired by the sentiment contagion theory [54]. Different kinds of context utilisation can be investigated, too. Other directions are using of the contextual long short-term memory (CLSTM) [55], which combines contexts in the core architecture of the LSTM, and using context representation as LSTM initial state, instead of using zero-initialised state.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iD

Abdalsamad Keramatfar

Notes

References

Agarwal

Xie

Vovsha

et al. Sentiment analysis of twitter data. In: Proceedings of the workshop on language in social media (LSM 2011), Sakaka, Saudi Arabia, 3–4 April 2011, pp. 30–38.

Majumder

Poria

Peng

et al. Sentiment and sarcasm classification with multitask learning. IEEE Intel Syst 2019; 34: 38–43.

Hajhmida

Oueslati

. Predicting mobile application breakout using sentiment analysis of Facebook posts. J Inform Sci. Epub ahead of print 26 May 2020. DOI: 10.1177/0165551520917099.

Keramatfar

Amirkhani

. Bibliometrics of sentiment analysis literature. J Inform Sci 2019; 45: 3–15.

Poria

Cambria

Hazarika

et al. Context-dependent sentiment analysis in user-generated videos. In: Proceedings of the 55th annual meeting of the association for computational linguistics (Volume 1: Long Papers), Vancouver, CA, 12 July 2017.

Zou

Yang

Zhang

. Microblog sentiment analysis using social and topic context. PLoS ONE 2018; 13: e0191163.

Giachanou

Crestani

. Like it or not: A survey of twitter sentiment analysis methods. ACM Comput Surv 2016; 49: 1–41.

Sánchez-Rada

Iglesias

. Social context in sentiment analysis: Formal definition, overview of current trends and framework for comparison. Inform Fusion 2019; 52: 344–356.

Bhayani

Huang

. Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford, 2009, https://www-cs.stanford.edu/people/alecmgo/papers/TwitterDistantSupervision09.pdf

10.

Speriosu

Sudan

Upadhyay

et al. Twitter polarity classification with label propagation over lexical links and the follower graph. In: Proceedings of the First Workshop on Unsupervised Learning in NLP, Edinburgh, Scotland, 2011, pp. 53–63. Stroudsburg, PA: Association for Computational Linguistics.

11.

Liu

. Sentiment analysis: Mining opinions, sentiments, and emotions. Cambridge: Cambridge University Press, 2015.

12.

Huang

Cao

Dong

. Modeling rich contexts for sentiment classification with LSTM, 2016, https://arxiv.org/abs/1605.01478

13.

Emadi

Rahgozar

. Twitter sentiment analysis using fuzzy integral classifier fusion. J Inform Sci 2020; 46: 226–242.

14.

Tang

et al. Exploiting social relations for sentiment analysis in microblogging. In: Proceedings of the Sixth ACM International conference on web search and data mining, Rome, 4–8 February 2013, pp. 537–546. New York: ACM.

15.

Feng

Wang

Liu

et al. Attention based hierarchical LSTM network for context-aware microblog sentiment classification. World Wide Web 2019; 22: 59–81.

16.

. Semi-supervised microblog sentiment analysis using social relation and text similarity. In: 2015 international conference on big data and smart computing (Bigcomp), Jeju, 9–11 February 2015, pp. 194–201. New York: IEEE.

17.

Abelson

. Whatever became of consistency theory? Pers Social Psychol Bull 1983; 9: 37–54.

18.

Kouloumpis

Wilson

Moore

. Twitter sentiment analysis: The good the bad and the OMG! In: Fifth International AAAI conference on weblogs and social media, Barcelona, 17–21 July 2011.

19.

Basile

Novielli

. Uniba: Sentiment analysis of English tweets combining micro-blogging, lexicon and semantic features. In: Proceedings of the 9th international workshop on semantic evaluation (Semeval 2015), Denver, CO, 4–5 June 2015, pp. 595–600.

20.

Angiani

Ferrari

Fontanini

et al. A comparison between preprocessing techniques for sentiment analysis in Twitter. IOP Conf Ser 2016; 495: 012007.

21.

Zhang

Song

Zhang

et al. A quantum-inspired sentiment representation model for twitter sentiment analysis. Appl Intel 2019; 49: 3093–3108.

22.

Ruan

Xiao

. Microblog sentiment analysis using user similarity and interaction-based social relations. Int J Web Serv Res 2020; 17: 39–55.

23.

Vanzo

Croce

Basili

. A context-based model for sentiment analysis in twitter. In: Proceedings of COLING 2014, the 25th international conference on computational linguistics: technical papers, 2014, pp. 2345–2354, https://www.aclweb.org/anthology/volumes/C14-1/

24.

Hatfield

Cacioppo

Rapson

. Emotional contagion. Curr Direct Psycholo Sci 1993; 2: 96–100.

25.

Tang

Wei

Yang

et al. Learning sentiment-specific word embedding for Twitter sentiment classification. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (Volume 1: Long Papers), Baltimore, Maryland, June 2014 pp. 1555–1565. Stroudsburg, PA: Association for Computational Linguistics.

26.

Pan

Yang

et al. Learning word representations for sentiment analysis. Cognit Comput 2017; 9: 843–851.

27.

Maas

Daly

Pham

et al. Learning word vectors for sentiment analysis. In: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, Stroudsburg, PA, 21 June 2011, pp. 142–150. New York: ACM.

28.

Wang

Liu

Sun

C-J

et al. Predicting polarities of tweets by composing word embeddings with long short-term memory. In: Proceedings of the 53Rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (Volume 1: Long Papers), Beijing, China, 26–31 July 2015, pp. 1343–1353. Stroudsburg, PA: Association for Computational Linguistics.

29.

Rouvier

Favre

. SENSEI-LIF at SemEval-2016 Task 4: polarity embedding fusion for robust sentiment analysis. In: Proceedings of the 10th international workshop on semantic evaluation (Semeval-2016), San Diego, CA, 16–17 June 2016, pp. 202–208. Stroudsburg, PA: Association for Computational Linguistics.

30.

Severyn

Moschitti

. Twitter sentiment analysis with deep convolutional neural networks. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, Santiago, 9–13 August 2015, pp. 959–962. New York: ACM.

31.

Cliche

. BB_twtr at SemEval-2017 Task 4: Twitter sentiment analysis with CNNs and LSTMs. In: Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), Vancouver, BC, Canada, 3–4 August 2017, pp. 573–580. Stroudsburg, PA: Association for Computational Linguistics.

32.

Croce

Castellucci

Basili

. Injecting sentiment information in context-aware convolutional neural networks. In: 7th Italian information retrieval workshop, IIR 2016, Venezia, 30–31 May 2016.

33.

Boyd

Golder

Lotan

. Tweet, tweet, retweet: conversational aspects of retweeting on Twitter. In: 2010 43rd Hawaii international conference on system sciences, Honolulu, HI, 5–8 January 2010, pp. 1–10. New York: IEEE.

34.

Barnes

Klinger

Schulte im Walde

. Assessing State-of-the-Art Sentiment Models on State-of-the-Art Sentiment Datasets. In: Proceedings of the 8th workshop on computational approaches to subjectivity, sentiment and social media analysis, Copenhagen, 7–11 September 2017, pp. 2–12. Stroudsburg, PA: Association for Computational Linguistics .

35.

Pennington

Socher

Manning

. Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), Doha, Qatar, October 2014, pp. 1532–1543. Stroudsburg, PA: Association for Computational Linguistics.

36.

Goodfellow

Bengio

Courville

. Deep learning. Cambridge, MA: MIT Press, 2016.

37.

Hochreiter

Schmidhuber

. Long short-term memory. Neural Comput 1997; 9: 1735–1780.

38.

Schuster

Paliwal

. Bidirectional recurrent neural networks. IEEE Trans Signal Pr 1997; 45: 2673–2681.

39.

Guo

et al. Hierarchical attention based long short-term memory for Chinese lyric generation. Appl Intel 2019; 49: 44–52.

40.

Yang

Dyer

et al. Hierarchical attention networks for document classification. In: Proceedings of the 2016 Conference of the North American chapter of the association for computational linguistics: human language technologies, San Diego, CA, 12–17 June 2016, pp. 1480–1489. Stroudsburg, PA: Association for Computational Linguistics.

41.

Zhang

Wang

Liu

. Deep learning for sentiment analysis: a survey. Wires Data Min Knowl Discover 2018; 8: e1253.

42.

Olah

. Understanding LSTM networks. Colah’s Blog, 2015, https://colah.github.io/posts/2015-08-Understanding-LSTMs/

43.

Mohammad

. #Emotional Tweets. In: The first joint conference on lexical and computational semantics (*SEM), Montréal, QC, Canada, 8 July 2012, pp. 246–255. Stroudsburg, PA: Association for Computational Linguistics.

44.

Davidov

Tsur

Rappoport

. Enhanced sentiment learning using twitter hashtags and smileys. In: COLING ’10: Proceedings of the 23rd international conference on computational linguistics: Posters, Beijing, China, 23–27 August 2010, pp. 241–249. New York: ACM.

45.

Janssens

Verstockt

Mannens

et al. Influence of weak labels for emotion recognition of tweets. In: Mining intelligence and knowledge exploration, Cork, 10–12 December 2014, pp.108–118. New York: Springer.

46.

Alfina

Sigmawaty

Nurhidayati

et al. Utilizing hashtags for sentiment analysis of tweets in the political domain. In: Proceedings of the 9th international conference on machine learning and computing, Singapore, February 2017, pp. 43–47. New York, NY: Association for Computing Machinery.

47.

Abadi

Barham

Chen

et al. TensorFlow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX conference on operating systems design and implementation, Savannah, GA, 2–4 November 2016, pp. 265–283. Berkeley, CA: USENIX Association.

48.

Pedregosa

Varoquaux

Gramfort

et al. Scikit-learn: Machine Learning in Python. J Mach Learn Res 2011; 12: 2825–2830.

49.

Mikolov

Chen

Corrado

et al. Efficient estimation of word representations in vector space, 2013, https://arxiv.org/abs/1301.3781

50.

Çano

Morisio

. Word embeddings for sentiment analysis: a comprehensive empirical survey, 2019, https://arxiv.org/abs/1902.00753

51.

Srivastava

Hinton

Krizhevsky

et al. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 2014; 15: 1929–1958.

52.

Kingma

. Adam: a method for stochastic optimization, 2014, https://arxiv.org/abs/1412.6980

53.

Jiang

Lewris

Voltmer

et al. Integrating rich document representations for text classification. In: 2016 IEEE systems and information engineering design symposium (SIEDS) Charlottesville, 2016, pp. 303–308. New York: IEEE.

54.

Hill

Rand

Nowak

et al. Emotions as infectious diseases in a large social network: the SISa model. Proc Royal Soc B: Biol Sci 2010; 277: 3827–3835.

55.

Ghosh

Vinyals

Strope

et al. Contextual LSTM (CLSTM) models for large scale NLP tasks, 2016, https://arxiv.org/abs/1602.06291