The classification of rumour standpoints in online social network based on combinatorial classifiers

Abstract

It is a fact that most of the rumours related to hot events or emergencies can be propagated rapidly on the hotbed of online social networks. In order to track the standpoints of the participants of rumour topics to regulate the development of rumour, we propose a multi-features model combining classifiers to classify the rumour standpoints, defined as classifying the standpoints of online social network conversations into one of ‘agree’, ‘disagree’, ‘comment’ or ‘query’ on previous comment about the rumour. Testing the performance of the combinatorial model – decision tree with adaptive boosting classifier and extremely randomised trees with adaptive boosting classifier – on different features, that is, structuring the weight matrix based on combination of term frequency (TF), inverse document frequency (IDF) and term frequency – inverse document frequency (TFIDF) method and constructing the features vector with Word2vec method. The experiments show that the combinatorial classifiers that exploit different combination features in the online social network conversations outperform binary classification; especially, the topology of the social network has a highly positive impact on the classification results. Furthermore, the ‘comment’ and ‘query’ of rumour standpoints have a better classification effect based on the features of different categories.

Keywords

Breaking news combinatorial classifications online social network standpoints classification

1. Introduction

Rumour is a type of opinion on false facts that relate to hot events or emergencies, its spreading may cause social panic, in which the spreader will achieve some ulterior purpose [1 –4]. In recent years, the rapid development of information technology [4], especially the emergence of online social network platforms, such as the Twitter and Facebook [5], has improved the rate and breadth of rumour spreading more than ever before [6]. For example, a classic rumour spreading event about iodine being able to ward of nuclear radiation happened after the Fukushima nuclear disaster, which caused the Chinese, Americans, Russians and Koreans to rush for materials containing iodine [3,7]. The rumour event of Fukushima nuclear disaster led to certain extent of social unrest [8,9]. Therefore, it is necessary to guide and control rumour spreading in order to maintain social stability and regulate public opinions and views to help reduce the probability of social unrest caused by rumours [10 –12]. This leads the researchers to increase the attention on the rumour of the online social network. Especially, the research on whether the hot events or breaking news are rumours (i.e. the rumour detection or identification), which helps to control the process of rumours or even eliminate them at the source of rumour development. Zubiaga et al. [13] proposed a rumour-detection framework based on the rumour classification system, that is, determining whether the event is a rumour based on the classification result of the event standpoints generated from the posts of topic participants.

The classification of standpoints has been studied in many different scenarios or fields by some researchers. The classification of the rumour standpoints was also mentioned in preceding work: Qazvinian et al. [14] explored the effectiveness of three categories of features: content-based, network-based, and microblog-specific memes for correctly identifying rumours performing two-way classification of each tweet as supporting or denying a long-standing rumour; however, the number of the set standpoints is not enough to cover most of the tweets. In addition, the selection of three categories of features is not specific. Hamidian and Diab [15] employed the tweet latent vector (TLV) feature, which creates a 100-dimension vector representative of each tweet for increasing the precision of rumour retrieval task by adopting a supervised rumours classification task with the standard data set. The research of Hamidian and Diab extended the work of classification of rumour standpoints. Zeng et al. [16] explored the task of automatically classifying rumour stances expressed in crisis-related content posted on social media by utilising a data set of over 4300 manually coded tweets. However, too much data are duplicated in training set and testing set when dealing with data imbalance.

The focus of research for this article is to classify the opinions of the confirmed rumours, so as to analyse the characteristics and regulations of the classification of events standpoints and then help to detect and identify rumours. How do we confirm the standpoints of each tweet? The basic statement are as follows: such the current tweet replies with ‘yes, I agree’ or ‘of course’ affirmative contents corresponding to agree the previous opinion, otherwise, ‘nonsense’ or ‘no, it’s not true’ negative contents indicating that the current tweet disagree the preceding view, and other conditions are similar to this. Multiple researchers believe that combining the percentage of standpoints of tweets in events can help determine the truth and reliability of events [6,13], that is, if more than a certain percentage of tweets categorise their standpoints as ‘disagree’ determining that the event is a rumour and not credible [17 –19]. However, the tweets of the online social networks have some problems including many nonstandard expressions such as slangs, dialects, abbreviations or even the dirty words, as well as the dictions generally are not in line with grammar [20,21]. These add difficulty to our experiment for classification of rumour standpoints. That is why we employ the natural language processing and data mining techniques to experiment with tweets of the online social networks.

Aiming at these conundrum, the focus of this article intends to assess whether and the extent to which tree-structured combination classifiers can be of help in rumour standpoints classification. Cerri et al. [22] experimentally analysed the behaviour of different decision tree–based hierarchical multilabel classification methods based on the local and global classification approaches. However, in a threshold selection scheme, their method presented a tendency to favour empty predictions. Amin [23] presented a new technique for the recognition of Arabic text using the C4.5 machine learning system generating a decision tree for classifying each word. Nevertheless, there is a phenomenon of inconsistency in segmentation of words and sub-words. On the other hand, Simm et al. [24] proposed a new tree-based ensemble multi-task learning method for classification and regression (MT-ExtraTrees), based on extremely randomised trees (ExtraTrees). However, the application of decision tree and the ExtraTrees are simply, and they cannot match the scattered characteristics of tweets. In our experiments, we apply the tree-structured classifiers to simulate the network topology of different tweets in online social networks. The experimental performance of classifier in rumour standpoints classification is explored, where the classifiers include two conformations: combining the adaptive boosting classifier with decision tree and ExtraTrees, respectively, comparing experimental performance with results of selected baseline classifier. In previous studies, many of them were based on the features extracted from the tweets themselves for experimenting [25 –27]. Hence, we creatively apply the feature based on the network topology structure to test the performance of experiments and achieve extremely favourable classification results. The ID of the published tweet is unique and can be used as an identity symbol to identify the publisher and analyse the association between different IDs, where the interaction between different tweets presents the characteristics of tree-structured and complex network [28,29].

Zhang et al. [30] proposed a method for sentiment classification based on Word2vec and SVMperf, where the Word2Vec shows the capability to capture the semantic features in selected domain and Chinese language. Chen [31] proposed a new distance-based term weighting method for overcoming the bias of term frequency – inverse document frequency (TFIDF) algorithm by considering a basic characteristic, whereby each news article must be similar or different from others while processing big news that include large amounts of news. We study the features of tweets, especially the interaction of tweets (e.g. the topology of online social networks) and exploring whether and the strength to which they help to perform the classification of rumour standpoints by the combinational classifiers with tree-structured tweets. Based on the combination model of TFIDF, term frequency (TF) and inverse document frequency (IDF), we process the initial tweet data and combine Word2vec to achieve vector representation of vocabulary and features. TFIDF is a commonly used weighting technique for information retrieval and data mining, that is, reflecting how important a word is to a document in corpus or a sentence in a document [32,33].

The remainder of the article is structured as follows. Second section illustrates the methodology containing four parts: data set, the overall technology framework, classifiers part, as well as the baseline classifiers and experiment settings. The experimental results calculated from the methodology part will be displayed in the third section, that is, ‘Results and discussion’. Some conclusions are listed in the final section, as well as some suggestions for future directions.

2. Methodology

The main purpose of our study is to analyse whether the statement structure and selected features of online social network conversations can be exploited to improve the classification of the standpoints expressed by different participators towards the trending event. Each post of rumour topic represents personal views of the current topic participator, so it can be defined or annotated as a standpoint on the rumour topic just like the ‘agree’, ‘disagree’, ‘comment’ or ‘query’. However, there are many relationships between different posts in rumour topics. Based on this, we focus on improving the performance of classifiers that determine the rumour standpoint of each post.

2.1. Data set

Our data set is adopted from eight rumour topics (namely A, B, C, D, E, F, G, H) corresponding to eight breaking news, collected from the tweets of Twitter platform [6], where the data set is part of the PHEME project [34,35]. The rumour standpoints label (i.e. ‘agree’, ‘disagree’, ‘comment’ or ‘query’) of tweets in the eight rumour topics are all revised and annotated by the journalists and linguist based on the tweets annotated labels.

One important characteristics of eight rumour topics is that the uneven distribution of the rumour standpoints label, especially the tweets annotated as ‘comment’, which account for 63.91% of the total tweets. However, the rumour standpoint labels ‘query’ just account for 4.18% of the total tweets. This imbalance acts slightly different across the eight rumour topics in the data set. Meanwhile, the imbalance distribution of the rumour standpoints increases the challenge of classification for the rumour standpoints, specific as shown in Table 1 and Figure 1, which is closer to the real-world scenario, that is, it has great practical significance.

Table 1.

Standpoints distribution of eight rumour topics.

Rumour topics	Agree, n (%)	Disagree, n (%)	Comment, n (%)	Query, n (%)	Total, n
A	263 (22.65)	114 (9.82)	745 (64.17)	39 (3.36)	1161
B	3 (8.82)	6 (17.65)	24 (70.59)	1 (2.94)	34
C	167 (14.52)	174 (15.13)	755 (65.65)	54 (4.7)	1150
D	63 (22.03)	24 (8.39)	186 (65.03)	13 (4.55)	286
E	152 (19.36)	108 (13.76)	486 (61.91)	39 (4.97)	785
F	19 (19.00)	10 (10.00)	63 (63.00)	8 (8.00)	100
G	10 (15.62)	11 (17.19)	37 (57.81)	6 (9.38)	64
H	223 (19.26)	165 (14.25)	732 (63.21)	38 (3.28)	1158
Total	900 (18.99)	612 (12.92)	3028 (63.91)	198 (4.18)	4738

Figure 1.

Standpoints distribution of eight rumour topics in each standpoint label.

As we can see in Table 1, there are totally 4738 tweets for the rumour topics in data set. Our purpose is to classify the standpoints of each tweet in rumour topics, which is described as follows: the collected data set includes eight rumour events, namely $D = {A, B, C, D, E, F, H}$ , each discussing a rumour topics. Each topic (e.g. the rumour topic A) is subdivided into many sub-topics, $A = {S_{A 1}, S_{A 2}, S_{A 3}, \dots, S_{Aj}}$ , which consist of different amounts of tweets (i.e. the tweets for the sub-topic), $S_{Aj} = {t_{Aj, 1}, t_{Aj, 2}, \dots, t_{Aj, k}}$ , where the $t_{Aj, 1}$ represents the source tweet (the root of tree-structured tweets) of the sub-topic $S_{Aj}$ and the $t_{Aj, k}$ (the leaf of tree-structured tweets) represents a reply to the $t_{Aj, 1}$ or $t_{Aj, k - 1}$ . Our purpose is to determine the standpoint of the tweet $t_{Aj, k}$ for all the rumour topics. An example of a sub-topic is shown in Figure 2.

Figure 2.

Example of a tree-structured sub-topic discussing the Ottawa shooting incident, where the annotated labels (‘agree’, ‘disagree’, ‘comment’ and ‘query’) are the rumour standpoints of each ID.

2.2. The overall technology framework

The overall technology framework consists of two parts, that is, the data processing part and classifiers part, as shown in Figure 3. In the stage of the data processing, we apply the method of the combinatorial TFIDF (TF, IDF and TFIDF) and the Word2vec to process the tweets for getting the M × N dimensional weight matrix and the 100-dimensional features vector, respectively. Then the weight matrix and features vector are combined and sent to combinatorial classifiers for classification experiments. As we can see from the Figure 3, where the ‘F’ is the 100-dimensional various features vector generated from the rumour tweets processed using the Word2vec method. The specific content will be explained in the data processing part and the classifiers part, respectively.

Figure 3.

The overall technology framework of classification of rumour standpoints.

First, preprocessing operations such as removing stop words and part of speech tagging are applied to the original rumour tweets corpus. Then, we combine the TF, IDF and TFIDF methods to construct the weight matrix of $M \times N$ dimensions. In the case of TF, we calculate it as

TF (x_{i}, t_{Aj, k}) = \frac{t_{x_{i}}}{n}

(1)

where the $t_{x_{i}}$ is the number of word $x_{i}$ in the tweet $t_{Aj, k}$ , and n represents the number of words in the tweet $t_{Aj, k}$ .

In the case of IDF, we calculate it as

IDF (x_{i}) = \log (1 + \frac{N}{T_{x_{i}}})

(2)

where N is the number of tweets in the rumour topics and the $T_{x_{i}}$ is the number of tweets containing the word $x_{i}$ . By combining the equations (1) and (2), the TFIDF score of word $x_{i}$ in a tweet, denoted as $TFIDF (x_{i}, t_{Aj, k})$ , is then calculated as the word of $TF (x_{i}, t_{Aj, k}) \times IDF (x_{i})$ . Based on the TFIDF method, we construct a weight feature containing three values: TF score, IDF score and TFIDF score described as equation (3)

W_{TFIDF} = TF (x_{i}, t_{Aj, k}) * IDF (x_{i}) * TFIDF (x_{i}, t_{Aj, k})

(3)

where the $W_{TFIDF}$ is the weight feature illustrated in the Figure 4.

Figure 4.

The working process of combination for TF, IDF and TFIDF.

While focusing on the study of combinatorial classifiers for rumour standpoints classification, we design our experiments with two types of features: original features and contextual features, which are derived from the tweets themselves and contextual relationship between tweets, respectively. In the original features section, we employ these features to evaluate the performance of combinatorial classifier under the same condition, which make it a fairer comparison that we can quantitatively assess and boost the performance of each feature on the results of classification, where features are extracted from the current tweet. In succession, that is, the contextual features part generated from the relevant tweets in the same sub-topic, which drive us to further promote the performance of classification. The original features and contextual features include several subtypes of features, as shown in Table 2.

Table 2.

The description of original features and contextual features.

Original features
Lexicon	Word embedding
Sentiment words	Negative words
	Bad words
	Exclamation mark
	Question mark
Reliability proof	URL link
Reliability proof	Pictures or video marks
Contextual features
Correlation	Cosine similarity between current tweet and source tweet
Correlation	Cosine similarity between current tweet and previous tweet
Network topology	Number of reply
Network topology	Degree distribution

These features are explained as follows:

Word embedding: we apply the Word2vec to support the content of each tweet by constructing the vector values. We structure the 100-dimensional Word2vec model by training the Wikipedia corpus and the eight rumour topics corpus. Based on the Word2vec model, we represent each tweet by calculating the average vector of all the words in the tweet.

Negative words: this feature presents the number of negative words that exist in the tweet, where the negative words consist of the following words: not, no, nobody, nothing, none, never, neither, nor, nowhere, hardly, scarcely, barely, don’t, isn’t, wasn’t, shouldn’t, wouldn’t, couldn’t, doesn’t, few, seldom, little, avoid, ban, cancel, deny, deprive, forbid, ignore, lack, lose, miss, neglect, stop and prohibit.

Bad words: this feature captures the number of bad words that present in the tweet, where the 371 bad words are stored in a list.

Exclamation mark: this feature indicates the number of exclamation marks in the tweet.

Question mark: this feature indicates the number of question marks in the tweet.

URL link: this feature determines the number of URL labels when the URL link is in the tweet.

Pictures or video marks: this feature indicates the number of pictures or video marks when they exist in the tweet.

Cosine similarity between current tweet and source tweet: this feature presents the semantic correlation between the current tweet and the source tweet in each rumour sub-topic by calculating the cosine similarity between the words vector of current tweet and the words vector of source tweet. This function will help us to research the impact of source tweet on the current tweet.

Cosine similarity between current tweet and previous tweet: as described in the previous feature, this feature explains the semantic correlation between the current tweet and the previous tweet, that is, which is directly responding to by the current tweet.

Number of reply: this feature determines the number of a tweet that has been responded.

Degree distribution: degree distribution is a concept existing in graph theory and complex network theory [36]. It reflects the probability distribution of each node in the network. Based on this, we can apply the degree distribution in the online social network which is a kind of complex network, where the degree is the number of other IDs connected to the current ID. Therefore, this feature helps us to study the influence of the network structure on the classification of the rumour standpoints. And the overall network connection figure of the eight rumour topics is shown in the Figure 5.

Figure 5.

(a) The overall network connection figure of the eight rumour topics and (b) zoomed in version of part (a).

As we can see from Figure 5, the size of the node depends on the size of the degree. The greater the degree, the larger the node, which means that the more important the node is, the more influence it has on the classification of the rumour standpoint. This is why we added the degree distribution as a feature to the experiment.

The mathematical vector generated by Word2vec is much denser [37]. Therefore, the Word2vec method is applied to vectorization of words, so that each word can be mapped to a set of vectors for mathematical computation and simulation. The Word2vec working process can be illustrated in Figure 6.

Figure 6.

The Word2vec CBOW model working procedure.

The words are put into the Word2vec – continuous bag-of-words (CBOW) model to calculate the 100-dimensional word vector of the target word $W (t)$ , which the words are generated from the tweets. The Word2vec has two language conversion models, CBOW and Skip-gramme, where the CBOW model can output the word vector of the target word $W (t)$ according to the adjacent words information $(W (t - 2), W (t - 1), W (t + 1), W (t + 2))$ of the target word. The word vector of the target word $W (t)$ corresponds to the tweet containing the word $W (t)$ . After Word2vec processing, a vector model is built for subsequent experiment calls, and the parameter setting will be introduced as follows:

The vector dimension: $s ize = 100$ . The distance of words: $window = 2$ . The learning rate: $alpha = 0.0001$ . The Number of iterations: $iter = 10$ , and other parameters will be set default.

2.3. Classifiers part

In this section, we describe the combinatorial classifiers that are applied to our experiments. As we can see from the Figure 2, the discussion about the rumour topics in the online social network is the communication between the people represented by the different ID numbers, which presents tree-structured characteristics. Based on the tree-structured characteristics, we apply the ExtraTrees classifier and decision tree classifier. In order to further enhance the performance of the classification, we employ the ExtraTrees classifier and decision tree classifier as the base estimator to test the adaptive boosting classifier.

The idea of decision tree algorithm mainly comes from the ID3 algorithm [38] and its improved algorithm, the C4.5 algorithm [39], as well as the classified regression decision tree (CART) [40]. We mainly experiment with decision tree based on the CART algorithm. CART algorithm constructs trees using the feature and threshold that yield the largest information gain at each node based on the Gini coefficient described as follows

Gini (D) = 1 - \sum_{k = 1}^{K} \frac{{| C_{k} |}^{2}}{| D |}

(4)

where $Gini (D)$ denotes the uncertainty of the D, k is the number of the rumour standpoints, and D is the tweets of eight rumour topics after being processed by the TFIDF and Word2vec. $C_{k}$ is a subset belonging the class k in D. Besides, if the D is divided into $D_{1}$ and $D_{2}$ according to whether the feature $B$ takes a certain possible value $b$

D_{1} = {(x, y) \in D | B (x) = b}

(5)

D_{2} = D - D_{1}

(6)

Based on the equations (5) and (6), we can give the Gini coefficient of D under the condition of feature $B$

Gini (D, B) = \frac{D_{1}}{D} Gini (D_{1}) + \frac{D_{2}}{D} Gini (D_{2})

(7)

where $Gini (D, B)$ indicates that the uncertainty after D is decomposed when $B = b$ . Based on the equations (4)–(6), the steps to generate the classification tree are described as follows:

Calculating the Gini coefficient of D by the feature $B$ , that is, dividing D into $D_{1}$ and $D_{2}$ when $B = b$ .

Determining the feature $B$ and the segmentation points $b$ when the Gini coefficient reaches the least value.

Calling the step 1 and step 2 at the optimal subtree.

Generating the CART tree.

At last, pruning the decision tree by minimising the overall loss function of the decision tree to avoid overfitting. And the parameter setting will be introduced as follows:

The function to measure the quality of a split: $criterion =' gini'$ . The maximum depth of the tree: $\max_depth = 10$ . The minimum number of samples required to be at a leaf node: $\min_samples_leaf = 1$ . The seed used by the random number generator: $random_state = 2$ . Other parameters will be set default.

Geurts et al. [41] proposed the ExtraTrees algorithm derived from the random forest (RF) algorithm [42]. The main process of the ExtraTrees is described as follows:

Providing the sample data set $D = {(x_{1}, y_{1}), (x_{2}, y_{2}), \dots, (x_{k}, y_{k})}$ and setting the number of iterations as M for weak classifiers.

Generating the strong classifiers based on the step 1.

Sampling k times at the m iterations to acquire the data set $D_{m}$ containing k samples.

Then, it will randomly select a feature value for classifiers while training the m classifier based on the $D_{m}$ .

This operation is the difference between the ExtraTrees and the RF, which help to enhance the generalisation ability of ExtraTrees. The parameter setting will be introduced as follows:

The number of trees in the forest: $n_estimators = 90$ . The number of features to consider when looking for the best split: $\max_features = 19$ . The maximum depth of the tree: $\max_depth = 16$ . The minimum number of samples required to split an internal node: $\min_samples_split = 190$ . Other parameters will be set default.

Freund and Schapire [43] proposed the adaptive boosting algorithm (i.e. the AdaBoost Classifier), which has the following characteristics: AdaBoost is a classifier with high accuracy and provides a framework that can be used to construct sub-classifiers in a variety of ways. Otherwise, the weak classifier constructed on the basis of A is extremely simple. In addition, there is no feature screening and overfitting problem. Based on the characteristics of the adaptive boosting algorithm, we employ the AdaBoost Classifier as the framework and experiment with decision tree algorithm and the ExtraTrees algorithm as the sub-classifiers, constructing the combination classifiers, that is, AdaBoost–DecisionTree (AB-DT) and AdaBoost–ExtraTrees (AB-ET).

2.4. Baseline classifiers and experiment settings

Based on our previous experimental plan, we adopt the baseline classifiers from four categories: support vector machines (SVMs), RF, conditional random fields (CRFs) and the logistic regression (LR). The training data set is 60% of the total data, and the testing data set is 40%. We use their implementation in the scikit-learn Python package to all the experiments. In addition, the parameters of each baseline classifier will be set default.

We experiment in a five-fold cross-validation setting. In our cross-validation setting, we run the classifier five times, on each occasion having a different fold for testing, with the other four folds for training. In this way, each fold is tested once, and the aggregation of all folds enables experimentation on all data. In the tweets of data set, there are many slang and abbreviations that do not conform to grammar, which results in extreme imbalance and irregularity of the tweets. Furthermore, this article intends to assess whether and the extent to which the tree-structured combination classifiers can be of help in rumour standpoints classification. Aiming at this problem, we employ the macro-averaged F1 scores (Macro-F1) as the standard for comparison, where the Macro-F1 judges the overall performance of each category, that is, computing locally over each category first and then average over all categories is taken.

3. Results and discussion

On the basis of the experimental plan mentioned above, first of all, we assess the performance of the experiment with classifiers on the original features only, which make it a fairer comparison that we can quantitatively assess and boost the performance of each feature on the results of classification, where features are extracted from the current tweet. The original features, extracted solely from each tweet in isolation, can evaluate whether the performance of the classification results of different classifiers can be enhanced based on the same type features of each tweet separately.

Table 3 shows the classification results of diverse classifiers under the combinations of original features, that is, experimenting with each subtype of original features and then employing the combination of subtypes, at last, applying the all of the subtypes in the original features.

Table 3.

The performance results of Macro-F1 using only the original features (OFs).

Classifiers	OF1	OF2	OF3	OF12	OF13	OF23	OF123
LR	0.321	0.383	0.329	0.364	0.320	0.385	0.365
SVM	0.273	0.387	0.321	0.336	0.269	0.359	0.342
CRF	0.327	0.396	0.366	0.374	0.328	0.391	0.394
RF	0.337	0.456	0.385	0.417	0.335	0.406	0.407
AB-DT	0.371	0.446	0.399	0.426	0.375	0.436	0.431
AB-ET	0.347	0.453	0.349	0.429	0.371	0.471	0.436

OF1: lexicon; OF2: sentiment words; OF3: reliability proof; LR: logistic regression; SVM: support vector machine; CRF: conditional random field; RF: random forest; AB-DT: AdaBoost–DecisionTree; AB-ET: AdaBoost–ExtraTrees.

Bold values indicate the optimal results of different classifiers under each feature.

From the results in Table 3, we can present some observations as follows:

Compared with the baseline classifiers, the combinatorial classifier of AB-DT and AB-ET perform good results overall.

The AB-ET classifier and AB-DT classifier gains three optimal Macro-F1 scores in each feature, respectively. That is to say, the AB-ET classifier and AB-DT classifier achieves the best classification results and the second best classification results in each feature, respectively, except the classification result of RF classifier under the feature OF2. Otherwise, the RF classifier is better than other baseline classifiers. Therefore, the combinatorial classifiers outperform single classifiers. This also corresponding to our presuppositions and experimental objectives.

The AB-ET classifier acquires the optimal Macro-F1 score 0.471 under the feature OF23; however, the RF classifier employs the Macor-F1 score 0.456 under the Feature OF2, which is second only to the former. Moreover, the Macro-F1 scores of the classifiers under the feature OF2 are generally better than other features, for example, the Macro-F1 score 0.446 of AB-DT classifier under the feature OF2. By and large, the Macro-F1 score of each classifier increases with the increase in number of features. The reason is that OF2 consists of four features (negative words, bad words, exclamation mark and question mark), meanwhile, the OF1 contains only one feature (word embedding), and the OF3 owns two features (URL link, pictures or video marks). In addition, the Macro-F1 scores of the combination features containing the OF2 is often better than the combination features without OF2. In conclusion, the combination of multiple features contributes to the improvement of the Macro-F1 score.

Even though the combination of multiple features is beneficial to increase the Macro-F1 score, some combined features will weaken this improvement. For instance, the Macro-F1 scores of SVM classifier under the feature OF12, OF23 and OF123 are 0.336, 0.359 and 0.342, respectively, which are all less than the Macro-F1 score 0.387 under the feature OF2. Therefore, it is not the more the number of features, the better the Macro-F1 score.

In brief, the Macro-F1 scores, using only the original features, provide a preferable result. Then, we obtain the combination features of negative words, bad words, exclamation mark, question mark, URL link, pictures or video marks calculating the great precision results (i.e. the Macro-F1 score 0.471 of AB-ET classifier under the feature OF23). Based on this, we experiment with the combination of original features and the contextual features in the next section.

From the results in Table 4, we can present some observations as follows:

The Macro-F1 scores of classifiers under the feature CF12 in Table 4 are overall better than the classifiers under the feature OF123 in Table 3, except the Macro-F1 scores of the feature CF2 in Table 4 (i.e. on the contrary, the Macro-F1 scores of the feature OF2 are superior to the feature CF2). Moreover, the Macro-F1 scores of classifiers under the feature CF1 are better than the classifiers under the features OF1 and OF3. Therefore, the Macro-F1 scores of the contextual features has been significantly improved comparing to the original features.

The AB-ET and AB-DT classifiers gain four and three optimal Macro-F1 scores in each feature, respectively, that is, the best classification results and the second best classification results in each feature. Just like the results discussed in Table 3, the combinatorial classifiers outperform single classifiers under the combination features of OF and CF.

Even though the Macro-F1 scores of classifiers under the feature CF2 are inferior to that of the feature OF2, they gain Macro-F1 scores with two features (number of reply and degree distribution) that are superior to the other features. Moreover, the best Macro-F1 score 0.476 compared with all of the features comes from the AB-ET classifier with the feature OF + CF2. That is to say, the features (degree distribution and number of reply) derived from the network topology have a strong positive impact on improving the performance of the Macro-F1 score.

Table 4.

The performance results of Macro-F1 using only the contextual features (CFs), as well as the combination of the OF and CF.

Classifiers	OF	CF1	CF2	CF12	OF + CF1	OF + CF2	OF + CF
LR	0.385	0.382	0.386	0.387	0.390	0.399	0.403
SVM	0.359	0.379	0.382	0.385	0.397	0.414	0.406
CRF	0.391	0.398	0.401	0.414	0.402	0.439	0.407
RF	0.406	0.397	0.407	0.401	0.415	0.445	0.424
AB-DT	0.436	0.430	0.439	0.438	0.416	0.423	0.426
AB-ET	0.471	0.423	0.448	0.436	0.472	0.476	0.422

CF1: correlation; CF2: network topology; LR: logistic regression; SVM: support vector machine; CRF: conditional random field; RF: random forest; AB-DT: AdaBoost–DecisionTree; AB-ET: AdaBoost–ExtraTrees.

Bold values indicate the optimal results of different classifiers under each feature.

Based on the combination of the original features and the contextual features, the Macro-F1 scores are significantly improved than that of a single one, for example, on the feature OF + CF2 and OF + CF1, the AB-ET classifier achieve the best Macro-F1 score 0.476 and the second best Macro-F1 score 0.472, respectively. What needs to be noted is that the feature of the network topology introduced for the first time exhibits excellent experimental performance, where the network topology can better fit the tree-structured characteristics, that is, the flow of information through the Twitter community is similar to the complex network in which nodes are individual user accounts, where the flow is the edge of the network. The example diffusion network is shown in Figure 5.

In order to further analyse our experiments, we give the confusion matrices of different classifiers for each standpoints. Table 5 shows the confusion matrices embodying the deviations calculated from the different classifiers, where the deviations help us to research the performance of classification. The bold values displayed in the diagonals are the correct classification probability for each rumour standpoint.

Table 5.

Confusion matrices of deviations for different classifiers.

LR
	Agree	Comment	Disagree	Query
Agree	0.351	0.619	0.008	0.022
Comment	0.087	0.808	0.056	0.049
Disagree	0.083	0.706	0.175	0.036
Query	0.132	0.431	0.112	0.325
SVM
	Agree	Comment	Disagree	Query
Agree	0.377	0.525	0.068	0.030
Comment	0.148	0.719	0.095	0.038
Disagree	0.178	0.612	0.107	0.103
Query	0.102	0.393	0.153	0.352
CRF
	Agree	Comment	Disagree	Query
Agree	0.397	0.505	0.068	0.030
Comment	0.128	0.769	0.075	0.028
Disagree	0.158	0.612	0.147	0.083
Query	0.042	0.353	0.113	0.492
RF
	Agree	Comment	Disagree	Query
Agree	0.451	0.502	0.014	0.033
Comment	0.061	0.896	0.021	0.022
Disagree	0.063	0.717	0.164	0.056
Query	0.014	0.507	0.028	0.451
AB-DT
	Agree	Comment	Disagree	Query
Agree	0.468	0.437	0.065	0.030
Comment	0.031	0.861	0.080	0.028
Disagree	0.155	0.395	0.367	0.083
Query	0.042	0.320	0.127	0.511
AB-ET
	Agree	Comment	Disagree	Query
Agree	0.494	0.451	0.014	0.041
Comment	0.041	0.907	0.022	0.030
Disagree	0.083	0.478	0.360	0.079
Query	0.011	0.268	0.028	0.693

LR: logistic regression; SVM: support vector machine; CRF: conditional random field; RF: random forest; AB-DT: AdaBoost–DecisionTree; AB-ET: AdaBoost–ExtraTrees.

The bold values are the correct classification probability for each rumour standpoint.

From the deviations, we can see that the combinational classifiers with tree-structured characteristic present the best results in the overall state, where the probability 0.907 of the AB-ET is the best result comparing with the other classifiers under the standpoints type of ‘comment’. At the same time, a large number of standpoints are misclassified into ‘comment’; the reason is that the standpoint of ‘comment’ accounts for the largest proportion of data set, reaching 63.91%. Correspondingly, the probability of standpoint ‘disagree’ only has a maximum of 0.367 for the AB-DT classifier, with a minimum of 0.107 for the SVM classifier, where the standpoint of ‘disagree’ only accounts for a proportion of 12.92% for the data set.

In other words, as far as the standpoint of ‘comment’ is concerned, there is a large amount of data for training, which shows that the amount of data has an extremely important positive effect on the classification results. If the training set is balanced by reducing the number of standpoint ‘comment’, this will disrupt the structure of the online social network, which will have a negative impact on the classification of rumour standpoints. Aiming at this scenario, it can be solved by increasing the amount of data or perfecting the experiment scheme (e.g. employing the more appropriate features, models or algorithms).

4. Conclusion and future work

The research on rumour stance classification in online social networks based on the natural language processing is rarely covered by the previous work. Our study on the rumour standpoints classification in this article pays some significant contributions for future study: experimenting with the tweets that present the characteristic of extreme imbalance and irregularity; we consider not only the properties of the tweets themselves but also the tree-structured characteristic of the tweets with correlation; we take into account the feature of the social network topology creatively, which performs favourable classification results. Comparing to a range of baseline classifiers, we apply the combinational classifiers AB-DT and the AB-ET to construct the tree-structured classifier of tweets for acquiring preferable results.

From the above experiments, it can be known that the results by considering the combination of tweets and features are better than the experimental results that simply considered the tweets, and the effect of multi-features combinations is particularly obvious. In particular, the feature of the online social network topology that we creatively apply highlights the better results. Hence, in the future work, we try to experiment with classification of rumour standpoints as well as the identification of rumour by considering the properties of complex network, for example, the shortest path, average degree, out-degree, in-degree and clustering coefficient. At the same time, the presentation of the effects for each feature helps us to understand and analyse the influence of individual feature or combinational features on the classification effect. This will provide reference and help for further research on the classification of rumour standpoints in the future.

Moreover, the study of rumour stance classification for online social networks has great practical significance. Based on the research of rumour classification, our future work will not only further improve the effectiveness and accuracy of classification but also improve further applications, that is, combining natural language processing and data mining technology to detect and confirm rumour events for suppressing or eliminating them, as well as we can analyse the public views and opinions on the hot events or breaking news to provide suggestions for administrators.

Footnotes

Acknowledgements

The authors are grateful for the helpful suggestions from the reviewers. J.M. and Y.L. contributed equally to this work.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

This research was supported by the National Natural Science Foundation of China (Grant No.: 71373123) and the Fundamental Research Funds for the Central Universities (Grant No.: NW2018004).

ORCID iD

Yongcong Luo

References

Zhao

Wang

Huang

, et al. Sentiment contagion in complex networks. Physica A: Stat Mech Appl 2014; 394: 17–23.

Chen

Shen

Chen

, et al. A new kinetic model to discuss the control of panic spreading in emergency. Physica A: Stat Mech Appl 2015; 417: 345–357.

Sugimoto

Nomura

Tsubokura

, et al. The relationship between media consumption and health-related anxieties after the Fukushima Daiichi nuclear disaster. PLoS ONE 2013; 8: 1–7.

Jiang

Chen

Liu

KJR

. Graphical evolutionary game for information diffusion over social networks. IEEE J Select Topics Signal Pr 2014; 8(4): 524–536.

Jiang

Chen

Liu

KJR

. Evolutionary dynamics of information diffusion over social networks. IEEE T Signal Pr 2014; 62(17): 4573–4586.

Zubiaga

Liakata

Procter

, et al. Analysing how people orient to and spread rumours in social media by looking at conversational threads. PLoS ONE 2016; 11: 1–29.

Zhao

Wang

Cheng

, et al. The impact of authorities’ media and rumor dissemination on the evolution of emergency. Physica A: Stat Mech Appl 2012; 391(15): 3978–3987.

Rahu

Health effects of the Chernobyl accident: fears, rumours and the truth. Eur J Cancer 2003; 39(3): 295–299.

Huo

Huang

Fang

An interplay model for authorities’ actions and rumor spreading in emergency event. Physica A: Stat Mech Appl 2011; 390(20): 3267–3274.

10.

Zhu

Zhao

Wang

Complex dynamic behavior of a rumor propagation model with spatial-temporal diffusion terms. Inform Sci 2016; 349–350: 119–136.

11.

Huo

The interaction evolution model of mass incidents with delay in a social network. Physica A: Stat Mech Appl 2017; 484: 440–452.

12.

Luo

The influence of positive news on the rumor spreading in social networks with scale-free characteristics. Int J Modern Phys C 2018; 29(09): 1850078.

13.

Zubiaga

Aker

Bontcheva

, et al. Detection and resolution of rumours in social media: a survey. ACM Comput Surv 2018; 51(2): 32.

14.

Qazvinian

Rosengren

Radev

, et al. Rumor has it: identifying misinformation in microblogs. In: Proceedings of the conference on empirical methods in natural language processing, Edinburgh, 27–31 July 2011, pp.1589–1599. New York: ACM.

15.

Hamidian

Diab

. Rumor identification and belief investigation on twitter. In: Proceedings of the conference of the North American chapter of the association for computational linguistics: human language technologies, NAACL-HLT, San Diego, CA, 3–5 June 2016, pp.3–8. Cambridge: MIT Press.

16.

Zeng

Starbird

Spiro

#Unconfirmed: classifying rumor stance in crisis-related social media messages. In: Proceedings of the international AAAI conference on web and social media, Cologen, 17–20 May 2016, pp.747–750. Palo Alto: AAAI.

17.

Chen

Zhang

Yeo

, et al. Unsupervised rumor detection based on users’ behaviors using neural networks. Pattern Recogn Lett 2018; 105: 226–233.

18.

Mendoza

Poblete

Castillo

Twitter under crisis: can we trust what we RT? In:

Proceedings first workshop on social media analytics of the, New York, 25–28 July 2010, pp.71–79. New York: ACM.

19.

Liu

Nourbakhsh

, et al. Real-time rumor debunking on Twitter. In: Proceedings of the 24th ACM international on conference on information and knowledge management, CIKM, New York, 2015, pp.1867–1870. New York: ACM.

20.

Oostdijk

van Halteren

. Shallow parsing for recognizing threats in Dutch tweets. In: Proceedings of the 2013 IEEE/ACM international conference on advances in social networks analysis and mining, Niagara, ON, Canada, pp.1034–1041. New York: IEEE.

21.

Celebi

Ozgur

Segmenting hashtags and analyzing their grammatical structure. J Assoc Inform Sci Technol 2018; 69(5): 675–686.

22.

Cerri

Pappa

Carvalho

ACP

, et al. An extensive evaluation of decision tree-based hierarchical multilabel classification methods and performance measures. Comput Intel 2015; 31(1): 1–46.

23.

Amin

Recognition of printed Arabic text based on global features and decision tree learning techniques. Pattern Recogn 2000; 33(8): 1309–1323.

24.

Simm

Magrans

Abril

Sugiyama

Tree-based ensemble multi-task learning method for classification and regression. IEICE T Inform Syst 2014; E97D(6): 1677–1681.

25.

Schulz

Menca

Schmidt

A rapid-prototyping framework for extracting small-scale incident-related information in microblogs: application of multi-label classification on tweets. Inform Syst 2016; 57: 88–110.

26.

Zubiaga

Voss

Procter

, et al. Towards real-time, country-level location classification of worldwide tweets. IEEE T Knowl Data Eng 2017; 29(9): 2053–2066.

27.

Liu

Cheng

, et al. TASC: topic-adaptive sentiment classification on dynamic tweets. IEEE T Knowl Data Eng 2015; 27(6): 1696–1709.

28.

Luo

Osborne

, et al. Structuring tweets for improving Twitter search. Assoc Inform Sci Tech 2015; 66(12): 2522–2539.

29.

Wang

Shou

Chen

, et al. On summarization and timeline generation for evolutionary tweet streams. IEEE T Knowl Data Eng 2015; 27(5): 1301–1315.

30.

Zhang

, et al. Chinese comments sentiment classification based on Word2vec and SVMperf. Expert Syst Appl 2015; 42(4): 1857–1863.

31.

Chen

CH.

Improved TFIDF in big news retrieval: an empirical study. Pattern Recogn Lett 2017; 93: 113–122.

32.

Jones

KS.

A statistical interpretation of term specificity and its application in retrieval. J Document 2004; 60(5): 493–502.

33.

Robertson

Understanding inverse document frequency: on theoretical arguments for IDF. J Document 2004; 60(5): 503–520.

34.

Zubiaga

Liakata

Procter

, et al. PHEME rumour scheme dataset: journalism use case, 2016, https://figshare.com/articles/PHEME_rumour_scheme_dataset_journalism_use_case/2068650 (accessed 5 February 2019).

35.

Zubiaga

Liakata

Procter

. Crowdsourcing the annotation of rumourous conversations in social media. In: Proceedings of the 24th international conference on World Wide Web (WWW), Florence, 18–22 May 2015, pp.347–353. New York: ACM.

36.

Newman

The structure and function of complex networks. SIAM Rev 2003; 45(2): 167–256.

37.

, et al. A domain keyword analysis approach extending term frequency-keyword active index with Google Word2vec model. Scientometrics 2018; 114(3): 1031–1068.

38.

Quinlan

Induction of decision trees. Mach Learn 1986; 1(1): 81–106.

39.

Quinlan

. C4.5: programs for machine learning. San Francisco, CA: Morgan Kaufmann Publishers Inc., 1993.

40.

Breiman

Friedman

Olshen

, et al. Classification and regression trees. Boca Raton, FL: Chapman and Hall/CRC Press, 1984.

41.

Geurts

Ernst

Wehenkel

Extremely randomized trees. Mach Learn 2006; 63(1): 3–42.

42.

Breiman

Random forests. Mach Learn 2001; 45(1): 5–32.

43.

Freund

Schapire

RE.

A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 1997; 55(1): 119–139.