Aspect-level sentiment analysis for based on joint aspect and position hierarchy attention mechanism network

Abstract

The purpose of aspect-level sentiment analysis is to identify the contextual sentence expressions given by sentiment for some aspects. For previous works, many scholars have proved the importance of the interaction between aspects and contexts. However, most existing methods ignore or do not specifically capture the position information of the aspect targets in the sentence. Thus, we propose an aspect-level sentiment analysis based on joint aspect and position hierarchy attention mechanism network. At the same time, the model adopts a joint approach to make the model of the aspect features and the position features. On the one hand, this method clearly captures the interaction between aspect words and context when inputting word vector information. On the other hand, this method can enhance the importance of position information in the sentence and boost the information retrieval ability of the model. Additionally, the model utilizes a hierarchical attention mechanism to extract feature information and to differentiate sentiment towards, which is similar to filtering useless information again. Experiment on the SemEval 2014 dataset represent that our model achieves better performance on aspect-level sentiment classification.

Keywords

Aspect-level position information hierarchy attention mechanism sentiment analysis sentiment polarity

1 Introduction

With the development of technology, online reviews have become a way for consumers to evaluate goods. Sentiment analysis has not only highly attention from domestic and foreign scholars in academic circles, but also has been favored by industry. In Natural Language Processing (NLP), aspect-based sentiment analysis is a fine-grained classification task, which can infer to two and more sentiment polarities according to the aspects in a given sentence. For example, given a restaurant review “while there’s a decent menu, it shouldn’t take ten minutes to get your drinks, and 45 minutes to get a dessert pizza”. We need to extract the three aspects of “menu”, “drinks” and “dessert pizza” from the sentence, and also be able to identify the sentiment polarity corresponding to the aspect terms in the model. In this review, the sentiment polarities of “menu”, “drinks” and “dessert pizza” are different, and their corresponding sentient polarities are positive and neutral respectively.

A large number of methods have been proposed for fine-grained classification tasks, and the effectiveness of these methods has been verified. In traditional methods, the bag-of-words model and TF-IDF are mainly used as features. Support vector machines (SVM) [26] and k-nearest neighbor (KNN) are used as classifiers to achieve classification tasks [29]. Although traditional methods are easy to implement and have a small amount of calculation, they are not suitable for highly complex data. In addition, if scholars want to obtain a robust model, they must manually construct features, which inevitably wastes a lot of manpower and financial resources. With the recent development of neural networks, neural network-based methods have also been used in aspect-level sentiment analysis tasks, such as Long short-term memory (LSTM) [23], Convolutional Networks [30] and Target-Connection LSTM (TC-LSTM) [7]. However, the above methods cannot effectively recognize the importance of words in a given sentence, leading to poor final results. But the attention mechanism effectively overcomes their shortcomings.

The attention mechanism was originally used in image recognition [18], machine translation and reading comprehension [4, 18] to extract local key features. Later, many scholars discovered that the attention mechanism can also be used to extract important features in the text. Compared with the model without attention mechanism, the effect of the model using the attention mechanism has been greatly improved. Therefore, a large number of scholars have proposed models based on attention mechanisms in fine-grained sentiment analysis tasks, such as AE-LSTM [32], ATAE-LSTM [32] and IAN [5].

As mentioned above, many models ignore the importance of location information to aspect terms. In order to solve this problem, we have added location information features to the model. At the same time, existing research shows that the interaction between aspect and context is very important for aspect-level sentiment analysis. By taking advantage of this idea, a joint aspect and position hierarchy attention mechanism is also adopted to the model. For instance, a sentence from the restaurant dataset that “this particular location certainly uses substandard meats”. After adding the position index to the model, the index of “meat” is regarded as “0”, which makes the model more accurate in terms of word extraction. If there is no location information, the model can extract both “specific location” and “meat” as aspect terms. In addition, the location information is used to classify the aspect word sentiment polarity, which also makes the network forget useless information when extracting features. Therefore, we make the model of the position information and obtained the feature representation of the position information. Since position information can be regarded as a feature in the sentence, which helps the model to better learn the important information in the sentence. At the same time, the model uses the position index to better extract the aspect words and identify the corresponding emotional polarity.

Eventually, the results of various baseline models are reproduced on the SemEval-2014 dataset and compare these results with joint modeling of aspect and position hierarchical attention mechanism. The results of comparing our model with many baseline models reveal that our model performs better than the baseline model.

In summary, the contribution of our proposed model can be summarized in the following perspectives:

The position information of the aspect words is effectively added to the model, and the Bi-GRU network is adopted to better capture the important part of the context. GRU network not only has lower computational complexity than LSTM network, but also has fewer network parameters.

A joint approach is utilized for aspect features and position features in the model, and a simple hierarchical attention mechanism is used to optimize the vector information in aspect-level sentiment analysis, which is conducive to the accurate extraction of the network.

In order to optimize all parameters to prevent overfitting, the model is trained in a supervised learning framework through an end-to-end manner.

In this paper, the remaining parts are as follows. Specifically, the second part is relevant work of the article. The third part is a detailed introduction to the model. The fourth part is to interpret the experiment of the model and have a better understanding of our proposed model through the case analysis. The fifth section is the conclusion of the article.

2 Related work

The whole task of aspect-based sentiment analysis consists of several subtasks such as aspect extraction, entity extraction, and aspect sentiment classification [16]. In this part, we briefly introduce the development of sentiment analysis research methods in recent years. The research methods can be segmented into two aspects, including the traditional machine learning methods and the neural network methods.

Traditional machine learning methods are widely used in early sentiment analysis tasks. This method is mainly based on artificial feature engineering to realize sentiment classification. Many scholars usually use sentiment dictionaries to extract features, and use SVM [14], Bayes [9] and other classifiers to identify the sentiment direction corresponding to aspect words. Target-dependent [15] proposed statistic methods, which largely depended on the effectiveness of feature engineering works. Dcu [12] used the SVM classifier to distinguish the emotional polarity of the aspect words from the various features extracted by manual design. Nevertheless, traditional machine learning methods mentioned above are labor-intensive and prone to lead to high -dimensional and sparse text representation [22].

Therefore, many models based on neural network have been put forward in recent years. Recurrent Neural Networks (RNNs) such as LSTM [1] and GRU [10] have been used for sentiment analysis on data instances having variable length [6, 7]. Many tasks are also based on the convolutional neural networks (CNNs) [30] for feature extraction in NLP, which also confirmed that convolution operations can capture significant information in the text, and does not require a lot of labor to manually feature. At present, aspect-based sentiment analysis studies pay more attention to the fusion of target words and contextual words. TC-LSTM [7] take target information into account and achieve good results in target dependent sentiment classification. AT-LSTM [32] and ATAE-LSTM [32] input different target adopted the focus of attention mechanism in contextual words. ATAE-LSTM [6] and IAN [5] adopt two separate LSTM layers and an interactive attention mechanism. MemNet [32], RAM [20]and IARM [19] in sentence level memory into the deep memory network and multiple-hops attention model, integrating target information to the characterization of the sentence. CEA [13] proposed a unified model that leverages and integrates multiple knowledge sources to enhance its capabilities on ABSA. AF-LSTM [31] obtained a joint representation by independently learning between the context and aspect relationship, which joint attention mechanism to learn the relative importance of the context. GCAE [28] and PG-CNN [2] are based on CNN models, gate mechanism is used to realize the interaction between the target with the context token. Unified Generative Framework [8] defines the subtask target as a sequence of pointer index and emotion index, and adopts a unified framework for subtasks to handle multiple subtasks. SENTA [33] resolves confounding factors such as non-target aspects by applying backdoor adjustments. DCRAN [25] proposes the interactive relationship between the explicit self-supervision strategy and the subtasks of deep contextual information. MTKFN-Senti [24] proposed a unified model that leverages and integrates multiple knowledge sources to enhance its capabilities on ABSA. ATAE-LSTM(AA) [3] and AA-LSTM [3] are novel variant of LSTM, which merged the aspect information into the LSTM unit in the context modeling stage before the attention mechanism. SPAN-collapsed [17] learned the deep associations between sentences and aspects and the long-term dependencies within sentences by using the pre-trained Bert model. HAM-GLOVE [1] proposed a hierarchical attention-based approach for aspect-based polarity classification of text.

In spite of the above models having achieved certain results, it is still a challenge to distinguish different emotional polarities in fine-grained terms. Therefore, we propose a hierarchical attention mechanism model that combines aspect terms and location information. This model not only verifies the importance of aspect terminology’s location information in aspect hierarchical sentiment analysis, but also uses a hierarchical attention mechanism. The enhancement vector is regarded as the necessity of the object.

3 Methods

Although most of the researches on aspect-based sentiment analysis that rely on aspects have achieved certain results, these studies ignore the importance of the position index of the aspect word in the sentence. To solve this problem, the JAP-HAM model adds location information to enhance the characteristics of important information. At the same time, this paper also uses the Bi-GRU network to obtain the hidden units of the position vector and the aspect word vector, so that their hidden units can be combined to obtain new features through the LSTM network for feature extraction. Finally, the model uses the attention mechanism to obtain aspect words and their corresponding emotional polarity, and an end-to-end method is adopted to prevent the model from overfitting.

In this section, we introduce detail the Aspect-level sentiment analysis based on joint aspect and position hierarchy attention mechanism network (JAP-HAM). As shown in Fig. 1, the JAP-HAM model consists of input model, feature extraction module, attention mechanism module and training loss function model.

Fig. 1

The JAP-HAM model.

3.1 Word embedding

In order to categorize the sentiment towards of the aspect in the sentence. In the model, the 300-dimensional GloVe [10] vector is used as a pre-training language model to acquire the representation of each word in the sentence. For a given sentence, the text c is composed of n words, $w^{c} = {w_{c}^{1}, w_{c}^{2}, w_{c}^{3}, \dots, w_{c}^{n}}$ , the aspect term contains k words, $w^{t} = {w_{1}^{t}, w_{2}^{t}, w_{3}^{t}, \dots, w_{k}^{t}}$ . Let $L \in ℝ^{d_{emb} \times | V |}$ to be the pre-trained GloVe embedding matrix, where d_emb is the dimension of word vectors and |V| is the vocabulary size. And the vector corresponding to each word can be obtained through M^V×dw. Then, we map into a low-dimensional real-value vector for each word [27]. Finally, the vectors corresponding to the sentence and the word vectors corresponding to the aspect word are obtained through the embedded query.

3.2 Position embedding

In order to reduce the interference of information irrelevant to the aspect term, and the weight of other words far away from the aspect term is reduced more quickly. Inspired by [22], we introduced positional embedding. For a given sentence, the position of the aspect word in the sentence is marked as “0”, and the corresponding position sequence is obtained according to the distance between the aspect words and other words in the given sentence. The formula is as follows: $p_{i} = {\begin{matrix} | i - j_{s} |, i < j_{s} \\ 0, j_{s} \leq i \leq j_{e} \\ | i - j_{e} |, i > j_{e} \end{matrix}$ (1) where p_i represents the relative distance of the i-th word for the aspect term in the sentence, j_s deotes the starting position of aspect term and j_e indicates the ending position of aspect term in the sentence by indexing. As an example, “then after 4 or so months the charger stopped working so I was forced to go out and buy new hardware just to keep this computer running.” in a sentence, p={7,6,5,4,3,2,1,0,1,2,3, . . . , 19}. From this sentence, we can see that the aspect word is “charger”, so the position index of “charge” is “0”, and then the position information is indexed to the left and right sides with the aspect word as the center. And when we observe the sentiment polarity corresponding to the aspect word “charge”, we first look at the surrounding words of “charge”, we can see that the index position of “stopped work” is “1”, and “stopped work” is opposite to “charge” played an important role. After that, we proceeded to observe whether more distant words have an effect on aspect words. Therefore, the words of the adjacent aspect words play an increasingly important role in judging the emotional orientation. Thus, it illustrates the significance of position information by this example.

3.3 Bi-GRU feature representation

In the traditional RNN, since the activation function continuously accumulates and expands during the back propagation process, it is easy to give rise to the gradient to explode and disappear. To solve this trouble, we proposed Bi-GRU to extract features.

Firstly, aspect word vectors are embedded into the left Bi-GRU network to obtain the hidden context representation, which is composed of forward hidden state $\vec{h_{i}^{t}} \in ℝ^{d_{h}}$ and backward hidden state $\overset{\leftarrow}{h_{i}^{t}} \in ℝ^{d_{h}}$ , where d_h denotes the number of hidden units. Secondly, the position vectors are embedded into the right Bi-GRU to receive the hidden context representation with rich information. Then, the hidden contextual representation of aspects term $h_{i}^{t}$ is obtained by concatenating $\vec{h_{i}^{t}}$ and $\overset{\leftarrow}{h_{i}^{t}}$ . In addition, we fusion aspect hidden contextual representation and position hidden representation $\vec{h_{i}^{p}}$ and $\overset{\leftarrow}{h_{i}^{p}}$ . After obtaining the hidden unit of location information and the hidden unit of aspect words, we obtained a hidden unit with location information and aspect words in the form of splicing the two hidden units. At last, the model averages the hidden units with aspect information and aspect words to obtain m auxiliary information vectors. The auxiliary information refers to hidden unit vectors with position information and aspect word information. The formula is as follows: $\vec{h_{t}} = \vec{GRU} ([t_{1}; t_{2}; t_{3}; \dots; t_{k}])$ (2) $\overset{\leftarrow}{h_{t}} = \overset{\leftarrow}{GRU} ([t_{1}; t_{2}; t_{3}; \dots; t_{k}])$ (3) $h_{t} = [\vec{h_{t}}, \overset{\leftarrow}{h_{t}}]$ (4) $v_{a} = \sum_{i = 1}^{m} \frac{h_{i}^{t} \oplus h_{i}^{p}}{m}$ (5) where ⊕ denotes the fusion of the hidden layer features of the aspect word and the hidden layer features of the location information, m indicates the fusion dimension.

3.4 LSTM feature representation

This model combines auxiliary information vector and word vector as the input of LSTM network. When the information feature is extracted by the LSTM network, the LSTM network obtains the unit status and updates the h_t by looking at the hidden unit and the input information at the previous moment. For example, “Restaurant service is very good.”. When this sentence vector is input to the LSTM network, the sigmoid unit in the forgetting gate determines what information the cell state needs to discard or retain. The aspect term here may be restaurant and service. According to the location index, it be easier to find that “service” is an aspect term. The formula is as follows. $X = [\begin{matrix} h_{t - 1} \\ x_{t} \end{matrix}]$ (6) $f_{t} = σ (W_{f} \cdot X + b_{f})$ (7) $i_{t} = σ (W_{i} \cdot X + b_{i})$ (8) $o_{t} = σ (W_{o} \cdot X + b_{o})$ (9) $C_{t} = f_{t} ⊙ \tanh (W_{c} \cdot X + b_{c})$ (10) $h_{t} = o_{t} ⊙ \tanh (C_{t})$ (11) where W_f represents the weight matrix of the forgetting gate, b_f is the bias term of the forgetting gate. σ is the sigmoid function, $[\begin{matrix} h_{t - 1} \\ x_{t} \end{matrix}]$ means to connect two vectors into a longer vector. W_i represents the weight matrix of the input gate, b_i is the bias term of the input gate. ⊙ stands for element-wise multiplication and C_t stands for cell state.

3.5 Attention mechanism

The LSTM network cannot well identify the emotional polarity corresponding to the aspect words in the sentence. In order to solve this problem, we adopt an attention mechanism to capture the important information in the sentence. As can be seen in Fig. 1, H is the matrix output by the LSTM network, which is a matrix composed of N hidden vectors. v_a represents the embedding of aspect words and location information. $e_{N} \in ℝ^{N}$ is a 1 s vector. The attention mechanism is represented by an attention weight vector α and a weighted hidden vector H. The formula is as follows. $M = tanh (\begin{matrix} W_{h} H \\ W_{v} v_{a} \otimes e_{N} \end{matrix})$ (12) $α = softmax (w^{T} M)$ (13) $γ = H α^{T}$ (14) where M ∈ R^(d+d_a)×N, α ∈ R^N, γ ∈ R^d, W_h ∈ R^d×d, W_v ∈ R^d_a×d_a and w ∈ R^d+d_a are projection parameters. α is a vector consisting of attention weights and γ is a weighted representation of sentence with given aspects. The operator in 7(a circle with a multiplication sign inside, OP for short here) means: v_a ⊗ e_N = [v ; v ; ⋯ ; v], that is, the operator repeatedly concatenates v for N times, where e_N is a column vector with N 1 s. W_vv_a ⊗ e_N is repeating the linearly transformed v_a as many times as there are words in sentence.

The final sentence representation is given by: $h^{*} = tanh (W_{p} γ + W_{x} h_{N})$ (15) where h^* ∈ R^d, W_p and W_x are project parameters to be learned during training. We found that if the W_xh_N added to the final representation of the sentence, it will be better. h^* is considered as the feature representation of a sentence given an input aspect. We add a linear layer converts sentence vector e, e is the number of length equal to class |C| the real value of the vector. Then, a softmax layer is followed to transform e to conditional probability distribution. $y = softmax (W_{s} h^{*} + b_{s})$ (16) where W_s and b_s are the parameters for softmax layer.

3.6 Model training

Although the model can predict the overall samples in the training data set well, the prediction results in the test set samples are very poor. This prediction result shows that the generalization ability of the model is not good. Such a problem may be that the model is too complex and leads to overfitting. However, in practice, we cannot well control the number of layers and neurons in the network well. Therefore, we use the neural network L2 regularization and dropout method to prevent the network from overfitting.

The L2 regularization model is also called ridge regression. In the neural network, we have added a regular term, which can effectively prevent the weight matrix from being too large. If the regularization value is large enough, the weight matrix is close to zero. In this way, the influence of some hidden units can be eliminated, and the neural network is also simplified to a certain extent, turning a complex network into a very small network.

The dropout method prevents the network from overfitting by modifying the number of neurons in the hidden layer. When training each batch of data, dropout randomly removes some neurons according to the given probability P, and only updates the parameters of the neurons that are not removed or retained. The random removal of neurons makes the network have a certain degree of sparsity, which can reduce the synergistic effect between different features. Because the neurons eliminated each time are different, the parameters of the entire network neurons are only partially updated. This can eliminate the weakening of the joint adaptability between neurons, but enhance the generalization ability and robustness of the neural network.

And the cross-entropy is utilized as a loss function to optimize all parameters in an end-to-end manner. The formula is as follows. $loss = - \sum_{i = 1}^{S} y_{i} \log (\hat{y_{i}}) + \frac{1}{2} λ ∥ θ ∥^{2}$ (17) where λ is the regularization factor and θ contains all the parameters, y_i represents the valid sentiment towards, and $\hat{y_{i}}$ means to predict sentiment towards of a given sentence.

4 Experiments and discussion

4.1 Dataset

The experiment adopts SemEval 2014 datasets to effectively verify the model. The dataset contains laptop reviews and restaurant reviews. Each review contains two or more aspect terms, and each aspect term has a corresponding sentiment polarity. The sentiment polarity in this experiment including three parts {positive, neutral and negative}. The training, verification and testing samples for each sentiment polarities are shown in Table 1.

Table 1
The detailed data from Semeval 2014 DataSets

Dataset Positive Neutral Negative

Laptop-Train 767 373 673

Laptop-Dev 220 87 193

Laptop-Test 341 169 128

Restaurant-Train 1886 531 685

Restaurant-Dev 278 102 120

Restaurant-Test 728 196 196

Dataset	Positive	Neutral	Negative
Laptop-Train	767	373	673
Laptop-Dev	220	87	193
Laptop-Test	341	169	128
Restaurant-Train	1886	531	685
Restaurant-Dev	278	102	120
Restaurant-Test	728	196	196

4.2 Hyperparameters setting

In this experience, the model uses 300-dimension word vectors as pre-trained by GloVe on the SemEval 2014 dataset to initialize. The JAP-HAM model adopts randomly initialize from uniform distribution U (- 0.1, 0.1) to obtain the initial value of all weight matrices, and biases are all set to 0. The dimensions of word embedding and aspect embedding are both set to 300. The size of the hidden state is set to 200, and the size of the position embedding is set to 100. In the experiment, we use Pytorch as the model framework. The initial learning rate of the Adam optimizer is 0.01, the batch is size to 64 and the dropout rate is set to 0.1.

4.3 Evaluation metric

In order to assess the effect of the JAP-HAM model. We use Accuracy and Macro-F1 for evaluation, which is defined as: $Acc = \frac{TP + TN}{TP + TN + FP + FN}$ (18) $F 1 = \frac{2 * Precision * Recall}{Precision + Recall}$ (19) $Macro - F 1 = \frac{1}{3} \sum F 1$ (20) where TP is the correct number of positive predictions, TN is the correct number of negative predictions, FN is the number of positive prediction errors and FP is the number of negative prediction errors. Accuracy designates the percentage of all samples that are correctly predicted. Precision is the ratio of the number of correctly classified samples to the total number of samples for given datasets. Recall is used to describe the ratio of positive examples judged to be true in the classifier to the total positive examples. F1 value is the harmonic meaning of precision and recall. Marco -F1 indicates calculating F1 after calculating the Precision and Recall of each class, and finally averaging F1. In general, good performance of a system has higher precision.

4.4 Model comparisons

In order to testify to performance availability of the JAP-HAM model by accuracy and Macro F1 value are used as the criteria to measure performance. At the same time, we also compare with several baseline models, including LSTM, CNN, TC-LSTM, AT-LSTM, ATAE-LSTM, ATAE-Bi-LSTM, MemNet, IAN, RAM, AF-LSTM(CORR), AF-LSTM(CONV), GCAE, DAuM, IARM, CEA, MTKFN-Senti, AA-LSTM, ATAE-LSTM(AA), PG-CNN and HAM-GLOVE. And we show in Tables 2 3.

Table 2
Comparison of the performance on restaurant data sets

Methods Year Restaurant-Accuracy Restaurant-Macro F1

LSTM [23] 1997 74.30 63.00

CNN [30] 2014 75.18 60.25

TC-LSTM [7] 2016 77.41 66.72

AT-LSTM [32] 2016 78.04 63.37

ATAE-LSTM [32] 2016 76.79 63.72

ATAE-Bi-LSTM [32] 2016 75.98 63.43

MemNet [7] 2016 73.39 61.09

IAN [5] 2017 76.70 65.12

RAM [20] 2017 77.41 66.76

AF-LSTM(CORR) [31] 2018 75.96 64.00

AF-LSTM(CONV) [31] 2018 76.46 65.54

GCAE [28] 2018 77.41 65.06

DAuM [21] 2018 77.91 66.47

IARM [19] 2018 77.73 66.66

CEA [13] 2018 78.44 66.78

MTKFN-Senti [24] 2019 77.74 66.30

AA-LSTM [3] 2019 78.21 66.24

ATAE-LSTM(AA) [3] 2019 78.31 66.46

PG-CNN [2] 2019 78.90 –

SPAN-collapsed [17] 2019 – 57.85

HAM-GLOVE [1] 2021 76.61 64.96

JAP-HAM 78.125 67.86

Methods	Year	Restaurant-Accuracy	Restaurant-Macro F1
LSTM [23]	1997	74.30	63.00
CNN [30]	2014	75.18	60.25
TC-LSTM [7]	2016	77.41	66.72
AT-LSTM [32]	2016	78.04	63.37
ATAE-LSTM [32]	2016	76.79	63.72
ATAE-Bi-LSTM [32]	2016	75.98	63.43
MemNet [7]	2016	73.39	61.09
IAN [5]	2017	76.70	65.12
RAM [20]	2017	77.41	66.76
AF-LSTM(CORR) [31]	2018	75.96	64.00
AF-LSTM(CONV) [31]	2018	76.46	65.54
GCAE [28]	2018	77.41	65.06
DAuM [21]	2018	77.91	66.47
IARM [19]	2018	77.73	66.66
CEA [13]	2018	78.44	66.78
MTKFN-Senti [24]	2019	77.74	66.30
AA-LSTM [3]	2019	78.21	66.24
ATAE-LSTM(AA) [3]	2019	78.31	66.46
PG-CNN [2]	2019	78.90	–
SPAN-collapsed [17]	2019	–	57.85
HAM-GLOVE [1]	2021	76.61	64.96
JAP-HAM		78.125	67.86

Table 3

Comparison of the performance on laptop data sets

Methods	Year	Laptop-Accuracy	Laptop-Macro F1
LSTM [23]	1997	66.50	60.10
CNN [30]	2014	66.93	57.75
TC-LSTM [7]	2016	67.08	61.11
AT-LSTM [32]	2016	69.44	63.16
ATAE-LSTM [32]	2016	67.40	58.47
ATAE-Bi-LSTM [32]	2016	70.53	63.43
MemNet [7]	2016	64.42	58.10
IAN [5]	2017	68.50	60.90
RAM [20]	2017	67.55	59.73
AF-LSTM(CORR) [31]	2018	69.97	63.70
AF-LSTM(CONV) [31]	2018	69.78	63.38
GCAE [27]	2018	65.83	59.20
DAuM [21]	2018	70.36	65.06
IARM [19]	2018	68.63	63.30
CEA [13]	2018	70.52	64.52
MTKFN-Senti [24]	2019	69.55	62.96
AA-LSTM [3]	2019	66.93	61.45
ATAE-LSTM(AA) [3]		69.28	62.10
PG-CNN [2]	2019	69.10	–
SPAN-collapsed [17]	2019	–	48.66
HAM-GLOVE [1]	2021	69.59	64.02
JAP-HAM		70.846	65.777

LSTM networks are good at capturing remote information dependencies. It is widely used to generate sentence context and semantic expression. However, this model cannot capture information related to a given aspect in a sentence.

CNN: The input of CNN is an independent sentence, and the model does not have the combination of aspect information and attention mechanism, which hides the inherent dependence of the inability to obtain text. Moreover, the model loses structural information in the global max pooling, and it is difficult to find complex patterns such as transition relations in the text.

TC-LSTM: Firstly, TC-LSTM averages the vector representation of the target word and the spliced word vector. Then, the model uses two LSTMs to model the information on the left and right sides respectively. Finally, the model predicts the emotional polarity of the acquired information.

AT-LSTM adds an attention mechanism on the basis of LSTM to capture the important part of the sentence as a specific aspect. The representation of the sentence is obtained through the attention mechanism and the final sentiment classification is obtained.

ATAE-LSTM makes better use of the aspect word information. The model uses the word vector of the aspect term and the average vector of the sentence word vector as the input layer of LSTM. The output of the LSTM network is combined with the word vector as the input of the attention mechanism. Finally, the model focuses on multiplying the output vector with the attention weight to obtain the final vector representation of the sentence and perform sentiment classification.

MemNet first constructs external memory through text word vectors. Then the model uses the aspect sentiment analysis of the attention mechanism. Finally, the model calculates the attention expression of each layer as the input of the next layer.

IAN: The word vector of a specific aspect item and its context vector are respectively used as the input of two different LSTM networks. The hidden layer input of LSTM is combined with the attention mechanism to obtain the feature vector representation of the final aspect and context. Eventually, the softmax classifier is used for classification.

RAM uses a multiple-attention mechanism to capture emotional features that are far apart. This method is more robust to irrelevant information.

GCAE proposes a model based on CNN and gating mechanism. First, the new Gated Tanh-ReLU unit can have a choice of output emotional features through a given aspect. Second, the model can be trained in parallel and the speed is faster.

DAuM uses auxiliary memory to implicitly generate aspect or terminology. And through the interaction between the two memories, the aspect words can provide more general terminology.

IARM utilizes a cyclic memory network with a multi-hop attention mechanism. At the same time, it shows that one aspect of the sentence is affected by its neighboring aspects.

CEA proposes memory modeling of contexts, entities, and aspects. And it has been further improved through the dependency tree.

MTKFN-Senti uses structural and emotional knowledge and integrates it into the model, which can improve its ability to capture important context words. And identify the emotional polarity represented by the aspect-specific context.

AA-LSTM can select important information for a given target. And the model filters out useless information through information flow control. Aspect-aware LSTM can not only generate more effective context vectors than classic LSTM, but it is also compatible with other modules.

PG-CNN introduces parameterized filters and parameterized gates to integrate aspect information into the architecture of CNN.

SPAN-collapsed can determine the beginning and end positions of a summary of opinions on a given input. By using the pre-trained Bert model, the deep associations between sentences and aspects and the long-term dependencies in sentences can be considered.

HAM-GLOVE proposes a hierarchical attention model (HAM) to address specific aspects of emotional polarity.

The effectiveness of the model in the data set is evaluated using accuracy and macro f1 values and compared with a series of baseline models. We can be seen from the Tables 2 3, the performance of our proposed model is the best model, and the performance of the two baseline models of LSTM and CNN are the worst model.

Since LSTM neural network ignores the importance of aspect terms, its accuracy is the lowest in whole baseline models. Although the CNN model uses different convolution kernels to extract information, its accuracy is 0.88%and 0.43%higher than that of the LSTM network in different data sets. However, a large amount of important information and merging operations will be lost during the convolution process, resulting in poor performance of the CNN model. TC-LSTM, AT-LSTM, ATAE-LSTM and ATAE-Bi-LSTM are derived models of LSTM. Compared with the LSTM model, the TC-LSTM network strengthens the interaction between aspects and sentences, and the accuracy of the two data sets is increased by 3.11%and 0.58%, respectively. Because ATAE-LSTM considers the attention mechanism, it is better than TC-LSTM in the accuracy of sentiment classification. In order to better capture semantic information and increase the accuracy of feature extraction, the ATAE-LSTM model is improved to a left-to-right and right-to-left bidirectional LSTM model. MemNet, IAN, and RAM models all consider the modeling of aspect words and use the attention mechanism to pay attention to important information. It can be seen from the above table that the use of aspect words and attention mechanism has a certain impact on the improvement of model performance.

Obviously, the attention mechanism enhances the performance of aspect-level sentiment analysis tasks to a certain extent, so the GCAE, DAuM, IARE and CEA models continue to improve the memory module, but their performance is lower than our model. Although the PG-CNN model uses parameterized filters and parameterized gates to effectively capture specific aspects of the characteristics, the CNN network loses information due to the reason that the performance is not as good as our model. It can be seen from Tables 2 3 that the accuracy of the PG-CNN network in the restaurant dataset is better than our model. However, it can be seen that our model is better than the PG-CNN model from the results of the laptop dataset and the F1 value of the restaurant. SPAN-collapsed model uses the Bert model for pre-training, which considers the deep associations between sentences and aspects and the long-term dependencies in sentences, but its effect is still inferior to our model. Because the HAM-GLOVE model ignores the importance of the location information corresponding to the aspect words. Compared with the HAM-GLOVE model, our model has improved accuracy by 1.515%and 1.256%in the two data and the F1 value has also increased 2.9%and 1.757%. In short, the JAP-HAM model uses a joint method to obtain features for the position hidden state and aspect word hidden state, and utilizes a hierarchical attention mechanism to strengthen the weight of important information. It is obvious that the JAP-HAM model has better performance. This method can effectively estimate the emotion of diverse aspects of the corresponding sentence and improve the accuracy of classification.

4.5 Analysis of model

In this part, several models are utilized to prove the availability of the JAP-HAM model. First of all, through the analysis of the ATAE-Bi-LSTM model and the ATAE-LSTM model, because ATAE-Bi-LSTM is a derivative model of ATAE-LSTM, and the merely distinction between the two models is that the ATAE-Bi-LSTM model utilizes bidirectional LSTMs model, which is more conducive to capturing grammatical information between sentences. In addition, comparing the HAM-LSTM model with the ATAE-LSTM model, it can be seen that the difference between this model and the ATAE-LSTM is that the model uses a hierarchical attention mechanism, so that the model can optimize the vector. The Bi-GRU network is used in the ATAE-Bi-GRU model because the network parameters and calculations are less. Finally, it can be seen from the HAM-Bi-GRU model that the model only uses aspect terms and sentence representations, and the model does not consider the position information of the aspect words. As can be seen from Tables 4, 5 and Figs. 2, 3, among these models, the JAP-HAM model achieves optimum performance.

Table 4
The Accuracy value of different models

Model Restaurant-Acc Laptop-Acc

ATAE-LSTM 76.79 67.40

HAM-LSTM 76.61 69.59

ATAE-Bi-LSTM 75.98 70.53

HAM-Bi-GRU 77.41 66.61

JAP-HAM 78.125 70.846

Model	Restaurant-Acc	Laptop-Acc
ATAE-LSTM	76.79	67.40
HAM-LSTM	76.61	69.59
ATAE-Bi-LSTM	75.98	70.53
HAM-Bi-GRU	77.41	66.61
JAP-HAM	78.125	70.846

Table 5

The Marco F1 value of different models

Model	Restaurant-Marco F1	Laptop-Marco F1
ATAE-LSTM	63.37	58.47
HAM-LSTM	64.96	64.02
ATAE-Bi-LSTM	63.43	63.43
HAM-Bi-GRU	65.10	60.36
JAP-HAM	67.86	65.777

Fig. 2

The Accuracy value of different models.

Fig. 3

The Macro-F1 value of different models.

Since the Bi-LSTM network can obtain more semantic information, the performance of the ATAE-Bi-LSTM model is significantly better than that of the ATAE-LSTM model. For the HAM-LSTM model, the performance of this model is better than the ATAE-LSTM model. At the same time, it can be seen from the model effect that the ATAE-Bi-GRU model is better than the ATAE-Bi-LSTM model. Compared with HAM-Bi-GRU, it is concluded that the JAP-HAM model embeds the position vector to obtain the attention weight, instead of only strengthening the weight of the aspect word vector like the HAM-Bi-GRU model. Thus, based on the results of five experiments, the importance of position information in aspect-level sentiment analysis tasks can be demonstrated.

In summary, the JAP-HAM model is the best among the several experiments. We not only consider the importance of position information to aspect words, but also use the Bi-GRU model to reduce the amount of calculation and parameters, so that the model fulfills the best effect.

4.6 Case analysis

In this part, we use the three comments in the test set as cases to better understand our model. In order to analyze the contribution of words to sentences, we visualize the different importance the sentence attaches to the word in Table 6. In addition, the depth of the color represents the significance of the word in the sentence.

Table 6
Case analysis

Aspect term Sentence Polarity

Audio volume The audio volume is quite low and virtually unusable in a room with any background activity. Negative

Food D $\overset{´}{e}$ cor The food is surprisingly good, and the d $\overset{´}{e}$ cor is nice. Positive Positive

Waiters Food The food did take a few extra minutes to come, but the cute waiters’ jokes and friendliness make up for it. Positive Neutral

Service Food I was highly disappointed by their service and food. Negative Negative

Hard disk I always use a backup hard disk to store important files at all times. Neutral

Dock Program Program I love the dock where I can simply drop a file on top of a program, and the program will simply open that file. Positive Neutral Positive

Aspect term	Sentence	Polarity
Audio volume	The audio volume is quite low and virtually unusable in a room with any background activity.	Negative
Food D $\overset{´}{e}$ cor	The food is surprisingly good, and the d $\overset{´}{e}$ cor is nice.	Positive Positive
Waiters Food	The food did take a few extra minutes to come, but the cute waiters’ jokes and friendliness make up for it.	Positive Neutral
Service Food	I was highly disappointed by their service and food.	Negative Negative
Hard disk	I always use a backup hard disk to store important files at all times.	Neutral
Dock Program Program	I love the dock where I can simply drop a file on top of a program, and the program will simply open that file.	Positive Neutral Positive

As we can see from the first instance, i.e., “the audio volume is quite low and virtually unusable in a room with any background.” According to the model we proposed, in this sentence, we first index the sentence, the index of the aspect word “audio volume” is “0”, and index to the left and right with the aspect word “audio volume” as the center. The result obtained is p={1,0,1,2,..., 12}. At the same time, the aspect words are embedded, and the result is t={0,1,1,0,..., 0}. Then, when the two auxiliary information vectors are obtained, the model uses the auxiliary vector and the sentence vector as the input of the LSTM network. In addition, in the LSTM network, important information is retained for the sentence by forgetting the gate and updating. Finally, the model uses the attention mechanism to add weight to some important words. For example, the model focuses on the word “low” around the aspect word according to the location index, and then focus on the relatively distant word “unavailable”. Using the attention mechanism, the model can well strengthen the weight of important information, and then gradually reduce the weight of other information. Obviously, the two words “low” and “unusable” are very important for the aspect term “audio volume” because they need to infer the emotion of the aspect term. It can be seen from Table 6 that these words all use darker colors to indicate how much the model attaches importance to the words, and the displayed results are consistent with our expected results.

5 Conclusion

Because traditional methods cannot effectively judge aspect information and emotional information. Therefore, this paper proposes a based on joint aspect and position hierarchy attention mechanism network (JAP-HAM). The model uses position information and aspect word information to form auxiliary information, and the important information is extracted in the LSTM network through the combination of auxiliary information vector and word vector. The cell state in the LSTM network is forgotten about the useless information in the sentence, and the important information is retained. At the same time, the model uses a hierarchical attention mechanism to enhance the important parts of the sentence to obtain the emotional polarity of the aspect words. Compared with traditional methods, the model proposed in this paper can effectively learn important information in sentences. At the same time, the JAP-HAM model is visually analyzed using case studies. Finally, the comparative experimental results on the two datasets of restaurant and laptop show that compared with multiple baseline models, the accuracy of our model and the macro-f1 value get better results.

Acknowledgments

This work was supported by the Postdoctoral Science Foundation of China (2016M592894XB), the National Natural Science Foundation of China (61866020), the general program of Yunnan Provincial Department of Science and Technology (2019FB082), and the general program of basic research in Yunnan Province (202001AT070047).

References

Lakizadeh

and Zinaty

, A novel hierarchical attention-based method for aspect-level sentiment classification, Journal of AI and data mining 9 (2021), 87–97.

Huang

and Carley

K.M.

Parameterized convolutional neural networks for aspect level sentiment classification, arXiv preprint arXiv:1909.06276 (2019).

Xing

, Liao

, Song

, Wang

, Zhang

, Wang

and Huang

Earlier attention? Aspect-aware LSTM for aspect-based sentiment analysis, arXiv preprint arXiv:1905.07719 (2019).

Bahdanau

, Cho

and Bengio

Neural machine translation by jointly learning to align and translate, arXiv preprint arXiv:1409.0473 (2014).

, Li

, Zhang

and Wang

Interactive attention networks for aspect-level sentiment classification, arXiv preprint arXiv:1709.00893 (2017).

Tang

, Qin

and Liu

Aspect level sentiment classification with deep memory network, arXiv preprint arXiv:1605.08900 (2016).

Tang

, Qin

, Feng

and Liu

Effective LSTMs for target-dependent sentiment classification, arXiv preprint arXiv:1512.01100 (2015).

Yan

, Dai

, Qiu

and Zhang

A Unified Generative Framework for Aspect-Based Sentiment Analysis. arXiv preprint arXiv:2106.04300 (2021).

Rish

An empirical study of the naive Bayes classifier, 2001, pp. 41–46.

10.

Chung

, Gulcehre

, Cho

and Bengio

Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint arXiv:1412.3555 (2014).

11.

Pennington

, Socher

and Manning

C.D.

Glove: Global vectors for word representation, 2014, pp. 1532–1543.

12.

Wagner

, Arora

, Cortes

, Barman

, Bogdanova

, Foster

and Tounsi

Dcu: Aspect-based polarity classification for semeval task 4, (2014).

13.

Yang

, Yang

, Wang

and Xie

Multi-entity aspect based sentiment analysis with context, entity and aspect memory, 2018.

14.

Suykens

J.A.K.

and Vandewalle

, Least squares support vectormachine classifiers, Neural processing letters 9 (1999), 293–300.

15.

Jiang

, Yu

, Zhou

, Liu

and Zhao

Target-dependent twitter sentiment classification, 2011, pp. 151–160.

16.

Zhang

, Wang

and Liu

, Deep learning for sentiment analysis: A survey, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8 (2018), e1253.

17.

, Zhao

, Guo

, Cheng

and Su

Learning to detect opinion snippet for aspect-based sentiment analysis, arXiv preprint arXiv:1909.11297 (2019).

18.

Luong

M.-T.

, Pham

and Manning

C.D.

Effective approaches to attention-based neural machine translation, arXiv preprint arXiv:1508.04025 (2015).

19.

Majumder

, Poria

, Gelbukh

, Akhtar

M.S.

, Cambria

and Ekbal

IARM: Inter-aspect relation modeling with memory networks in aspect-based sentiment analysis, 2018, pp. 3402–3411.

20.

Chen

, Sun

, Bing

and Yang

Recurrent attention network on memory for aspect sentiment analysis, 2017, pp. 452–461.

21.

Zhu

and Qian

Enhanced aspect level sentiment classification with auxiliary memory, 2018, pp. 1077–1087.

22.

, Zhang

, Hou

and Song

A position-aware bidirectional attention network for aspect-level sentiment analysis, 2018, pp. 774–784.

23.

Hochreiter

and Schmidhuber

, Long short-term memory, Neural computation 9 (1997), 1735–1780.

24.

, Xu

, Wu

, Yuan

, Huang

and Li

, Aspect-basedsentiment analysis via fusing multiple sources of textual knowledge, Knowledge-Based Systems 183 (2019), 104868.

25.

, Lee

, Whang

, Park

, Seo

, Kim

and Kim

Deep Context-and Relation-Aware Learning for Aspect-based Sentiment Analysis. arXiv preprint arXiv:2106.03806 (2021).

26.

Joachims

Text categorization with support vector machines: Learning with many relevant features, Springer, 1998, pp. 137–142.

27.

Bengio

, Ducharme

, Vincent

and Janvin

, A neural probabilistic language model, The Journal of Machine Learning Research 3 (2003), 1137–1155.

28.

Xue

and Li

Aspect based sentiment analysis with gated convolutional networks, arXiv preprint arXiv:1805.07043 (2018).

29.

Zhang

, Yoshida

and Tang

TFIDF, LSI and multiword in information retrieval and text categorization, IEEE, 2008, pp. 108–113.

30.

LeCun

and Bengio

, Convolutional networks for images, speech, and time series, , The handbook of brain theory and neural networks 3361 (1995), 1995.

31.

Tay

, Tuan

L.A.

and Hui

S.C.

Learning to attend via word-aspect associative fusion for aspect-based sentiment analysis, 2018.

32.

Wang

, Huang

, Zhu

and Zhao

Attention-based LSTM for aspect-level sentiment classification, 2016, pp. 606–615.

33.

, Zhang

, Ye

, Yu

, Chen

and Chen

Interventional Aspect-Based Sentiment Analysis. arXiv preprint arXiv:2104.11681 (2021).

Aspect-level sentiment analysis for based on joint aspect and position hierarchy attention mechanism network

Abstract

Keywords

1 Introduction

2 Related work

3 Methods

3.2 Position embedding

4.1 Dataset

Table 1 The detailed data from Semeval 2014 DataSets Dataset Positive Neutral Negative Laptop-Train 767 373 673 Laptop-Dev 220 87 193 Laptop-Test 341 169 128 Restaurant-Train 1886 531 685 Restaurant-Dev 278 102 120 Restaurant-Test 728 196 196

4.3 Evaluation metric

Table 4 The Accuracy value of different models Model Restaurant-Acc Laptop-Acc ATAE-LSTM 76.79 67.40 HAM-LSTM 76.61 69.59 ATAE-Bi-LSTM 75.98 70.53 HAM-Bi-GRU 77.41 66.61 JAP-HAM 78.125 70.846

Acknowledgments

References

Table 1
The detailed data from Semeval 2014 DataSets

Dataset Positive Neutral Negative

Laptop-Train 767 373 673

Laptop-Dev 220 87 193

Laptop-Test 341 169 128

Restaurant-Train 1886 531 685

Restaurant-Dev 278 102 120

Restaurant-Test 728 196 196

Table 4
The Accuracy value of different models

Model Restaurant-Acc Laptop-Acc

ATAE-LSTM 76.79 67.40

HAM-LSTM 76.61 69.59

ATAE-Bi-LSTM 75.98 70.53

HAM-Bi-GRU 77.41 66.61

JAP-HAM 78.125 70.846