Abstract
BACKGROUND:
Recognising emotions in humans is a great challenge in the present era and has several applications under affective computing. Deep learning (DL) is found as a successful tool for prediction of human emotions in different modalities.
OBJECTIVE:
To predict 3D emotions with high accuracy in multichannel physiological signals, i.e. electroencephalogram (EEG).
METHODS:
A hybrid DL model consisting of convolutional neural network (CNN) and gated recurrent units (GRU) is proposed in this work for emotion recognition in EEG data. CNN has the capability of learning abstract representation, whereas GRU can explore temporal correlation. A bi-directional variation of GRU is used here to learn features in both directions. Discrete and dimensional emotion indices are recognised in two publicly available datasets SEED and DREAMER, respectively. A fused feature of energy and Shannon entropy
RESULTS:
The performance of the presented model is measured in terms of average accuracy, which is obtained as 86.9% and 93.9% for SEED and DREAMER datasets, respectively.
CONCLUSION:
The proposed convolution bi-directional gated recurrent unit neural network (CNN-BiGRU) model outperforms most of the state-of-the-art and competitive hybrid DL models, which indicates the effectiveness of emotion recognition using EEG signals and provides a scientific base for the implementation in human-computer interaction (HCI).
Introduction
Emotions are psychological interactions and behaviours that play a vital part in ordinary human social interactions [1]. They typically emerge in a reaction through the human body that is indicated as external reactions such as facial expression, body gestures and voice changes [2]. Certain reactions are not visible but can be recorded by physiological signals [3]. EEG signals are responses of the human body recorded in the form of electrical activity in the presence of external stimuli [4]. Physiological signals are usually acquired through different wearable devices in forms such as electroencephalogram (EEG), electrocardiogram (ECG) and blood volume pulse (BVP) [5]. These signals are treated as more reliable tools for monitoring cognition activities, as these signals are collected directly from the human body [6]. The ways different physiological signals are collected are shown in Fig. 1.
Different physiological signals collection methods [7].
Emotion recognition (ER) is being applied in various applications such as online gaming, shopping and healthcare monitoring [8]. Thus, integrating effective computing into human-computer interaction (HCI) can help us to create an interface that considers human emotive states, making the interface intelligent, user-friendly and efficient [9].
Currently, with the advancement in technologies in machine learning (ML) [10], deep learning (DL) [11] and artificial intelligence (AI) [12], different pattern recognition problems are solved easily and efficiently. With the same motivation, a hybrid DL model is proposed in this research to identify emotional states in EEG signals. Multichannel EEG data is the preferred tool for psychiatric research because it provides information from different lobes of the brain [13]. Moreover, multichannel physiological (also known as EEG) signals do not suffer from depicting human behaviours; thus, EEG-based emotion identification offers persistent results when correlated to other approaches such as gesture, body, speech and facial expression [14].
Human emotions are measured in the form of discrete and dimensional representations. Six fundamental emotions, namely happiness, surprise, disgust, fear, sadness and anger were defined by Ekman and Friesen [15] as discrete emotions. Furthermore, regardless of culture, humans perceive these emotions. The two orthogonal dimensions were defined as valence and arousal [16]. Valence is represented as pleasant or unpleasant on a negative to positive scale, while arousal is represented as calm or excited on a low to high scale. Further, third-dimension dominance was added by Russell, who describes the state of control or empowered emotions [17]. Figure 2 shows a pictorial representation of different discrete and dimensional human emotions.
Discrete and dimensional emotion model [18].
As EEG signals produced from various brain areas play a varying influence on emotion detection, thus using spatial and temporal knowledge from the EEG signals can aid in the improvement of ER tasks [19]. To detect the spatial and temporal knowledge of the EEG signal, CNN and recurrent neural network (RNN) models have been used in the past [20]. The shortcoming of RNN was addressed in long short-term memory (LSTM) [21]. Further, the gates of LSTM were optimised in a recently devised advanced neural network called gated recurrent units (GRU) [22]. GRU was proposed with two instead of the three gates in LSTM [46]. GRU is capable of keeping relevant features and forgetting irrelevant ones. Thus, a GRU model is selected along with CNN for ER in the present research work. The CNN layer first retains the structure of signals, whereas GRU extracts the temporal features present in the EEG data.
Typically, ER research has been conducted in two ways: subject dependent and subject independent. Though subject-dependent research attains higher accuracy, subject-independent research provides more generalised results [23]. Thus, the current research is conducted in a subject-independent manner.
Another important aspect to be managed in ER research is feature extraction (FE), which has been performed in the past by extracting statistical, temporal, spectral and time-frequency features. The commonly extracted features in the above three domains are energy (En), entropy (Et), power spectral density (PSD) and Hjorth parameters [24].
Among all FE methods, features extracted in the time-frequency domain are found best [25, 26, 27]. Thus, En, shannon entropy (SE) and differential entropy (DE) features are extracted here after applying wavelet transform (time-frequency representation). Researchers successfully extracted various emotions using combined rather than single features [28, 29]. Hence, a fused feature of energy and shannon entropy (
After extracting fused features, the temporal information present in signals is extracted in backward and forward directions by a bidirectional gated recurrent unit (BiGRU). To access spatial information across multiple areas of the brain with long-term interdependence, a convolution bi-directional gated recurrent unit neural network (CNN-BiGRU) model is proposed in this work. The proposed model can remember the interdependence of time series data and would aid in enhancing classification accuracy. The overall flow diagram of the research is depicted in Fig. 3.
Flow diagram of the proposed ER system.
Dimensional and discrete emotions are recognised using two publicly accessible datasets, i.e. SEED and DREAMER. EEG bands are extracted from multilevel Daubechies6 (db6) wavelet transform. Three different features, En, SE and DE are extracted after extracting bands from a discrete wavelet transform (DWT). These features are then combined to prepare fused vectors such as ( 100 optimal features are selected using principal component analysis (PCA) from a fused feature matrix. The selected features are fed into the proposed novel hybrid CNN-BiGRU-based model for ER. The model’s performance is evaluated in terms of average accuracy and compared with three hybrid DL models such as CNN-LSTM, CNN-BiLSTM and CNN-GRU.
The sections presented in this work include a literature review in part 2, a description of DL models and a proposed model in Section 3, a detailing of experiments in Section 4, a presentation of results in Section 5, discussion in Section 6, and lastly a statement of conclusions and presentation of future directions in Section 7.
Several shallow learning and DL models have been applied to ER in recent years [11]. Prior to that, Shallow learning, formally known as ML, was in use, whereas ensembles and DL models have gained popularity in the present years.. The details of important related recent works are presented in Table 1. It summarises other research conducted in the last five years. From Table 1, it is seen that in feature extraction, most extracted features are PSD [31], statistical features [33, 41], wavelet features [33, 37, 41] and DE [32, 39]. Mainly single features are used in this research. In the classification part ML,
Related work on ER in last five years
Related work on ER in last five years
ensemble method and DL are applied to the most explored datasets, such as DEAP and SEED [32, 34, 36, 37, 38, 39, 40, 41].
From Table 1, it is also observed that state-of-the-art ML methods such as SVM, RF, DT and KNN were implemented up to 2020, leading to a maximum accuracy of 74.85%. In contrast, DL models such as CNN, LSTM, RNN and BiLSTM were majorly applied after 2020, and these models attained high accuracies up to 99%. However, most of the DL models suffer from the problem of over fitting.
A common note from these studies is that they all utilised single ML and DL results with low accuracy. The above studies show a demand to investigate additional DL-based algorithms to improve overall emotion identification efficiency in various datasets. Hybrid DL models are sophisticated tools to improve classification efficiency by adding the diversified characteristics of different models. In addition, the hybrid DL model proposed here efficiently reduces the effects of long-term non-stationarity characteristics present in EEG data while improving the accuracy and resilience of ER models.
A hybrid DL model CNN-BiGRU is proposed in the present research and compared with three other ambitious hybrid DL models: CNN-LSTM, CNN-BiLSTM and CNN-GRU. All applied models use capabilities of CNN, LSTM, GRU and bi-directionality. Due to their powerful self-learning capabilities for a complicated depiction of features, these models have recently gained popularity in applications such as natural language processing (NLP), image classification, video tracking and speech sequence modelling. In contrast, these models are not very popular in EEG signal classification. Short descriptions of the models used in this work are listed below:
CNN
CNN [42, 43] is a classical DL model widely used in image recognition. The EEG can be represented as a 2D matrix with local correlation features such as images. Thus, the similarity in the EEG signals and image makes CNN suitable for classifying EEG signals for ER tasks. CNN architecture generally consists of convolution, pooling and fully connected layers. The convolution layer aids in feature extraction from previous layer input. The pooling layers maximise the local data, control the overfitting and thus reduce the calculations. The fully connected layers help to classify features in multiple classes [44]. Figure 4 represents the basic working of CNN models.
Basic CNN architecture [45].
The GRU [46] was invented in 2014. GRU is an optimised version of LSTM that consists of two gates, forget and update instead of three. The forget gate is responsible for forgetting the irrelevant features, whereas the update gate keeps important features related to time updated. Unlike LSTM, it has one hidden state (Ht) that makes GRU simpler in design, as shown in Fig. 5.
Brief architecture of GRU network.
GRU gates help to determine what is to be retained, passed or dropped to the next state, represented by ‘closed to 0’ and ‘closed to 1’, where ‘closed to 0’ means the data is less important and ‘closed to 1’ means the data is important and needs to be kept. The detailed architecture of a GRU model is shown in Fig. 6. The activation function is always sigmoid and is represented by
Detailed architecture of GRU network.
The GRU outperforms LSTM because its architecture is less complicated [22]. In addition, GRU helps to solve the vanishing gradient problem. Other important aspect of the GRU-based model is that it uses less memory and is a faster and easier solution while training. GRU also works efficiently in the presence of fewer amounts of data [47].
A bidirectional neural network [48] is a sequence processing model composed of two neural networks, one receiving input in forward directions and the other in backward directions. The output layer retains information from past: backwards and future, i.e. forward states. Thus, the directionality property helps to keep features in both directions at any point in time. The network has applications in fields other than emotion recognition such as speech recognition, handwriting recognition, sentiment analysis and protein structure prediction. They show good results, as they understand the context in a better way. Bidirectional architecture is shown in Fig. 7.
Structure of bidirectional models [49].
In our model, one 1D-CNN is applied for convolution, followed by batch normalisation and max pooling layers. This part of our model preserves the association among the signals. The formula for convolution operation is given by Eq. (1):
Where
Where
After the convolution layer, a bidirectional GRU layer is added to extract important temporal features. The BiGRU hidden layer includes forward and reverse hidden layers, which are represented through Eqs (3) and (4):
After BiGRU flattening, a dense and dropout layer is added to get the final output as shown in Eq. (5):
Where
Hybrid CNN-BiGRU network architecture.
Where
The deep learning experiments in current research were conducted in Google Collaboratory GPU version whereas preprocessing and feature extraction is done on an 8th Generation Intel Core i5-8250U CPU at 1.60 GHz with 4GB RAM, X64 based processor.
Datasets
SEED
The SEED [50] dataset was gathered by 15 participants using the 62-channel ESI NeuroScan system. The electrodes were positioned using the international 10–20 system [51]. The positive, neutral and negative emotions of all participants were evoked by 15 Chinese movie snippets. All participants were asked to complete a 45-second self-assessment questionnaire after watching emotional videos. The experiments were repeated three times for each participant.
DREAMER
The DREAMER [52] dataset was collected using 14 electrode placements using a 10–20 international electrode placement system to collect EEG data induced by video stimuli. In DREAMER, emotions are annotated in the form of valence, arousal and dominance (VAD) that correspond to self-assessments given by the participants. The EEG signals were gathered from the 23 healthy participants using a portable wireless device with a sampling rate of 128 Hz. The participants’ emotions were evoked using 18 audio-visual stimuli of length 65–393 min. The description of data and labels present in the above-described datasets are shown in Table 2.
Dimensions of data and labels present in SEED and DREAMER datasets
Dimensions of data and labels present in SEED and DREAMER datasets
EEG signals are low in amplitude, unstable and highly susceptible to noise; hence, data was pre-processed first for accurate emotion recognition.
The EEG signal in DREAMER consists of artefacts such as eye flickers and cardiac irregularities. These artefacts are primarily prominent below 4Hz, whereas other artefacts, such as muscular movement, are active above frequencies of 30 Hz. Noise present in the EEG signal is active in the range of 50–60 Hz [37]. DREAMER datasets were pre-processed in the following steps:
In the first step, the Python function firwin is used to extract the frequency range of 4–30 Hz. In the second step, the linear-digital filter filtfilt function was employed to filter the undesired noise in the EEG signals. After eliminating noise from stimulus and baseline signals in the DREAMER dataset, the baseline signals were divided from the stimulus signal in the third step. A 414 rows x 14 columns dataset was found after pre-processing, where 414 rows correspond to 18 videos x 23 subjects, and the 14 columns correspond to the number of channels used for data collection.
The VAD emotions of the DREAMER dataset are converted into ‘low (0)’ and ‘high (1)’ for each dimension. Table 3 represents the scheme for low and high values in each dimension.
Conversion of VAD from scale (1–5) to (0–1)
Next, En, SE and DE features were extracted from the CSVs created in both datasets. To extract features for the different bands, band extraction is employed with multilevel db6 wavelet transform. The next subsection states the band extraction procedure.
A DWT [53] method is used to extract bands from the signals in the time-frequency domain. The coefficients are calculated using Eqs (7) and (8):
Where,
Feature extraction is followed after band extractions. In this work, the
Where
After pre-processing, the features’ dimensions obtained from both datasets are fused for
Feature scaling is conducted on data to eliminate numerical properties of datasets with extremely significant proportional disparities, which typically leads to poor performance of ML algorithms. Therefore, the data must be scaled to a pre-set range of values. Scalar transform and normalisation of data is done using principal component analysis (PCA). Then the 100 most important features were selected to reduce the dimension of features, thus reducing the number of nodes as input to the classifier. The final matrices obtained were 675 x 100 and 414 x 100 in SEED and DREAMER, respectively.
Classification of emotions
The classification of emotions was performed by the proposed CNN-BiGRU model, which took selected features from both the datasets as input and classified the positive, negative and neutral emotions in the SEED and the VAD emotions in the DREAMER dataset. In this work, a 1D convolutional layer is implemented, followed by batch normalisation and max pooling. After this, a bidirectional GRU layer was added, followed by a flatten layer and dense layer. The sigmoid function was implemented at the outer dense layer. In the output part, classified binary emotions were taken. The input and output dimensions are shown in Table 4 The layered architecture of CNN-BiGRU used for implementation is shown in Fig. 9.
Input and output data dimensions fed to classification model
Input and output data dimensions fed to classification model
The experiments were conducted on two datasets SEED and DREAMER to classify discrete and dimensional emotions respectively. The parameter values used in the proposed model are depicted in Table 5.
Parameters value used to build the CNN–BiGRU model
Parameters value used to build the CNN–BiGRU model
CNN-BiGRU network architecture (proposed model).
The proposed CNN-BiGRU model was also compared with three competitive hybrid DL models i.e. CNN-LSTM, CNN-BiLSTM and CNN-GRU. The model performance is evaluated in terms of accuracy, precision, recall and F1-Score given in Eqs (14)–(17):
Where,
TP
The experiments were also conducted for single (En, SE or DE) and fused (
After experimentation for single and fused features, it is observed that the accuracies obtained using fused features are higher than single features. Thus, further emotions were identified for fused features with four different hybrid DL models: CNN-LSTM, CNN-BiLSTM, CNN-GRU and CNN-BiGRU. The accuracy, precision, recall and F1 score obtained from the four hybrid DL models for SEED and DREAMER datasets are shown in Table A2 (Appendix A); the highest accuracy is shown in bold. From Table A2, the CNN-BiGRU model is found to have the highest accuracy for all emotion dimensions for the SEED dataset, while the highest accuracy is obtained in the dominance dimension for the DREAMER dataset for
Further, the comparison of proposed model is done on the basis of average accuracies. The average accuracies for SEED and DREAMER dataset are calculated by Eqs (18) and (19) respectively.
A comparison of the hybrid CNN-BiGRU model with the CNN and BiGRU model for both datasets is shown in Fig. 10. From Fig. 10, the average accuracies obtained by CNN-BiGRU are found to be higher than a single CNN or BiGRU model.
Average accuracy of CNN, BiGRU and hybrid CNN-BiGRU model for SEED dataset and DREAMER dataset.
The average accuracy of all hybrid DL models is compared in Fig. 11. The average accuracies obtained by the CNN-BiGRU model are found to be the highest for both datasets. A comparison of the proposed model with existing work is presented in Table 6. The table shows that DL models GRU and LSTM outperformed other models with an average accuracy of 82% [54, 55, 56, 57]. Fused feature models are more accurate for the SEED dataset [57, 58]. Matching the previous research, the proposed hybrid DL model with fused features achieved the highest average accuracy of 86.9% in the SEED dataset.
Average accuracy of the different hybrid models in (a) SEED and (b) DREAMER.
Comparison of the current proposed work with the existing work
For the DREAMER dataset, from Table 6, it is observed that the ML method with PSD as a feature results in the lowest accuracy in a study by Katsigiannis and Ramzan [52]. At the same time, the hybrid DL and ELM methods improved average accuracy up to 79.11% [60]. Fused feature models with the DL method increased average accuracy to 80.3% [61], while the proposed hybrid DL model with fused features increased average accuracy to 93.9%.
From the various related work, it is observed that time-frequency features are more suitable in EEG-based ER tasks [11, 37, 62]. Besides ER time-frequency feature such as global wavelet spectrum (GWS) are also applied for epilepsy detection from EEG signals and provide promising results [63]. Thus, in the presented work, DWT is used for band extraction. After band extraction, fused features were taken for the experiment, as they give better accuracies than single features [64, 65]. In addition to emotion recognition, fused feature models are also used in other research work [66]. In many recent works [31, 33, 35, 41], ML and DL based model are implemented in ER for EEG signals. Beside these, ML and DL models are also implemented in other task such as epileptic seizure classification, for diagnosing Asthma and Chronic Obstructive Pulmonary Disease, for classification of Patau, Edwards, Down, Turner and Klinefelter Syndrome [67, 68, 69, 70].
However, single DL models are not efficient in finding the detailed context of the problem, while hybrid DL models are more accurate in ER [71, 72]. The same is found in other applications such as stress detection and vein pattern recognition [73, 74]. In this work, BiGRU is taken over GRU model as it helps in preserving important features in both directions [75, 76]. Thus, a hybrid CNN-BiGRU model with fused features
The accuracy and loss graph of the proposed model CNN-BiGRU is shown in Fig. 12.
(a) Accuracy and (b) Loss graph for the proposed model for the SEED dataset.
This work proposes an EEG-based hybrid DL model for ER, namely CNN-BiGRU. The above experiments and discussions claim that DL models are more efficient than basic ML models for EEG signal classification. In this work, different experiments and analyses were performed using a hybrid DL model – CNN was used to extract spatial information, which was fed into the BiGRU to retrieve temporal features. The suggested model’s outcomes in terms of average accuracy were compared to CNN-LSTM, CNN-BiLSTM and CNN-GRU. CNN-BiGRU achieved the best performance. This means that temporal characteristics produced from a BiGRU can successfully reflect EEG data diversity.
The present work’s experimental results show that fused features give the highest emotion classification accuracy in all three dimensions, i.e. positive, negative, neutral, and VAD dimensions. Though hybrid DL models are efficient in emotion recognition. However, are complex and computationally expensive whilst experimenting. However, attention based methods can be used in future work to boost the efficiency of the BiGRU-based network. Furthermore, Multimodal work may be used to learn from modalities.
Footnotes
Conflict of interest
None to report.
Supplementary data
The supplementary files are available to download from http://dx.doi.org/10.3233/THC-220458.
