A novel convolution bi-directional gated recurrent unit neural network for emotion recognition in multichannel electroencephalogram signals

Abstract

BACKGROUND:

Recognising emotions in humans is a great challenge in the present era and has several applications under affective computing. Deep learning (DL) is found as a successful tool for prediction of human emotions in different modalities.

OBJECTIVE:

To predict 3D emotions with high accuracy in multichannel physiological signals, i.e. electroencephalogram (EEG).

METHODS:

A hybrid DL model consisting of convolutional neural network (CNN) and gated recurrent units (GRU) is proposed in this work for emotion recognition in EEG data. CNN has the capability of learning abstract representation, whereas GRU can explore temporal correlation. A bi-directional variation of GRU is used here to learn features in both directions. Discrete and dimensional emotion indices are recognised in two publicly available datasets SEED and DREAMER, respectively. A fused feature of energy and Shannon entropy $({\overrightarrow{\textit{EnSE}}})$ and energy and differential entropy $({\overrightarrow{\textit{EnDE}}})$ are fed in the proposed classifier to improve the efficiency of the model.

RESULTS:

The performance of the presented model is measured in terms of average accuracy, which is obtained as 86.9% and 93.9% for SEED and DREAMER datasets, respectively.

CONCLUSION:

The proposed convolution bi-directional gated recurrent unit neural network (CNN-BiGRU) model outperforms most of the state-of-the-art and competitive hybrid DL models, which indicates the effectiveness of emotion recognition using EEG signals and provides a scientific base for the implementation in human-computer interaction (HCI).

Keywords

EEG hybrid deep learning CNN BiGRU emotion recognition

1. Introduction

Emotions are psychological interactions and behaviours that play a vital part in ordinary human social interactions [1]. They typically emerge in a reaction through the human body that is indicated as external reactions such as facial expression, body gestures and voice changes [2]. Certain reactions are not visible but can be recorded by physiological signals [3]. EEG signals are responses of the human body recorded in the form of electrical activity in the presence of external stimuli [4]. Physiological signals are usually acquired through different wearable devices in forms such as electroencephalogram (EEG), electrocardiogram (ECG) and blood volume pulse (BVP) [5]. These signals are treated as more reliable tools for monitoring cognition activities, as these signals are collected directly from the human body [6]. The ways different physiological signals are collected are shown in Fig. 1.

Figure 1.

Different physiological signals collection methods [7].

Emotion recognition (ER) is being applied in various applications such as online gaming, shopping and healthcare monitoring [8]. Thus, integrating effective computing into human-computer interaction (HCI) can help us to create an interface that considers human emotive states, making the interface intelligent, user-friendly and efficient [9].

Currently, with the advancement in technologies in machine learning (ML) [10], deep learning (DL) [11] and artificial intelligence (AI) [12], different pattern recognition problems are solved easily and efficiently. With the same motivation, a hybrid DL model is proposed in this research to identify emotional states in EEG signals. Multichannel EEG data is the preferred tool for psychiatric research because it provides information from different lobes of the brain [13]. Moreover, multichannel physiological (also known as EEG) signals do not suffer from depicting human behaviours; thus, EEG-based emotion identification offers persistent results when correlated to other approaches such as gesture, body, speech and facial expression [14].

Human emotions are measured in the form of discrete and dimensional representations. Six fundamental emotions, namely happiness, surprise, disgust, fear, sadness and anger were defined by Ekman and Friesen [15] as discrete emotions. Furthermore, regardless of culture, humans perceive these emotions. The two orthogonal dimensions were defined as valence and arousal [16]. Valence is represented as pleasant or unpleasant on a negative to positive scale, while arousal is represented as calm or excited on a low to high scale. Further, third-dimension dominance was added by Russell, who describes the state of control or empowered emotions [17]. Figure 2 shows a pictorial representation of different discrete and dimensional human emotions.

Figure 2.

Discrete and dimensional emotion model [18].

As EEG signals produced from various brain areas play a varying influence on emotion detection, thus using spatial and temporal knowledge from the EEG signals can aid in the improvement of ER tasks [19]. To detect the spatial and temporal knowledge of the EEG signal, CNN and recurrent neural network (RNN) models have been used in the past [20]. The shortcoming of RNN was addressed in long short-term memory (LSTM) [21]. Further, the gates of LSTM were optimised in a recently devised advanced neural network called gated recurrent units (GRU) [22]. GRU was proposed with two instead of the three gates in LSTM [46]. GRU is capable of keeping relevant features and forgetting irrelevant ones. Thus, a GRU model is selected along with CNN for ER in the present research work. The CNN layer first retains the structure of signals, whereas GRU extracts the temporal features present in the EEG data.

Typically, ER research has been conducted in two ways: subject dependent and subject independent. Though subject-dependent research attains higher accuracy, subject-independent research provides more generalised results [23]. Thus, the current research is conducted in a subject-independent manner.

Another important aspect to be managed in ER research is feature extraction (FE), which has been performed in the past by extracting statistical, temporal, spectral and time-frequency features. The commonly extracted features in the above three domains are energy (En), entropy (Et), power spectral density (PSD) and Hjorth parameters [24].

Among all FE methods, features extracted in the time-frequency domain are found best [25, 26, 27]. Thus, En, shannon entropy (SE) and differential entropy (DE) features are extracted here after applying wavelet transform (time-frequency representation). Researchers successfully extracted various emotions using combined rather than single features [28, 29]. Hence, a fused feature of energy and shannon entropy ( $\overrightarrow{\textit{EnSE}}$ ) and energy and differential entropy ( $\overrightarrow{\textit{EnDE}}$ ) features are extracted before classification. The fused features help to avoid linearity and Gaussian features in EEG signals.

After extracting fused features, the temporal information present in signals is extracted in backward and forward directions by a bidirectional gated recurrent unit (BiGRU). To access spatial information across multiple areas of the brain with long-term interdependence, a convolution bi-directional gated recurrent unit neural network (CNN-BiGRU) model is proposed in this work. The proposed model can remember the interdependence of time series data and would aid in enhancing classification accuracy. The overall flow diagram of the research is depicted in Fig. 3.

Figure 3.

Flow diagram of the proposed ER system.

1.1 Our contributions

•
Dimensional and discrete emotions are recognised using two publicly accessible datasets, i.e. SEED and DREAMER.
•
EEG bands are extracted from multilevel Daubechies6 (db6) wavelet transform.
•
Three different features, En, SE and DE are extracted after extracting bands from a discrete wavelet transform (DWT). These features are then combined to prepare fused vectors such as ( $\overrightarrow{\textit{EnSE}}$ ) and ( $\overrightarrow{\textit{EnDE}}$ ).
•
100 optimal features are selected using principal component analysis (PCA) from a fused feature matrix.
•
The selected features are fed into the proposed novel hybrid CNN-BiGRU-based model for ER.
•
The model’s performance is evaluated in terms of average accuracy and compared with three hybrid DL models such as CNN-LSTM, CNN-BiLSTM and CNN-GRU.

The sections presented in this work include a literature review in part 2, a description of DL models and a proposed model in Section 3, a detailing of experiments in Section 4, a presentation of results in Section 5, discussion in Section 6, and lastly a statement of conclusions and presentation of future directions in Section 7.
2. Related works

Several shallow learning and DL models have been applied to ER in recent years [11]. Prior to that, Shallow learning, formally known as ML, was in use, whereas ensembles and DL models have gained popularity in the present years.. The details of important related recent works are presented in Table 1. It summarises other research conducted in the last five years. From Table 1, it is seen that in feature extraction, most extracted features are PSD [31], statistical features [33, 41], wavelet features [33, 37, 41] and DE [32, 39]. Mainly single features are used in this research. In the classification part ML,

Table 1
Related work on ER in last five years

Ref., year	FE method	Classification method	Datasets	Emotions	Accuracy
[30, 2018]	Fractal dimension	Support vector machine (SVM)	Own-dataset	Calm, anger and happiness	Avg. 60%
[31, 2019]	PSD	SVM	Own-dataset	Arousal and valence	67.4%
[32, 2019]	1 ${}^{\text{st}}$ and 2 ${}^{\text{nd}}$ order difference in DEAP and DE in SEED	Decision Tree (DT), K-nearest neighbour (KNN) and random forest (RF)	DEAP and SEED	Positive, negative and calm	74.85%
[33, 2019]	Statistical measures, DE, Wavelets	ANN	Own dataset	Happy, Sad, Angry, Surprise, Scared	Avg. 53.18%
[34, 2020]	Time and frequency domain features	SVM, random forest	DEAP	Arousal	62.58%
[35, 2020]	FFT	LSTM	Own-dataset	Positive and negative	92.84%
[36, 2020]	2D spatial feature frame	Deep forest	DEAP and DREAMER	Valence, arousal and dominance	Avg. 97.69%
[37, 2021]	DWT and Empirical Mode Decomposition (EMD)	Ensemble ML (EML)	SEED, INTERFACES and MUSEC	Arousal and valence	95.81%
[38, 2021]	TQWT	Rotation forest	SEED	Positive, negative and neutral	93%
[39, 2022]	DE	CNN-LSTM	DEAP and SEED	Neutral, positive and negative	97.16%
[40, 2022]	Power topographic map and temporal statistics	STFFNN	DEAP	Arousal and valence	81.3%
[41, 2022]	Statistical features, wavelet features, and Hurst exponent	BiLSTM	DEAP	Valence, arousal, and liking	99.68%

ensemble method and DL are applied to the most explored datasets, such as DEAP and SEED [32, 34, 36, 37, 38, 39, 40, 41].

From Table 1, it is also observed that state-of-the-art ML methods such as SVM, RF, DT and KNN were implemented up to 2020, leading to a maximum accuracy of 74.85%. In contrast, DL models such as CNN, LSTM, RNN and BiLSTM were majorly applied after 2020, and these models attained high accuracies up to 99%. However, most of the DL models suffer from the problem of over fitting.

A common note from these studies is that they all utilised single ML and DL results with low accuracy. The above studies show a demand to investigate additional DL-based algorithms to improve overall emotion identification efficiency in various datasets. Hybrid DL models are sophisticated tools to improve classification efficiency by adding the diversified characteristics of different models. In addition, the hybrid DL model proposed here efficiently reduces the effects of long-term non-stationarity characteristics present in EEG data while improving the accuracy and resilience of ER models.

3. Method

A hybrid DL model CNN-BiGRU is proposed in the present research and compared with three other ambitious hybrid DL models: CNN-LSTM, CNN-BiLSTM and CNN-GRU. All applied models use capabilities of CNN, LSTM, GRU and bi-directionality. Due to their powerful self-learning capabilities for a complicated depiction of features, these models have recently gained popularity in applications such as natural language processing (NLP), image classification, video tracking and speech sequence modelling. In contrast, these models are not very popular in EEG signal classification. Short descriptions of the models used in this work are listed below:

3.1 CNN

CNN [42, 43] is a classical DL model widely used in image recognition. The EEG can be represented as a 2D matrix with local correlation features such as images. Thus, the similarity in the EEG signals and image makes CNN suitable for classifying EEG signals for ER tasks. CNN architecture generally consists of convolution, pooling and fully connected layers. The convolution layer aids in feature extraction from previous layer input. The pooling layers maximise the local data, control the overfitting and thus reduce the calculations. The fully connected layers help to classify features in multiple classes [44]. Figure 4 represents the basic working of CNN models.

Figure 4.

Basic CNN architecture [45].

3.2 GRU

The GRU [46] was invented in 2014. GRU is an optimised version of LSTM that consists of two gates, forget and update instead of three. The forget gate is responsible for forgetting the irrelevant features, whereas the update gate keeps important features related to time updated. Unlike LSTM, it has one hidden state (Ht) that makes GRU simpler in design, as shown in Fig. 5.

Figure 5.

Brief architecture of GRU network.

GRU gates help to determine what is to be retained, passed or dropped to the next state, represented by ‘closed to 0’ and ‘closed to 1’, where ‘closed to 0’ means the data is less important and ‘closed to 1’ means the data is important and needs to be kept. The detailed architecture of a GRU model is shown in Fig. 6. The activation function is always sigmoid and is represented by $\sigma$ . At any time $t$ , $x_{t}$ is the input, $z_{t}$ is the update gate that tells the model how much information needs to be kept, i.e. passed along to the future, and $r_{t}$ is the reset gate that is used in the model to decide how much of the past information needs to be forgotten. $\widetilde{h_{t}}$ is the candidate hidden state and helps to solve vanishing gradients problems. In the end, $h_{t}$ represents the hidden state and is the last step that holds information for the current unit and passes it down to the network with the help of update gates.

Figure 6.

Detailed architecture of GRU network.

The GRU outperforms LSTM because its architecture is less complicated [22]. In addition, GRU helps to solve the vanishing gradient problem. Other important aspect of the GRU-based model is that it uses less memory and is a faster and easier solution while training. GRU also works efficiently in the presence of fewer amounts of data [47].

3.3 Bi-directional network

A bidirectional neural network [48] is a sequence processing model composed of two neural networks, one receiving input in forward directions and the other in backward directions. The output layer retains information from past: backwards and future, i.e. forward states. Thus, the directionality property helps to keep features in both directions at any point in time. The network has applications in fields other than emotion recognition such as speech recognition, handwriting recognition, sentiment analysis and protein structure prediction. They show good results, as they understand the context in a better way. Bidirectional architecture is shown in Fig. 7.

Figure 7.

Structure of bidirectional models [49].

3.4 CNN-BiGRU (Proposed Model)

In our model, one 1D-CNN is applied for convolution, followed by batch normalisation and max pooling layers. This part of our model preserves the association among the signals. The formula for convolution operation is given by Eq. (1):

$\displaystyle X(n)=\left\{\begin{array}[]{ll}\mathop{\sum}\limits_{i=0}^{k}x({% n+i})h(i),&\text{if }n=0\\ \mathop{\sum}\limits_{i=0}^{k}x({n+i+({s-1})})h(i),&\text{otherwise}\\ \end{array}\right.$ (1)

Where $x$ is the input of length $n$ to the convolution layer, $h$ represents the kernel of length $k$ . The kernel window is shifted to position $s$ representing the strides after convolution operations. The output of convolution $y(n)$ is passed to the ReLU activation function. Then max pooling is done with the help of the mathematical expression given in Eq. (2):

$\displaystyle y(n)=\text{ReLU(X(n)) and }s(n)=y(n)\downarrow MP$ (2)

Where $s(n)$ is the output of $n^{\text{th}}$ neuron and $M P$ represents the max pooling operations.

After the convolution layer, a bidirectional GRU layer is added to extract important temporal features. The BiGRU hidden layer includes forward and reverse hidden layers, which are represented through Eqs (3) and (4):

$\displaystyle\overrightarrow{h_{t}}=\textit{GRU}(x_{t},\overrightarrow{h_{t-1}})$ (3) $\displaystyle\overleftarrow{h_{t}}=\textit{GRU}(x_{t},\overleftarrow{h_{t-1}})$ (4)

After BiGRU flattening, a dense and dropout layer is added to get the final output as shown in Eq. (5):

$\displaystyle Y=\text{sigmoid}(\hat{y})$ (5)

Where $\hat{y}$ is the output of the last dense layer. The loss function is given by Eq. (6):

$\displaystyle\text{Loss}=-\frac{1}{\textit{Output size}}\mathop{\sum}\limits_{% i=1}^{\textit{output size}}y_{i}.\log\widehat{y_{i}}+({1-y_{i}}).\log(1-% \widehat{y_{i}})$ (6)

Figure 8.

Hybrid CNN-BiGRU network architecture.

Where $\widehat{y_{i}}$ is the ith scalar value of the output model and $y_{i}$ is the actual target value and output size is the size of scalar values present in the model. A pictorial representation of our proposed model is shown in Fig. 8.

4. Experiments

The deep learning experiments in current research were conducted in Google Collaboratory GPU version whereas preprocessing and feature extraction is done on an 8th Generation Intel Core i5-8250U CPU at 1.60 GHz with 4GB RAM, X64 based processor.

4.1 Datasets

4.1.1 SEED

The SEED [50] dataset was gathered by 15 participants using the 62-channel ESI NeuroScan system. The electrodes were positioned using the international 10–20 system [51]. The positive, neutral and negative emotions of all participants were evoked by 15 Chinese movie snippets. All participants were asked to complete a 45-second self-assessment questionnaire after watching emotional videos. The experiments were repeated three times for each participant.

4.1.2 DREAMER

The DREAMER [52] dataset was collected using 14 electrode placements using a 10–20 international electrode placement system to collect EEG data induced by video stimuli. In DREAMER, emotions are annotated in the form of valence, arousal and dominance (VAD) that correspond to self-assessments given by the participants. The EEG signals were gathered from the 23 healthy participants using a portable wireless device with a sampling rate of 128 Hz. The participants’ emotions were evoked using 18 audio-visual stimuli of length 65–393 min. The description of data and labels present in the above-described datasets are shown in Table 2.

Table 2
Dimensions of data and labels present in SEED and DREAMER datasets

Datasets	Name	Shape	Contents
SEED	Data	3x15x62x48,000	(Experiments) $\times$ (Videos/trials) $\times$ channels $\times$ data (200 Hz $\times$ 240 s)
	Labels	3x15x3	(Experiments) $\times$ (videos/trials) $\times$ labels (Positive, Negative, Neutral)
DREAMER	Data	1x18x14x7168	(Experiments) $\times$ (Videos/trials) $\times$ channels $\times$ data (128 Hz $\times$ 199 s)
	Labels	1x18x3	(Experiments) $\times$ (videos/trials) $\times$ labels (Valence, Arousal and Dominance)

4.2 Data pre-processing

EEG signals are low in amplitude, unstable and highly susceptible to noise; hence, data was pre-processed first for accurate emotion recognition.

The EEG signal in DREAMER consists of artefacts such as eye flickers and cardiac irregularities. These artefacts are primarily prominent below 4Hz, whereas other artefacts, such as muscular movement, are active above frequencies of 30 Hz. Noise present in the EEG signal is active in the range of 50–60 Hz [37]. DREAMER datasets were pre-processed in the following steps:

•
In the first step, the Python function firwin is used to extract the frequency range of 4–30 Hz.
•
In the second step, the linear-digital filter filtfilt function was employed to filter the undesired noise in the EEG signals.
•
After eliminating noise from stimulus and baseline signals in the DREAMER dataset, the baseline signals were divided from the stimulus signal in the third step. A 414 rows x 14 columns dataset was found after pre-processing, where 414 rows correspond to 18 videos x 23 subjects, and the 14 columns correspond to the number of channels used for data collection.

The VAD emotions of the DREAMER dataset are converted into ‘low (0)’ and ‘high (1)’ for each dimension. Table 3 represents the scheme for low and high values in each dimension.

Table 3
Conversion of VAD from scale (1–5) to (0–1)

Emotions Low (0) High (1)

Valence (V) $<$ 3 $>=$ 3

Arousal (A) $<$ 3 $>=$ 3

Dominance (D) $<$ 3 $>=$ 3

Next, En, SE and DE features were extracted from the CSVs created in both datasets. To extract features for the different bands, band extraction is employed with multilevel db6 wavelet transform. The next subsection states the band extraction procedure.
4.3 Band extractions

Emotions	Low (0)	High (1)
Valence (V)	$<$ 3	$>=$ 3
Arousal (A)	$<$ 3	$>=$ 3
Dominance (D)	$<$ 3	$>=$ 3

A DWT [53] method is used to extract bands from the signals in the time-frequency domain. The coefficients are calculated using Eqs (7) and (8):

$\displaystyle C_{x(t)}^{({l,n})}=\mathop{\smallint}\nolimits x(t)\psi_{l,n}(t)dt$ (7) $\displaystyle\psi_{l,n}(t)=2^{-({l+1})}\psi({2^{-({l+1})}({t-2^{-1}n})})$ (8)

Where, $x(t)$ is the EEG signal in the time domain, $C_{x(t)}^{({l,n})}$ is wavelet coefficients. $l$ , $n$ is the scale and translation parameters. Db6 is applied as the mother wavelet for the multilevel wavelet transform. Five bands are obtained using a five-level wavelet transform. DWT gives an approximation coefficient $C_{i,l}$ and a series of detailed coefficients $d_{i,l}$ . The first (1/2 ${}^{5}$ ) lowest frequencies represent the approximation coefficient while the detailed coefficient represents the high-frequency present in the signal. Thus, only detailed coefficients are considered here, which are computed as in Eqs (9) and (10):

$\displaystyle d_{i,l}=\langle{x(t),\phi_{i,l}(t)}\rangle=\mathop{\smallint}% \nolimits x(t)2^{-\frac{l}{2}}{\psi}({2^{-i}t-l}){dt}$ (9) $\displaystyle\phi_{i,l}(t)=\phi({2^{-i}t-l})$ (10)

4.4 Feature extractions

Feature extraction is followed after band extractions. In this work, the $En_{l}$ , $SE_{l}$ and $DE_{l}$ features for each band are extracted as in Eqs (11)–(13):

$\displaystyle En_{l}=\mathop{\sum}\limits_{n}\ln B_{l}^{2}$ (11) $\displaystyle SE_{l}=-\mathop{\sum}\limits_{n}|{B_{l}^{2}}|\ln B_{l}^{2}$ (12) $\displaystyle DE_{l}=\mathop{\sum}\limits_{n}B_{l}\left(\frac{1}{2}\mbox{log}(% {2\ast\pi\ast e\ast\sigma^{2}})\right)$ (13)

Where $l=1,\ldots,5$ corresponds to five bands.

4.5 Feature scaling and normalization

After pre-processing, the features’ dimensions obtained from both datasets are fused for $\overrightarrow{\textit{EnSE}}$ and $\overrightarrow{\textit{EnDE}}$ . The dimensions for feature vectors obtained in the SEED dataset are 675 x 620, where 675 corresponds to 15 users x 3 sessions x 15 experiments and 620 corresponds to 10 features (5 En and 5 SE and DE) for 62 channels. The dimensions of the feature vector in the DREAMER dataset are 414 x 140, where 414 corresponds to 18 videos x 23 users and 140 corresponds to the fusion of 10 features (5 En and 5 SE and DE) for 14 channels.

Feature scaling is conducted on data to eliminate numerical properties of datasets with extremely significant proportional disparities, which typically leads to poor performance of ML algorithms. Therefore, the data must be scaled to a pre-set range of values. Scalar transform and normalisation of data is done using principal component analysis (PCA). Then the 100 most important features were selected to reduce the dimension of features, thus reducing the number of nodes as input to the classifier. The final matrices obtained were 675 x 100 and 414 x 100 in SEED and DREAMER, respectively.

4.6 Classification of emotions

The classification of emotions was performed by the proposed CNN-BiGRU model, which took selected features from both the datasets as input and classified the positive, negative and neutral emotions in the SEED and the VAD emotions in the DREAMER dataset. In this work, a 1D convolutional layer is implemented, followed by batch normalisation and max pooling. After this, a bidirectional GRU layer was added, followed by a flatten layer and dense layer. The sigmoid function was implemented at the outer dense layer. In the output part, classified binary emotions were taken. The input and output dimensions are shown in Table 4 The layered architecture of CNN-BiGRU used for implementation is shown in Fig. 9.

Table 4
Input and output data dimensions fed to classification model

Datasets	Data (X)		Labels (Y)
	Before PCA	After PCA	Positive (P)		Negative (Ng)		Neutral (N)
SEED	(675,620)	(675,100)	225		225		225
DREAMER	(414,140)	(414,100)	Valence (V)		Arousal (A)		Dominance (D)
			High (1)	Low (0)	High (1)	Low (0)	High (1)	Low (0)
			253	161	300	114	319	95

5. Results and discussion

The experiments were conducted on two datasets SEED and DREAMER to classify discrete and dimensional emotions respectively. The parameter values used in the proposed model are depicted in Table 5.

Table 5
Parameters value used to build the CNN–BiGRU model

Parameters	Values
Optimizer	Adam
Dropout	0.5
Loss function	Binary cross-entropy
Metrics	Average accuracy
Batch size	10
Epochs	100
Max pool size	2
Train-test split ratio	70:30
Kernel size	3
Activation function (1 ${}^{\text{st}}$ and 2 ${}^{\text{nd}}$ dense layer)	tanh
Activation function (3 ${}^{\text{rd}}$ dense layer)	ReLU

Figure 9.

CNN-BiGRU network architecture (proposed model).

The proposed CNN-BiGRU model was also compared with three competitive hybrid DL models i.e. CNN-LSTM, CNN-BiLSTM and CNN-GRU. The model performance is evaluated in terms of accuracy, precision, recall and F1-Score given in Eqs (14)–(17):

$\displaystyle\textit{Accuracy}=\frac{TP+TN}{TP+FN+TN+FP}\times 100\%$ (14) $\displaystyle\textit{Precision}=\frac{TP}{TP+FP}$ (15) $\displaystyle\textit{Recall}=\textit{Sensitivity}=\frac{TP}{TP+FN}$ (16) $\displaystyle\textit{F1-Score}=\frac{2\ast({\textit{Precision}\ast\textit{% Recall}})}{\textit{Precision}+\textit{Recall}}$ (17)

Where,

TP $=$ True positive TN $=$ True negative FN $=$ False negative FP $=$ False positive

The experiments were also conducted for single (En, SE or DE) and fused ( $\overrightarrow{\textit{EnSE}}$ and $\overrightarrow{\textit{EnDE}}$ ) features. The accuracies obtained for single and fused features by basic DL models such as CNN, LSTM, BiLSTM, GRU and BiGRU for SEED dataset are shown in Table A1 (Appendix A), and the highest accuracy is shown in bold. From Table A1, it is observed that fused features provide better accuracies (69–91%) when compared to single features (accuracy 66–88%).

After experimentation for single and fused features, it is observed that the accuracies obtained using fused features are higher than single features. Thus, further emotions were identified for fused features with four different hybrid DL models: CNN-LSTM, CNN-BiLSTM, CNN-GRU and CNN-BiGRU. The accuracy, precision, recall and F1 score obtained from the four hybrid DL models for SEED and DREAMER datasets are shown in Table A2 (Appendix A); the highest accuracy is shown in bold. From Table A2, the CNN-BiGRU model is found to have the highest accuracy for all emotion dimensions for the SEED dataset, while the highest accuracy is obtained in the dominance dimension for the DREAMER dataset for $\overrightarrow{\textit{EnDE}}$ feature. Though accuracy only is not sufficient to decide the superiority of a model. Thus precision, recall and F1 score has also been calculated. It has been observed that precision, recall and F1-Score of CNN-BiGRU is above 80% in most of the emotion indices. The confusion matrix obtained by proposed CNN-BiGRU is shown in Fig. B1 (Appendix B) for different emotions present in the SEED and DREAMER datasets.

Further, the comparison of proposed model is done on the basis of average accuracies. The average accuracies for SEED and DREAMER dataset are calculated by Eqs (18) and (19) respectively.

$\displaystyle\textit{Acc}_{\textit{avg}}=({P+Ng+N})/3$ (18) $\displaystyle\textit{Acc}_{\textit{avg}}=({V+A+D})/3$ (19)

A comparison of the hybrid CNN-BiGRU model with the CNN and BiGRU model for both datasets is shown in Fig. 10. From Fig. 10, the average accuracies obtained by CNN-BiGRU are found to be higher than a single CNN or BiGRU model.

Figure 10.

Average accuracy of CNN, BiGRU and hybrid CNN-BiGRU model for SEED dataset and DREAMER dataset.

The average accuracy of all hybrid DL models is compared in Fig. 11. The average accuracies obtained by the CNN-BiGRU model are found to be the highest for both datasets. A comparison of the proposed model with existing work is presented in Table 6. The table shows that DL models GRU and LSTM outperformed other models with an average accuracy of 82% [54, 55, 56, 57]. Fused feature models are more accurate for the SEED dataset [57, 58]. Matching the previous research, the proposed hybrid DL model with fused features achieved the highest average accuracy of 86.9% in the SEED dataset.

Figure 11.

Average accuracy of the different hybrid models in (a) SEED and (b) DREAMER.

Table 6

Comparison of the current proposed work with the existing work

Datasets	Ref, year	Feature extraction	Classification	Average accuracy (%)
SEED	[57, 2021]	Fused features (nodal degree entropy and clustering coefficient entropy)	SVM	68.4
	[58, 2021]	Fused features (holographic and topographic feature map)	CNN $+$ SVM	73.1
	[54, 2019]	Encode-decoder	GRU	74.4
	[56, 2021]	Energy threshold-based multi common spatial pattern	LSTM	82.0
	[55, 2022]	End-to-end	CNN	76.7
	Proposed model	Fused feature( $\overrightarrow{\textit{EnDE}}$ )	CNN-BiGRU	86.9
DREAMER	[52, 2017]	PSD	SVM	62.33
	[59, 2018]	DE	Dynamical graph convolutional neural networks (DGCNN)	85.2
	[60, 2019]	RGB heat map	CNN $+$ Extreme Learning Machines (ELM)	79.11
	[61, 2021]	fused feature (kurtosis, band power, DE)	Dilated Fully Convolutional Network (DFCN)	80.3
	[58, 2021]	Holographic feature map (HOLO-FM)	CNN $+$ SVM	89.31
	Proposed model	Fused feature( $\overrightarrow{\textit{EnDE}}$ )	CNN-BiGRU	93.9

For the DREAMER dataset, from Table 6, it is observed that the ML method with PSD as a feature results in the lowest accuracy in a study by Katsigiannis and Ramzan [52]. At the same time, the hybrid DL and ELM methods improved average accuracy up to 79.11% [60]. Fused feature models with the DL method increased average accuracy to 80.3% [61], while the proposed hybrid DL model with fused features increased average accuracy to 93.9%.

From the various related work, it is observed that time-frequency features are more suitable in EEG-based ER tasks [11, 37, 62]. Besides ER time-frequency feature such as global wavelet spectrum (GWS) are also applied for epilepsy detection from EEG signals and provide promising results [63]. Thus, in the presented work, DWT is used for band extraction. After band extraction, fused features were taken for the experiment, as they give better accuracies than single features [64, 65]. In addition to emotion recognition, fused feature models are also used in other research work [66]. In many recent works [31, 33, 35, 41], ML and DL based model are implemented in ER for EEG signals. Beside these, ML and DL models are also implemented in other task such as epileptic seizure classification, for diagnosing Asthma and Chronic Obstructive Pulmonary Disease, for classification of Patau, Edwards, Down, Turner and Klinefelter Syndrome [67, 68, 69, 70].

However, single DL models are not efficient in finding the detailed context of the problem, while hybrid DL models are more accurate in ER [71, 72]. The same is found in other applications such as stress detection and vein pattern recognition [73, 74]. In this work, BiGRU is taken over GRU model as it helps in preserving important features in both directions [75, 76]. Thus, a hybrid CNN-BiGRU model with fused features $\overrightarrow{\textit{EnSE}}$ and $\overrightarrow{\textit{EnDE}}$ results as the best model for EEG-based ER.

The accuracy and loss graph of the proposed model CNN-BiGRU is shown in Fig. 12.

Figure 12.

(a) Accuracy and (b) Loss graph for the proposed model for the SEED dataset.

6. Conclusion

This work proposes an EEG-based hybrid DL model for ER, namely CNN-BiGRU. The above experiments and discussions claim that DL models are more efficient than basic ML models for EEG signal classification. In this work, different experiments and analyses were performed using a hybrid DL model – CNN was used to extract spatial information, which was fed into the BiGRU to retrieve temporal features. The suggested model’s outcomes in terms of average accuracy were compared to CNN-LSTM, CNN-BiLSTM and CNN-GRU. CNN-BiGRU achieved the best performance. This means that temporal characteristics produced from a BiGRU can successfully reflect EEG data diversity.

The present work’s experimental results show that fused features give the highest emotion classification accuracy in all three dimensions, i.e. positive, negative, neutral, and VAD dimensions. Though hybrid DL models are efficient in emotion recognition. However, are complex and computationally expensive whilst experimenting. However, attention based methods can be used in future work to boost the efficiency of the BiGRU-based network. Furthermore, Multimodal work may be used to learn from modalities.

Footnotes

Conflict of interest

None to report.

Supplementary data

The supplementary files are available to download from http://dx.doi.org/10.3233/THC-220458.

References

Thompson

. Emotion regulation: A theme in search of definition. Monographs of the Society for Research in Child Development. 1994 Jan 1: 25-52.

De Gelder

De Borst

Watson

. The perception of emotion in body expressions. Wiley Interdisciplinary Reviews: Cognitive Science. 2015 Mar; 6(2): 149-158.

Egger

Ley

Hanke

. Emotion recognition from physiological signal analysis: A review. Electronic Notes in Theoretical Computer Science. 2019 May 4; 343: 35-55.

Mohammad

Nishida

. Using physiological signals to detect natural interactive behavior. Applied Intelligence. 2010 Aug; 33(1): 79-92.

Critchley

Garfinkel

. The influence of physiological signals on cognition. Current Opinion in Behavioral Sciences. 2018 Feb 1; 19: 13-18.

Picard

Vyzas

Healey

. Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2001 Oct; 23(10): 1175-1191.

Feidakis

. A review of emotion-aware systems for e-learning in virtual environments. Formative Assessment, Learning Data Analytics and Gamification. 2016 Jan 1: 217-242.

Vasiljevic

de Miranda

. Brain-computer interface games based on consumer-grade EEG Devices: A systematic literature review. International Journal of Human-Computer Interaction. 2020 Jan 20; 36(2): 105-142.

Padhmashree

Bhattacharyya

. Human emotion recognition based on time-frequency analysis of multivariate EEG signal. Knowledge-Based Systems. 2022 Feb 28; 238: 107867.

10.

Zhang

Yin

Chen

Nichele

. Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review. Information Fusion. 2020 Jul 1; 59: 103-126.

11.

Islam

Moni

Islam

Rashed-Al-Mahfuz

Islam

Hasan

Hossain

Ahmad

Uddin

Azad

Alyami

. Emotion recognition from EEG signal focusing on deep learning and shallow learning techniques. IEEE Access. 2021 Jun 22; 9: 94601-94624.

12.

Şengür

Siuly

. Efficient approach for EEG-based emotion recognition. Electronics Letters. 2020 Dec; 56(25): 1361-1364.

13.

Soroush

Maghooli

Setarehdan

Nasrabadi

. A review on EEG signals based emotion recognition. International Clinical Neuroscience Journal. 2017 Oct 1; 4(4): 118.

14.

Calvo

D’Mello

. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing. 2010 Jan; 1(1): 18-37.

15.

Ekman

Friesen

. Unmasking the face: A guide to recognizing emotions from facial clues. Ishk; 2003.

16.

Feldman

. Valence focus and arousal focus: Individual differences in the structure of affective experience. Journal of Personality and Social Psychology. 1995 Jul; 69(1): 153.

17.

Russell

. A circumplex model of affect. Journal of Personality and Social Psychology. 1980 Dec; 39(6): 1161.

18.

Mittal

Bhattacharya

Chandra

Bera

Manocha

. M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34, No. 02, 2020 Apr 3. pp. 1359-1367.

19.

Joshi

Ghongade

. EEG based emotion detection using fourth order spectral moment and deep learning. Biomedical Signal Processing and Control. 2021 Jul 1; 68: 102755.

20.

Zhang

Wang

. Relation classification: Cnn or rnn? In Natural Language Understanding and Intelligent Applications. Springer, Cham. 2016 Dec 2. pp. 665-675.

21.

Sherstinsky

. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Physica D: Nonlinear Phenomena. 2020 Mar 1; 404: 132306.

22.

Dey

Salem

. Gate-variants of gated recurrent unit (GRU) neural networks. In 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS). IEEE. 2017 Aug 6, pp. 1597-1600.

23.

Lin

Wang

Jung

Jeng

Duann

Chen

. EEG-based emotion recognition in music listening. IEEE Transactions on Biomedical Engineering. 2010 May 3; 57(7): 1798-1806.

24.

Nawaz

Cheah

Nisar

Yap

. Comparison of different feature extraction methods for EEG-based emotion recognition. Biocybernetics and Biomedical Engineering. 2020 Jul 1; 40(3): 910-926.

25.

Huang

Zhang

. EEG-based emotion recognition using empirical wavelet transform. In 2017 4th International Conference on Systems and Informatics (ICSAI). IEEE. 2017 Nov 11. pp. 1444-1449.

26.

Mohammadi

Frounchi

Amiri

. Wavelet-based emotion recognition system using EEG signal. Neural Computing and Applications. 2017 Aug; 28(8): 1985-1990.

27.

Momennezhad

. EEG-based emotion recognition utilizing wavelet coefficients. Multimedia Tools and Applications. 2018 Oct; 77(20): 27089-27106.

28.

Chen

Zhang

Mao

Huang

Jiang

Zhang

. Accurate EEG-based emotion recognition on combined features using deep convolutional neural networks. IEEE Access. 2019 Mar 29; 7: 44317-44328.

29.

Gao

Wang

Song

Dong

Song

. EEG based emotion recognition using fusion feature extraction method. Multimedia Tools and Applications. 2020 Oct; 79(37): 27057-27074.

30.

Kaur

Singh

Roy

. EEG based emotion classification mechanism in BCI. Procedia Computer Science. 2018 Jan 1; 132: 752-758.

31.

Lakhan

Banluesombatkul

Changniam

Dhithijaiyratn

Leelaarporn

Boonchieng

Hompoonsup

Wilaiprasitporn

. Consumer grade brain sensing for emotion recognition. IEEE Sensors Journal. 2019 Jul 15; 19(21): 9896-9907.

32.

Qing

Qiao

Cheng

. Interpretable emotion recognition using EEG signals. IEEE Access. 2019 Jul 15; 7: 94160-94170.

33.

César Cavalcanti Roza

Adrian Postolache

. Multimodal approach for emotion recognition based on simulated flight experiments. Sensors. 2019 Dec 13; 19(24): 5516.

34.

Kusumaningrum

Faqih

Kusumoputro

. Emotion recognition based on DEAP database using EEG time-frequency features and machine learning methods. In Journal of Physics: Conference Series. IOP Publishing. Vol. 1501, No. 1, 2020 Mar 1. p. 012020.

35.

Acharya

Goel

Bhardwaj

Sakalle

Bhardwaj

. A long short term memory deep learning network for the classification of negative emotions using EEG signals. In 2020 International Joint Conference on Neural Networks (IJCNN). IEEE. 2020 Jul 19. pp. 1-8.

36.

Cheng

Chen

Liu

Song

Liu

Chen

. Emotion recognition from multi-channel eeg via deep forest. IEEE Journal of Biomedical and Health Informatics. 2020 May 19; 25(2): 453-464.

37.

Kamble

Sengupta

. Ensemble machine learning-based affective computing for emotion recognition using dual-decomposed EEG signals. IEEE Sensors Journal. 2021 Dec 15; 22(3): 2496-2507.

38.

Subasi

Tuncer

Dogan

Tanko

Sakoglu

. EEG-based emotion recognition using tunable Q wavelet transform and rotation forest ensemble classifier. Biomedical Signal Processing and Control. 2021 Jul 1; 68: 102648.

39.

Iyer

Das

Teotia

Maheshwari

Sharma

. CNN and LSTM based ensemble learning for human emotion recognition using EEG recordings. Multimedia Tools and Applications. 2022 Apr 11: 1-14.

40.

Wang

Zhang

Yin

Song

. Spatial-Temporal Feature Fusion Neural Network for EEG-Based Emotion Recognition. IEEE Transactions on Instrumentation and Measurement. 2022 Apr 7; 71: 1-12.

41.

Algarni

Saeed

Al-Hadhrami

Ghabban

Al-Sarem

. Deep Learning-Based Approach for Emotion Recognition Using Electroencephalography (EEG) Signals Using Bi-Directional Long Short-Term Memory (Bi-LSTM). Sensors. 2022 Apr 13; 22(8): 2976.

42.

Fukushima

Miyake

Ito

. Neocognitron: A neural network model for a mechanism of visual pattern recognition. IEEE Transactions on Systems, Man, and Cybernetics. 1983; Sep(5): 826-834.

43.

LeCun

Boser

Denker

Henderson

Howard

Hubbard

Jackel

. Backpropagation applied to handwritten zip code recognition. Neural Computation. 1989 Dec; 1(4): 541-551.

44.

Liu

Wang

Zhao

Xin

Wang

. Subject-independent emotion recognition of EEG signals based on dynamic empirical convolutional neural network. IEEE/ACM Transactions on Computational Biology and Bioinformatics. 2020 Aug 24; 18(5): 1710-1721.

45.

Phung

Rhee

. A high-accuracy model average ensemble of convolutional neural networks for classification of cloud image patches on small datasets. Applied Sciences. 2019 Oct 23; 9(21): 4500.

46.

Cho

Van Merriënboer

Gulcehre

Bahdanau

Bougares

Schwenk

Bengio

. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. 2014 Jun 3.

47.

Zhu

Dai

. Speech emotion recognition model based on Bi-GRU and Focal Loss. Pattern Recognition Letters. 2020 Dec 1; 140: 358-365.

48.

Rana

. Gated recurrent unit (GRU) for emotion classification from noisy speech. arXiv preprint arXiv:1612.07778. 2016 Dec 13.

49.

Chen

Jiang

Zhang

. A hierarchical bidirectional GRU model with attention for EEG-based emotion classification. IEEE Access. 2019 Aug 22; 7: 118530-118540.

50.

Zheng

. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development. 2015 May 8; 7(3): 162-175.

51.

Dadebayev

Goh

Tan

. EEG-based emotion recognition: Review of commercial EEG devices and machine learning techniques. Journal of King Saud University-Computer and Information Sciences. 2021 Apr 1.

52.

Katsigiannis

Ramzan

. DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE Journal of Biomedical and Health Informatics. 2017 Mar 27; 22(1): 98-107.

53.

Sundararajan

. Discrete wavelet transform: a signal processing approach. John Wiley & Sons; 2016 Mar 7.

54.

Jolly

Aggrawal

Nath

Gupta

Grover

Shah

. Universal eeg encoder for learning diverse intelligent tasks. In 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM). IEEE. 2019 Sep 11. pp. 213-218.

55.

Rajpoot

Panicker

. Subject independent emotion recognition using EEG signals employing attention driven neural networks. Biomedical Signal Processing and Control. 2022 May 1; 75: 103547.

56.

Kim

Nguyen

Yang

Lee

. eRAD-Fe: Emotion recognition-assisted deep learning framework. IEEE Transactions on Instrumentation and Measurement. 2021 Sep 23; 70: 1-12.

57.

Yao

Wang

Zhang

. EEG-based emotion recognition by exploiting fused network entropy measures of complex networks across subjects. Entropy. 2021 Jul 30; 23(8): 984.

58.

Topic

Russo

. Emotion recognition based on EEG feature maps through deep learning network. Engineering Science and Technology, an International Journal. 2021 Dec 1; 24(6): 1442-1454.

59.

Song

Zheng

Song

Cui

. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing. 2018 Mar 21; 11(3): 532-541.

60.

Siddharth

Jung

Sejnowski

. Utilizing deep learning towards multi-modal bio-sensing and vision-based affective computing. IEEE Transactions on Affective Computing. 2019 May 14.

61.

Chai

Wang

Yang

. EEG emotion recognition based on 3-D feature representation and dilated fully convolutional networks. IEEE Transactions on Cognitive and Developmental Systems. 2021 Jan 13; 13(4): 885-897.

62.

Chen

Sun

Diao

Wang

. Happy or sad? Recognize emotions with wavelet coefficient energy mean of EEG signals. Technology and Health Care. 2022 Mar 18(Preprint): 1-13.

63.

Avdakovic

Omerhodzic

Badnjevic

Boskovic

. Diagnosis of epilepsy from EEG signals using global wavelet power spectrum. In 6th European Conference of the International Federation for Medical and Biological Engineering. Springer, Cham. 2015. pp. 481-484.

64.

Chen

Wan

Bezerianos

Wang

. Fusing frequency-domain features and brain connectivity features for cross-subject emotion recognition. IEEE Transactions on Instrumentation and Measurement. 2022 Apr 20; 71: 1-15.

65.

Yang

Hou

Kang

Liu

Song

Gao

Dong

. EEG-based emotion recognition with haptic vibration by a feature fusion method. IEEE Transactions on Instrumentation and Measurement. 2022 Jan 31; 71: 1-11.

66.

Lin

Xue

Shen

. Learning correlation filter with fused feature and reliable response for real-time tracking. Journal of Real-Time Image Processing. 2022 Apr; 19(2): 417-427.

67.

Sarić

Jokić

Beganović

Pokvić

Badnjević

. FPGA-based real-time epileptic seizure classification using Artificial Neural Network. Biomedical Signal Processing and Control. 2020 Sep 1; 62: 102106.

68.

Badnjevic

Gurbeta

Custovic

. An expert diagnostic system to automatically identify asthma and chronic obstructive pulmonary disease in clinical settings. Scientific Reports. 2018 Aug 3; 8(1): 1-9.

69.

Catic

Gurbeta

Kurtovic-Kozaric

Mehmedbasic

Badnjevic

. Application of Neural Networks for classification of Patau, Edwards, Down, Turner and Klinefelter Syndrome based on first trimester maternal serum screening data, ultrasonographic findings and patient demographics. BMC Medical Genomics. 2018 Dec; 11(1): 1-2.

70.

Shahamiri

Thabtah

Abdelhamid

. A new classification system for autism based on machine learning of artificial intelligence. Technology and Health Care. 2022 Jan 1; 30(3): 605-22.

71.

Patlar Akbulut

. Hybrid deep convolutional model-based emotion recognition using multiple physiological signals. Computer Methods in Biomechanics and Biomedical Engineering. 2022 Jan 24: 1-13.

72.

Zhao

Jiang

Feng

Yang

Wang

Hou

Wang

. A deep learning algorithm based on 1D CNN-LSTM for automatic sleep staging. Technology and Health Care. 2022 Jan 1(Preprint): 1-4.

73.

Malviya

Mal

. A novel technique for stress detection from EEG signal using hybrid deep learning model. Neural Computing and Applications. 2022 Jul 6: 1-12.

74.

Gopinath

Shivakumar

. Classification of vein pattern recognition using hybrid deep learning. Journal of Intelligent & Fuzzy Systems. (Preprint): 1-9.

75.

Orjesek

Jarina

Chmulik

. End-to-end music emotion variation detection using iteratively reconstructed deep features. Multimedia Tools and Applications. 2022 Feb; 81(4): 5017-5031.

76.

Zhang

Cheng

Zhang

. Chinese medical question answer selection via hybrid models based on CNN and GRU. Multimedia Tools and Applications. 2020 Jun; 79(21): 14751-14776.

A novel convolution bi-directional gated recurrent unit neural network for emotion recognition in multichannel electroencephalogram signals

Abstract

BACKGROUND:

OBJECTIVE:

METHODS:

RESULTS:

CONCLUSION:

Keywords

1. Introduction

Table 1 Related work on ER in last five years

3.1 CNN

4.1 Datasets

4.1.1 SEED

4.1.2 DREAMER

Table 2 Dimensions of data and labels present in SEED and DREAMER datasets

4.6 Classification of emotions

Table 4 Input and output data dimensions fed to classification model

Table 5 Parameters value used to build the CNN–BiGRU model

Footnotes

Conflict of interest

Supplementary data

References

Table 1
Related work on ER in last five years

Table 2
Dimensions of data and labels present in SEED and DREAMER datasets

Table 4
Input and output data dimensions fed to classification model

Table 5
Parameters value used to build the CNN–BiGRU model