Electroencephalography based human emotion state classification using principal component analysis and artificial neural network

Abstract

In recent decades, the automatic emotion state classification is an important technology for human-machine interactions. In Electroencephalography (EEG) based emotion classification, most of the existing methodologies cannot capture the context information of the EEG signal and ignore the correlation information between dissimilar EEG channels. Therefore, in this study, a deep learning based automatic method is proposed for effective emotion state classification. Firstly, the EEG signals were acquired from the real time and databases for emotion analysis using physiological signals (DEAP), and further, the band-pass filter from 0.3 Hz to 45 Hz is utilized to eliminate both high and low-frequency noise. Next, two feature extraction techniques power spectral density and differential entropy were employed for extracting active feature values, which effectively learn the contextual and spatial information of EEG signals. Finally, principal component analysis and artificial neural network were developed for feature dimensionality reduction and emotion state classification. The experimental evaluation showed that the proposed method achieved 96.38% and 97.36% of accuracy on DEAP, and 92.33% and 89.37% of accuracy on a real-time database for arousal and valence emotion states. The achieved recognition accuracy is higher compared to the support vector machine on both databases.

Keywords

Artificial neural network differential entropy emotion state classification power spectral density principal component analysis

1. Introduction

Human emotions are psychological experiences, which are characterized by the mental activities that entail synchronized features such as action tendency, response, expression, and knowledge [1, 2]. Generally, human emotions are recognized by body postures, facial expressions, speech, physiological activities, etc. [3, 4, 5]. The available techniques are based on the expressed emotions, which do not capture the innermost feeling of the humans. However, the Electroencephalography (EEG) signal superiorly provides the human emotional pattern and secret information [6, 7]. Since, due to the synchronized firing of the neurons, human emotions originate from the peripheral and central nervous systems, which cause transient agitation. The visual EEG signal analysis is a tedious and time-consuming procedure, and it leads to false point detection [8, 9]. In recent periods, many researchers implemented several machine learning methods: random forest, k-nearest neighbor, decision tree, etc. for emotion state classification using bio-signals [10, 11]. Compared to the conventional machine learning methods, the deep learning methods provide successful results in the emotion state classification using EEG signals [12]. Still, deep learning methods need to solve the non-stationary characteristics of the EEG signal [13, 14]. To enhance the emotion classification, a novel dimensionality reduced deep learning method is implemented in this article. The contributions of this article are stated below:

•
Initially, the EEG signals are acquired from databases for emotion analysis using physiological signals (DEAP) and a real time database. Further, the bandpass filter from 0.3 Hz to 45 Hz frequency is applied to eliminate higher and lower frequency noises like electrooculography from the acquired EEG signals.
•
In addition, a hybrid feature extraction: Power Spectral Density (PSD) and differential entropy techniques are utilized for extracting feature values from the denoised EEG signals. In this application, the hybrid feature extraction includes the benefits such as improved data visualization, over-fitting risk reduction, speed up the training process, and accuracy improvement in the classification model.
•
The extracted multi-dimensional feature values are optimized by the Principal Component Analysis (PCA) method to improve the computational time and system complexity of the proposed frame-work. Additionally, the feature dimensionality reduction enhances the interpretation of the hyper-parameters in the Artificial Neural Network (ANN) method.
•
Lastly, the ANN method is applied for emotion classification as arousal and valence. The PCA based ANN method’s performance is evaluated in terms of F1 score, recall, and recognition accuracy.

This article is prepared as follows: some papers related to emotion state classification are surveyed in Section 2. The theoretical and the mathematical explanation of the PCA based ANN method is denoted in Section 3. The experimental results and the conclusion of the PCA based ANN method is stated in Sections 4 and 5.
2. Literature review

Yin et al. [15] initially used differential entropy for extracting feature values from the EEG data, and then the extracted feature values were given to the deep learning models: Long Short Term Memory (LSTM) network and Graph Convolutional Neural Network (GCNN) to achieve efficient emotion classification result. In this study, the developed model’s efficiency was tested on the DEAP database in terms of accuracy. However, the presented deep learning model was too expensive for practical problems. Liu and Fu [16] integrated relief and PCA techniques for selecting significant signal channels. Further, the frequency and time domain feature extraction techniques were employed for extracting active feature values from the acquired EEG data. Finally, the extracted feature values were given to the Support Vector Machine (SVM) for emotion classification. The experiment conducted on the DEAP database showed the efficiency of the presented model. The SVM classifier supports only binary class, which was the main issue in this literature. Subasi et al. [17] implemented a noise reduction methodology: Discrete Wavelet Transform (DWT) to eliminate noise from the acquired EEG signals. Next, Tunable Q Wavelet Transform (TQWT) and Multi-Scale PCA (MS-PCA) were applied for discriminative feature extraction and dimensionality reduction. Lastly, the rotation forest ensemble classifier was utilized for emotion classification [41, 42]. However, the computational complexity of the presented model was higher, due to the incorporation of several models.

Salankar et al. [18] integrated both 2nd order difference plot and the Empirical Mode Decomposition (EMD) technique for extracting feature values from DEAP database. Further, the Multi-Layer Perceptron (MLP) classification technique was used for emotion recognition. The simulation result revealed that the developed model was useful for the clinical analysis of high and low dominance regions in the patients, whereas the labelling procedure was hard in the MLP classifier compared to other learning models. Song et al. [19] developed a Graph Embedded CNN (GECNN) model for EEG based emotion classification. The GECNN model was computationally costly because it required higher end graphics processing units for data validation. Gao et al. [20] integrated power spectral densities, sample entropy, differential and hjorth entropy for feature extraction. Then, GoogleNet and SVM techniques were utilized for feature fusion and emotion classification. The experimental results on the DEAP database showed that the developed model has attained excellent performance by means of recognition accuracy. Hence, the SVM was inappropriate for multi-class classification, because it mainly supports binary class classification. Shen et al. [21] presented a new multi-scale frequency band based ensemble learning method for emotion classification utilizing EEG signals. The experimental result represents the efficiency of the presented model in emotion classification. Still, it was essential to find the optimal frequency band scale to enhance emotion detection.

Sharma et al. [22] initially decomposed the acquired EEG signals using DWT technique, and then Particle Swarm Optimization (PSO) technique and LSTM network were applied for data reduction and emotion classification. In this literature, the presented model’s effectiveness was evaluated on the DEAP database with a ten-fold cross validation method. The simulation result confirmed that the presented model has the potential for rapid and accurate recognition of human emotions. In addition, Yang et al. [23] implemented a multi-column CNN model for emotion classification by using different bio-signals. The developed model achieved high classification accuracy on the DEAP database compared to the traditional models. As denoted earlier, the CNN and LSTM needed a larger number of training data that was computationally costly. Mert and Akan [24] integrated Multivariate EMD (MEMD) and Independent Component Analysis (ICA) for feature extraction and dimensionality reduction. Additionally, the dimensionally reduced feature values were given to the ANN for emotion classification as low/high valence and low/high arousal states. However, the linear decomposition delivers an in-appropriate analysis in detecting emotions from non-stationary signals. Song et al. [25] developed a Dynamical Graph CNN (DGCNN) model for multi-channel EEG based emotion classification. The simulation result demonstrated that the developed DGCNN model attained a significant classification performance than the prior models in terms of recognition accuracy. However, over-fitting was the major problem faced by the researchers in this literature.

Chen et al. [26] integrated approximation entropy and EMD technique for extracting feature values from the acquired EEG signal. Further, the extracted feature values were given to the Deep Belief Network (DBN) for classifying human emotions like fear, sadness, happiness, and calm. The simulation results demonstrated that the presented model attained higher accuracy than the existing models. The reduction of interference and the selection of effective emotion properties needed to be concentrated as a future work to further improve emotion recognition. He et al. [27] developed a Firefly Integrated Optimization Algorithm (FIOA) for effective emotion classification, whereas the computation was an imperative factor in this literature. Chao et al. [28] implemented a novel method based on a capsule network and multi-band feature matrix for emotion classification using EEG signal. The simulation outcomes showed that the capsule network obtained satisfactory results on the DEAP database. However, it was difficult to integrate spatial characteristics and common characteristics of the EEG signals. In addition, Ullah et al. [29] integrated sparse projection coefficients and the PCA technique for effective emotion classification. The experimental results on the online databases confirmed the superiority of the developed model over the conventional models. The trade-off between dimensionality reduction and information loss was a major concern in the PCA technique.

Li et al. [30] implemented Hierarchical CNN (HCNN) model for classifying negative, neutral, and positive emotional states. In this literature, the HCNN model determines the higher frequency waves like Gamma and Beta for processing the human emotions, since this process was computationally expensive. Additionally, Pandey and Seeja [31] combined Deep Neural Network (DNN) and Variational Mode Decomposition (VMD) technique for emotion classification. The simulation examination on the DEAP database confirmed the effectiveness of the developed model by means of recognition accuracy. However, the DNN model needed a larger number of data to perform better classification than the existing models. Liu et al. [32] integrated both capsule network and multilevel features for effective human emotion detection. As denoted earlier, the capsule network faces difficulty in integrating the common and spatial characteristics of the EEG signal. Zhang et al. [33] used two logistic regression models and a sparse autoencoder method for valence and arousal classification. The test results showed that the presented model attained acceptable classification performance on the DEAP database. The assumption of linearity between the independent and dependent variables was a main concern in the logistic regression models.

Aboneh et al. [43] has implemented a stack-based ensemble learning model for land use and land cover image classification. In this literature study, the implemented ensemble learning model integrates XGBoost classifier for improving the classification accuracy. The performance of the implemented ensemble learning model was tested on a real time database, which was collected from Bishoftu town, Ethiopia. The main objective of this literature study was to analyze the performance of land use and land cover classification on the multi-spectral images, but the implemented model was computationally complex. Alexandropoulos et al. [44] developed a hybrid decision support system based on the ensemble of classifiers in order to make strong predictions. In this literature study, the developed hybrid decision support system’s performance was compared to other existing known classifiers. In the resulting segment, the extensive experiments conducted on the standard benchmark databases showed the effectiveness of the developed hybrid decision support system. Kwon et al. [45] integrated machine learning classifiers such as generalized linear model, gradient boosted model, deep neural network and distributed random forest for breast cancer classification. As specified in the resulting section, the developed stacking ensemble model obtained higher classification accuracy, but computationally expensive and complex. For highlighting the above-stated concerns, a novel PCA based ANN method is implemented for effective human emotion classification.

3. Principal Component Analysis (PCA) based Artificial Neural Network (ANN) method

Generally, human emotion states are recognized from body movement, facial expression and speech, but the EEG signal is effective in identifying the human emotion state because it directly records and estimates the electrical activity of the brain. Additionally, the EEG signal estimates the voltage fluctuation in the brain that helps in extracting the essential information about the human emotion state. The proposed framework includes five phases signal collection: real-time and DEAP database, pre-processing: band-pass filter, feature extraction: differential entropy and PSD, feature dimensional reduction: PCA and emotion state classification: ANN. The schematic diagram of the proposed framework is stated in Fig. 1.

Figure 1.

Schematic diagram of the proposed framework.

3.1 Signal acquisition

In this article, the PCA based ANN method’s effectiveness is tested on DEAP and real time databases. First, the DEAP database comprises physiological EEG signals of thirty two individuals (16 male and female individuals), who range between 19 to 37 age. In the DEAP database, the EEG signal is recorded, when the individuals watch a 42-minute music video on different arguments. In this database, the EEG signal is sampled at 512 Hz frequency, and the signal is recorded from 32 locations like CP1, FC2, FC6, CP2, FC1, CP6, Fp1, AF3, Fp2, T7, FC5, P4, PO3, AF4, F8, P7, CP5, F3, T8, PO4, P3, Pz, Fz, P8, F4, C3, C4, F7, O1, Oz, Cz and O2. Few properties of the DEAP database are: number of individuals is 32, number of channels is 32, number of video/stimuli is 40, sampling rate is 128 Hz frequency, and labels (valence and arousal). DEAP database link: https://www.eecs.qmul.ac.uk/mmv/datasets/deap/ [34, 35].

In the real time database, the peripheral physiological EEG signals are acquired from 50 individuals, whose age ranges from 9 to 32. Few important characteristics of the real time database are: number of individuals is 50, number of channels is 50, number of video/stimuli is 2, sampling rate is 128 Hz frequency and labels (valence and arousal). The sample acquired EEG signals are depicted in Fig. 2.

Figure 2.

Sample acquired EEG signals.

3.2 Signal pre-processing and extraction of feature values

After signal denoising, a bandpass filter from 0.3 Hz to 45 Hz is employed for eliminating high and low frequency noises like electrooculography. The pre-processed EEG data is given to the feature extraction techniques differential entropy and PSD for extracting feature values from the denoised EEG signals. The differential entropy is applied to find the local non-linear information between adjacent elements, which are used to achieve a non-linear complexity of the EEG signals. The feature values extracted from the differential entropy are appropriate for classifying the human emotion states (valence and arousal) by using the energy spectrum and it is mathematically determined in Eq. (1).

$\displaystyle h(X)=\mathop{\smallint}\nolimits_{x}f(X)\log f(X)dX$ (1)

where $f(X)$ indicates probability density function of $X$ and $X$ represents random value. The Gaussian distribution function $N({{\mu},\sigma^{2}})$ of time series data is fixed as constant and the above-stated equation is re-written in Eq. (2).

$\displaystyle h(X)=-\mathop{\smallint}\limits_{-\infty}^{\infty}\frac{1}{\sqrt% {2\pi\sigma^{2}}}e^{\frac{-({x-\mu})^{2}}{2\sigma^{2}}}\log\left({\frac{1}{% \sqrt{2\pi\sigma^{2}}}e^{\frac{-({x-\mu})^{2}}{2\sigma^{2}}}}\right)dX$ (2)

By using Eq. (2), the discriminating pattern of the EEG signal is balanced between the high and low-frequency bands. Additionally, the power spectral properties of the EEG signals are divided into 5 frequency bands theta 4–8 Hz, lower-alpha 8–10 Hz, higher alpha 10–12 Hz, beta 12–30 Hz, and gamma 30 Hz. In this manuscript, the fourteen pairs of electrodes on the left and right hemisphere is chosen to estimate the asymmetry potential of the brain area for enhancing the power spectral properties of the EEG signal. Around 670 feature values are extracted using PSD and differential entropy, which are fed to the PCA technique for feature dimensionality reduction [36, 37].

3.3 Feature dimensionality reduction

The extracted 670 feature values are given to the PCA for dimensionality reduction by transforming the larger set of features into a smaller set of features with valuable information. The transformed feature values are named as principal components and it is a popular method used for predictive modelling and exploratory data analysis. The PCA is an effective method for drawing strong patterns from DEAP and real time databases by reducing the variance. In this scenario, the PCA concentrates on the co-variance and variance structure of the extracted feature values $x=x_{1},x_{2},\ldots x_{p}$ , which are multi-dimensional, because it has heavy weights [38].

The extracted 670 feature values are partitioned into different scales for decreasing the feature dimensions. Next, the correlation matrix $R$ is estimated for each principal component $p$ from $n$ observations. In the PCA method, the correlation matrix $R$ value and Eigenvectors are represented as $({\varepsilon_{1},e_{1}}),({\varepsilon_{2},e_{2}}),({\varepsilon_{3},e_{3}}),% ({\varepsilon_{4},e_{4}})\ldots({\varepsilon_{p},e_{p}})$ . The sample $p$ feature value is determined in Eq. (3).

$\displaystyle e_{i}Z=e_{i1}Z_{1}+e_{i2}Z_{2}+e_{i3}Z_{3}+?e_{i4}Z_{4},\ldots e% _{ip}Z_{p},i=1,2,3,4,\ldots p$ (3)

where $Z=Z_{1},Z_{2},Z_{3},Z_{4}\ldots Z_{p}$ indicates the observation of the standardized feature vector, and $e_{i}Z=({e_{i1},e_{i2},e_{i3},e_{i4}\ldots e_{ip}})$ represents $i^{\text{th}}$ Eigenvalue. The variance is indicated as $n_{i}$ and the covariance pair is zero in the PCA method. In this scenario, the sample variance value of all standardized feature vectors is equal to the variance value in the principal component $p$ . Though, the observation of the standardized feature vector is specified in Eq. (4).

$\displaystyle Z_{k}=\frac{x_{k}-\acute{x}_{k}}{\sqrt{v_{kk}}},k=1,2,3,4,5\ldots p$ (4)

where $x_{k}$ states sample mean and $v_{kk}$ represents sample variance of $x_{k}$ . Lastly, the optimum principal component or dimensionally reduced features of 429 are given to the ANN method for emotion classification as valence and arousal.

3.4 Human emotion state classification

The dimensionally reduced 429 feature values are given to the ANN method for emotion recognition. In this manuscript, the proposed ANN method comprises 2 softmax layers and 3 stacked autoencoders for recognition. The arousal and valence emotions are 2 softmax layers, where it shares the results of unlabelled raw EEG data. However, the input and output layers comprise numerous nodes which mainly depends on the input and output feature values. The dimensionally reduced feature values move between the layers in ANN across several weight connections. The current layer accepts the feature values from the previous layers and further, calculates the weighted sum of all the inputs $t$ that is defined in Eq. (5).

$\displaystyle t_{i}=\mathop{\sum}\limits_{j=1}^{n}W_{ij}X_{j}$ (5)

where $X_{j}$ specifies input of nodes $j$ , $n$ states dimensionally reduced 429 feature values, and $w$ states weighted connection between the nodes. Additionally, the node output $o_{i}$ is analyzed by integrating sigmoid activation function in the weighted value $t$ . The node output $o_{i}$ value is determined in Eq. (6).

$\displaystyle o_{i}=f({t_{i}})$ (6)

Figure 3.

Architecture of the ANN model.

As mentioned in Eq. (6), it holds only true neuron values with the linear function. Therefore, a non-linear function is employed for identifying the partial error sub-ordinates by utilizing chain rule [39], which is specified in Eqs (7) and (8).

$\displaystyle t_{i}=\emptyset(\textit{net})$ (7)

where $\textit{net}=\mathop{\sum}\limits_{j=1}^{n}W_{ij}X_{j}$

$\displaystyle\frac{\partial E}{\partial W_{ij}}=\frac{dE}{dt_{i}}\times\frac{% dt_{i}}{d_{\textit{net}}}\times\frac{d_{\textit{net}}}{\partial W_{ij}}$ (8)

where $\frac{\partial E}{\partial W_{ij}}$ , $\frac{dE}{dt_{i}}$ , $\frac{dt_{i}}{d_{\textit{net}}}$ , and $\frac{d_{\textit{net}}}{\partial W_{ij}}$ indicates error value, when the weight, output, weighted sum, and weight value is changed. The hyper-parameter assumption of the ANN classifier is given as follows: activation function is sigmoid function, error value is 0.01, momentum rate is 0.4, learning rate is 0.001, iteration number is 100, number of neurons in input, hidden, and output layers are 429, 40, and 2. The experimental examination of the developed PCA based ANN method is specified in the upcoming section. The architecture of the ANN model is represented in Fig. 3, and the pseudocode of the proposed PCA based ANN method is given in Algorithm 1.

Algorithm 1: Pseudocode of the proposed PCA based ANN method
Procedure: ANNs (Input, Neurons, Repeat)
Input $\leftarrow$ dimensionally reduced features by applying PCA on the collected databases.
Train ANNs
For Input $=$ 1 to End of Input do
For Neurons $=$ 1 to End of Neurons do
For Repeat $=$ 1 to End of Repeat do
End For
End For
Save best classification results on the basis of inputs
End For
Store library with best classifying ANN for each variable combination
End Procedure

4. Experimental analysis

In this scenario, the PCA based ANN method is implemented using a python software tool on a computer with 2 GB random access memory, windows 10 (64-bit operating system), Intel Core i5 processor and 1 TB hard disk. The developed PCA based ANN method efficiency was investigated in terms of F1-score, recall, and recognition accuracy on the DEAP and real-time databases. In the human emotion classification, the F1-score, recall, and recognition accuracy are the important statistical performance metrics, which are utilized to validate the efficiency of the proposed PCA based ANN method. The recognition accuracy is defined as the ratio of correct predictions to the total observations. Correspondingly, recall is defined as the ratio of correct positive predictions to the total positive observations. The mathematical illustration of the recognition accuracy and recall is depicted in the Eqs (9) and (10).

$\displaystyle\textit{Accuracy}=\frac{TP+TN}{TP+FP+TN+FN}$ (9) $\displaystyle\textit{Recall}=\frac{TP}{TP+FN}$ (10)

Similarly, the F1-score is defined as the harmonic mean of recall and precision value that is mathematically determined in Eq. (11), where FN indicates the false-negative examples and FP states the false positive examples of the error classification. TP states the true positive samples, which are precisely identified as low valence/arousal and TN states the true negative samples, which are precisely identified as high valence/arousal [40].

$\displaystyle\textit{F1-score}=\frac{2TP}{2TP+FP+FN}$ (11)

4.1 Quantitative analysis on the real-time database

In this segment, the PCA based ANN method’s efficiency is validated on the real time database. Further, dissimilar cross-fold validations are accomplished to analyze the efficiency of the proposed PCA based ANN method. The experimental outcome of the proposed method is analyzed for overall channels of valence and Arousal on the real time database. By viewing table 1, the results of over-all channels (FH, FS, FH $+$ FS (1), FH $+$ FS (1, 3), FH $+$ FS (1, 3, 4), and FH $+$ FS (1, 3, 4, 5) in valence is depicted for two classification methods such as SVM and ANN. By investigating Table 1, the ANN method attained maximum result in human emotion recognition with an F1-score of 74.55%, recall of 72.24% and accuracy of 89.37%. However, the comparative classification method (SVM) has attained limited results with an F1-score of 72.50%, recall of 71.23% and recognition accuracy of 82.56% on the real time database. The graphical evaluation of the proposed method in valence on the real time database is stated in Fig. 4.

Table 1
Experimental results of overall channels in valence on the real time database

Channels	ANN			SVM
	F1-score (%)	Recall (%)	Accuracy (%)	F1-score (%)	Recall (%)	Accuracy (%)
FH	52.40	34.57	81.39	61.24	50.14	79.23
FS	60.89	50.87	83.02	68.67	60.43	81.44
FH $+$ FS (1)	59.12	50.12	82.37	68.23	65.42	80.01
FH $+$ FS (1, 3)	69.88	59.12	85.94	69.70	67.56	80.72
FH $+$ FS (1, 3, 4)	70.12	66.16	87.75	71.45	69.90	80.29
FH $+$ FS (1, 3, 4, 5)	74.55	72.24	89.37	72.50	71.23	82.56

Figure 4.

Graphical evaluation of the proposed method in valence on the real time database.

Similar to Table 1, the experimental result of the proposed method is analysed for overall channels of Arousal on the real time database, which is indicated in Table 2. By investigating Table 2, the ANN classification technique with PCA attained maximum results on all channels related to the comparative SVM classification technique. In this scenario, the ANN classifier has attained an 84.47% of F1-score, 83.15% of recall and 92.33% of recognition accuracy in the human emotion classification. However, the comparative SVM method attained only 81.23% of F1-score, 82.36% of recall, and 89.97% of recognition accuracy. The graphical evaluation of the proposed method in Arousal on the real-time database is represented in Fig. 5.

Table 2

Experimental results of overall channels in Arousal on the real time database

Channels	ANN			SVM
	F1-score (%)	Recall (%)	Accuracy (%)	F1-score (%)	Recall (%)	Accuracy (%)
FH	52.30	33.57	83.93	54.24	52.14	76.54
FS	64.13	51.37	84.12	55.23	59.43	79.23
FH $+$ FS (1)	61.33	55.23	85.17	62.23	61.12	80.33
FH $+$ FS (1, 2)	77.88	60.25	86.35	70.70	63.36	81.12
FH $+$ FS (1, 2, 3)	78.24	68.26	88.14	73.34	69.10	83.39
FH $+$ FS (1, 2, 3, 4)	79.13	71.42	89.99	78.50	75.23	85.25
FH $+$ FS (1, 2, 3, 4, 5)	79.55	78.14	90.12	78.86	79.43	87.55
FH $+$ FS (1, 2, 3, 4, 5, 6)	80.35	79.42	91.17	79.11	81.44	89.11
FH $+$ FS (1, 2, 3, 4, 5, 6)	84.47	83.15	92.33	81.23	82.36	89.97

On the other hand, the proposed PCA based ANN method’s efficiency is tested on the real time multi-modal data such as body movements, facial expressions and EEG signals, and the obtained experimental results are compared with pre-trained models like AlexNet, and Graph convolutional networks. By investigating Table 3, the proposed PCA based ANN method achieved higher classification results compared to the existing models by means of recall and classification accuracy.

Table 3

Experimental results of the proposed PCA based ANN method and the existing models on the real time multi-modal data

Data	AlexNet		Graph convolutional network		PCA-ANN
	Recall (%)	Accuracy (%)	Recall (%)	Accuracy (%)	Recall (%)	Accuracy (%)
Body movements	50.12	74.04	52.18	76.80	58.19	79.87
Facial expressions	57.30	78.10	59.40	79.20	62.03	80.02
EEG signal	60.20	86.18	62.80	88.36	77.69	90.85
Multi-modal	72.28	90.82	74.30	92.80	82.03	93.20

Figure 5.

Graphical evaluation of the proposed method in Arousal on the real time database.

4.2 Quantitative analysis on the DEAP database

In this segment, the proposed PCA based ANN method’s efficiency is tested on the DEAP database. In the DEAP database, the PCA based ANN method utilizes 80:20% of the data for training and testing with a ten-fold cross-validation method. By viewing Table 4, the proposed PCA based ANN methodology attained f1-score of 96.79%, recall of 97.20% and accuracy of 97.36% in the EEG based emotion recognition. The obtained experimental outcome is higher compared to the PCA based SVM methodology, which achieved f1-score of 91.57%, recall of 91.23% and recognition accuracy of 92.52%. The graphical valuation of the proposed method in valence on the DEAP database is denoted in Fig. 6. In addition, the incorporation of PCA technique includes few benefits like low noise sensitivity, lack of data redundancy, and decreased requirements of memory and capacity.

Table 4
Experimental results of overall channels in valence on the DEAP database

Channels	ANN			SVM
	F1-score (%)	Recall (%)	Accuracy (%)	F1-score (%)	Recall (%)	Accuracy (%)
FH	62.40	64.70	71.30	62.28	60.18	71.20
FS	70.80	70.89	73.08	68.60	60	77.48
FH $+$ FS (1)	78.18	70.80	82.38	78.20	75.77	80
FH $+$ FS (1, 3)	89.82	89.75	88.98	79.55	77.50	87.60
FH $+$ FS (1, 3, 4)	90.11	93.60	94.88	87.48	89.90	90.20
FH $+$ FS (1, 3, 4, 5)	96.79	97.20	97.36	91.57	91.23	92.52

Figure 6.

Graphical evaluation of the proposed method in valence on the DEAP database.

The experimental results of the proposed method in Arousal on the DEAP database are stated in Table 5. By inspecting Table 5, the proposed method is individually investigated for over-all channels (FH, FS, FH $+$ FS (1), FH $+$ FS (1, 2), FH $+$ FS (1, 2, 3), FH $+$ FS (1, 2, 3, 4), FH $+$ FS (1, 2, 3, 4, 5), FH $+$ FS (1, 2, 3, 4, 5, 6), and FH $+$ FS (1, 2, 3, 4, 5, 6)) on the DEAP database. The PCA based ANN method has achieved a 96.40% of F1-score, 96.18% of recall, and 96.38% of recognition accuracy in the emotion classification on the DEAP database. However, the PCA with the SVM method obtained a 91.28% of F1-score, 94.44% of recall, and 94.96% accuracy. The graphical evaluation of the proposed method in Arousal on the DEAP database is represented in Fig. 7. In the EEG-based emotion classification, the ANN requires less statistical training and has the ability to detect complex relations between independent and dependent values, and all possible interactions between the predictor values to attain better classification results.

Table 5

Experimental results of overall channels in Arousal on the DEAP database

Channels	ANN			SVM
	F1-score (%)	Recall (%)	Accuracy (%)	F1-score (%)	Recall (%)	Accuracy (%)
FH	72.37	73.58	73.90	74.29	72.18	78.88
FS	74.40	81.30	84.66	75.28	79.46	78.28
FH $+$ FS (1)	81.38	85.20	75.18	82.28	81.18	82.37
FH $+$ FS (1, 2)	87.80	90.67	86.55	80.75	83.30	84.18
FH $+$ FS (1, 2, 3)	88.26	88.29	88.48	83.39	89.78	87.40
FH $+$ FS (1, 2, 3, 4)	89.16	91.47	89.97	88.58	85.20	87.28
FH $+$ FS (1, 2, 3, 4, 5)	89.50	88.80	90.10	88.80	89.40	88.58
FH $+$ FS (1, 2, 3, 4, 5, 6)	94.80	89.47	93.18	89.19	91.40	89.10
FH $+$ FS (1, 2, 3, 4, 5, 6)	96.40	96.18	96.38	91.28	94.44	94.96

Table 6

Comparative results between PCA based ANN method and the existing methods

Methods	Recognition accuracy (%)
	Arousal	Valence
LSTM with GCNN [15]	90.60	90.45
GoogleNet with SVM [20]	75.22	80.52
Sparse projection coefficients with PCA [29]	74.50	82.80
Two logistic regression with sparse autoencoder [33]	80.80	73.10
PCA based ANN	96.38	97.36

Figure 7.

Graphical evaluation of the proposed method in Arousal on the DEAP database.

4.3 Comparative evaluation

The comparative valuation between PCA based ANN method and the published methods are specified in Table 6. Yin et al. [15] utilized differential entropy as a feature extractor for extracting features from the EEG signals, which were acquired from the DEAP database. The hybrid deep learning models: LSTM and GCNN were implemented for emotion classification. The extensive experiments showed that the LSTM with the GCNN model achieved 90.60%, and 90.45% of accuracy for arousal and valence emotion states on the DEAP database. Gao et al. [20] combined GoogleNet and SVM for feature fusion and emotion classification. Hence, the experimental result on the DEAP database showed that the developed model has achieved 75.22% and 80.52% of recognition accuracy for arousal and valence emotion states.

Ullah et al. [29] used PCA and sparse projection coefficients for feature dimensionality reduction and emotion state classification. The experiments conducted on the DEAP database showed that the implemented model attained 74.50% and 82.80% of recognition accuracy for arousal and valence states on the DEAP database. Zhang et al. [33] integrated two logistic regression models and sparse auto-encoder for emotion state classification. The implemented model attained 80.80% and 73.10% of accuracy for arousal and valence emotion states on the DEAP database. Compared to these published methods, the PCA based ANN method attained significant classification performance in the human emotion state classification, whereas the proposed PCA based ANN method achieved 96.38% and 97.36% of accuracy for arousal and valence emotion states on the DEAP database. The comparative result is graphically specified in Fig. 8.

Figure 8.

Comparative evaluation of the proposed and the existing methods.

5. Conclusion

In this article, a PCA based ANN method is proposed for human emotion state classification. After the acquisition of EEG signals from real time and DEAP databases, the band-pass filter from 0.3 Hz to 45 Hz is employed for eliminating high and low-frequency noises. Further, the feature values were extracted from the denoised EEG signal utilizing differential entropy and PSD. The multi-dimensional feature values were decreased by implementing PCA and further the dimensionally reduced features were given to the ANN classifier for emotion detection as arousal and valence. The conducted experiments showed that the PCA based ANN achieved 96.38% and 97.36% of accuracy for arousal and valence states on the DEAP database. In the real time database, the PCA based ANN attained 92.33% and 89.37% of accuracy for arousal and valence emotion states. In addition, the PCA based ANN is computationally effective by decreasing the extracted feature dimension and it consumes 24.22 and 32.12 seconds for data training on the DEAP and real time databases. As a future extension, the multi-modal data such as body movements, facial expressions, and EEG signals can be given as the input to the proposed model to further enhance emotion state classification. In addition to this, an ensemble based deep learning model can be integrated with the proposed model to further improve classification accuracy.

Funding

This study was not funded by any organization.

Data availability statement

The datasets generated during and/or analysed during the current study are available in the [DEAPdataset] repository, [https://www.eecs.qmul.ac.uk/mmv/datasets/deap/].

Footnotes

Conflict of interest

The authors declare that they have no conflict of interest.

Author’s Bios

	Satyanarayana Naga V. Kanuboyina presently working as an assistant professor in Department of Electronics and Communication Engineering, S.R.K.R. Engineering College, Bhimavaram, A.P, India. He is currently pursuing PhD from Annamalai University. His current research interests include Image processing, Signal processing, and Internet of things.
	T. Shankar, received his B.E. (ECE), M.E. (PCI), and Ph.D (ECE) from Annamalai University, Chidambaram, Tamil Nadu, India. Presently, he is working as an Assistant Professor (Deputed from Annamalai University) in the Department of Electronics and Communication Engineering at Government College of Engineering, Srirangam, Tamil Nadu, India. He has published 28 research articles in reputed International Journals and presented reasearch work in various International Conferences. His area of interest includes Image and video Processing, Wireless Sensor Networks and Internet of Things.
	Rama Raju Venkata Penmetsa, worked as a Professor in Department of Electronics and Communication Engineering, S.R.K.R. Engineering College, A.P, India. His research interests include Biomedical Signal Processing, Signal Processing, Image Processing, VLSI Design, Antennas, and Microwave Anechoic Chambers Design. He is the author of several research studies published in national and international journals and conference proceedings.

References

Wei

Chen

L.L.

Song

Z.Z.

Lou

X.G.

and Li

D.D.

, EEG-based emotion recognition using simple recurrent units network and ensemble learning, Biomedical Signal Processing and Control 58 (2020), 101756. doi: 10.1016/j.bspc.2019.101756.

Qing

Qiao

and Cheng

, Interpretable emotion recognition using EEG signals, IEEE Access 7 (2019), 94160–94170. doi: 10.1109/ACCESS.2019.2928691.

Nakisa

Rastgoo

M.N.

Tjondronegoro

and Chandran

, Evolutionary computation algorithms for feature selection of EEG-based emotion recognition using mobile sensors, Expert Systems with Applications 93 (2018), 143–155. doi: 10.1016/j.eswa.2017.09.062.

Gupta

Chopda

M.D.

and Pachori

R.B.

, Cross-subject emotion recognition using flexible analytic wavelet transform from EEG signals, IEEE Sensors Journal 19(6) (2019), 2266–2274. doi: 10.1109/JSEN.2018.2883497.

Alazrai

Homoud

Alwanni

and Daoud

M.I.

, EEG-based emotion recognition using quadratic time-frequency distribution, Sensors 18(8) (2018), 2739. doi: 10.3390/s18082739.

Wang

Z.M.

S.Y.

and Song

, Channel selection method for EEG emotion recognition using normalized mutual information, IEEE Access 7 (2019), 143303–143311. doi: 10.1109/ACCESS.2019.2944273.

Qiu

Shen

Y.Y.

Liu

C.L.

and He

, Multisource transfer learning for cross-subject EEG emotion recognition, IEEE Transactions on Cybernetics 50(7) (2020), 3281–3293. doi: 10.1109/TCYB.2019.2904052.

Cimtay

and Ekmekcioglu

, Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition, Sensors 20(7) (2020), 2034. doi: 10.3390/s20072034.

Liu

Zhu

Huang

Zeng

Yao

Zhang

and Xu

, EEG based emotion recognition by combining functional connectivity network and local activations, IEEE Transactions on Biomedical Engineering 66(10) (2019), 2869–2881. doi: 10.1109/TBME.2019.2897651.

10.

Zhong

Wang

and Miao

, EEG-based emotion recognition using regularized graph neural networks, IEEE Transactions on Affective Computing, 2020. doi: 10.1109/TAFFC.2020.2994159.

11.

Wang

Qiu

and He

, A prototype-based SPD matrix network for domain adaptation EEG emotion recognition, Pattern Recognition 110 (2021), 107626. doi: 10.1016/j.patcog.2020.107626.

12.

Islam

M.R.

Moni

M.A.

Islam

M.M.

Rashed-Al-Mahfuz

Islam

M.S.

Hasan

M.K.

Hossain

M.S.

Ahmad

Uddin

Azad

Alyami

S.A.

Ahad

M.A.R.

and Lió

, Emotion recognition from EEG signal focusing on deep learning and shallow learning techniques, IEEE Access 9 (2021), 94601–94624. doi: 10.1109/ACCESS.2021.3091487.

13.

Wang

Jia

Sutcliffe

and Feng

, ScalingNet: extracting features from raw EEG data for emotion recognition, Neurocomputing 463 (2021), 177–184. doi: 10.1016/j.neucom.2021.08.018.

14.

and Qu

, Leveraging spatial-temporal convolutional features for EEG-based emotion recognition, Biomedical Signal Processing and Control 69 (2021), 102743. doi: 10.1016/j.bspc.2021.102743.

15.

Yin

Zheng

Zhang

and Cui

, EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM, Applied Soft Computing 100 (2021), 106954. doi: 10.1016/j.asoc.2020.106954.

16.

Liu

and Fu

, Emotion recognition by deeply learned multi-channel textual and EEG features, Future Generation Computer Systems 119 (2021), 1–6. doi: 10.1016/j.future.2021.01.010.

17.

Subasi

Tuncer

Dogan

Tanko

and Sakoglu

, EEG-based emotion recognition using tunable Q wavelet transform and rotation forest ensemble classifier, Biomedical Signal Processing and Control 68 (2021), 102648. doi: 10.1016/j.bspc.2021.102648.

18.

Salankar

Mishra

and Garg

, Emotion recognition from EEG signals using empirical mode decomposition and second-order difference plot, Biomedical Signal Processing and Control 65 (2021), 102389. doi: 10.1016/j.bspc.2020.102389.

19.

Song

Zheng

Liu

Zong

Cui

and Li

, Graph-Embedded Convolutional Neural Network for Image-based EEG Emotion Recognition, IEEE Transactions on Emerging Topics in Computing, 2021. doi: 10.1109/TETC.2021.3087174.

20.

Gao

Yang

Kang

Tian

and Song

, EEG-based emotion recognition with feature fusion networks, International Journal of Machine Learning and Cybernetics 13(2) (2022), 421–429. doi: 10.1007/s13042-021-01414-5.

21.

Shen

Peng

Kong

and Dai

, Multi-scale frequency bands ensemble learning for EEG-based emotion recognition, Sensors 21(4) (2021), 1262. doi: 10.3390/s21041262.

22.

Sharma

Pachori

R.B.

and Sircar

, Automated emotion recognition based on higher order statistics and deep learning algorithm, Biomedical Signal Processing and Control 58 (2020), 101867. doi: 10.1016/j.bspc.2020.101867.

23.

Yang

Han

and Min

, A multi-column CNN model for emotion recognition from EEG signals, Sensors 19(21) (2019), 4736. doi: 10.3390/s19214736.

24.

Mert

and Akan

, Emotion recognition from EEG signals by using multivariate empirical mode decomposition, Pattern Analysis and Applications 21(1) (2018), 81–89. doi: 10.1007/s10044-016-0567-6.

25.

Song

Zheng

Song

and Cui

, EEG emotion recognition using dynamical graph convolutional neural networks, IEEE Transactions on Affective Computing 11(3) (2020), 532–541. doi: 10.1109/TAFFC.2018.2817622.

26.

Chen

Yuan

Elhoseny

Ren

Fan

and Chen

, Emotion recognition using empirical mode decomposition and approximation entropy, Computers & Electrical Engineering 72 (2018), 383–392. doi: 10.1016/j.compeleceng.2018.09.022.

27.

Tan

Ying

and Zhang

, Strengthen EEG-based emotion recognition using firefly integrated optimization algorithm, Applied Soft Computing 94 (2020), 106426. doi: 10.1016/j.asoc.2020.106426.

28.

Chao

Dong

Liu

and Lu

, Emotion recognition from multiband EEG signals using CapsNet, Sensors 19(9) (2019), 2212. doi: 10.3390/s19092212.

29.

Ullah

Uzair

Mahmood

Ullah

Khan

S.D.

and Cheikh

F.A.

, Internal emotion classification using EEG signal with sparse discriminative ensemble, IEEE Access 7 (2019), 40144–40153. doi: 10.1109/ACCESS.2019.2904400.

30.

Zhang

and He

, Hierarchical convolutional neural networks for EEG-based emotion recognition, Cognitive Computation 10(2) (2018), 368–380. doi: 10.1007/s12559-017-9533-x.

31.

Pandey

and Seeja

K.R.

, Subject independent emotion recognition from EEG using VMD and deep learning, Journal of King Saud University-Computer and Information Sciences 34(5) (2022), 1730–1738. doi: 10.1016/j.jksuci.2019.11.003.

32.

Liu

Ding

Cheng

Song

Wan

and Chen

, Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network, Computers in Biology and Medicine 123 (2020), 103927. doi: 10.1016/j.compbiomed.2020.103927.

33.

Zhang

Chen

Zhan

Yang

and Xia

, Respiration-based emotion recognition with deep learning, Computers in Industry 92–93 (2017), 84–90. doi: 10.1016/j.compind.2017.04.005.

34.

Koelstra

Muhl

Soleymani

Lee

J.S.

Yazdani

Ebrahimi

Pun

Nijholt

and Patras

, Deap: A database for emotion analysis; using physiological signals, IEEE Transactions on Affective Computing 3(1) (2011), 18–31. doi: 10.1109/T-AFFC.2011.15.

35.

Placidi

Giamberardino

P.D.

Petracca

Spezialetti

and Iacoviello

, Classification of Emotional Signals from the DEAP dataset, in: Proceedings of the 4th International Congress on Neurotechnology, Electronics and Informatics (NEUROTECHNIX), Vol. 2, SCITEPRESS, 2016, pp. 15–21. doi: 10.5220/0006043400150021.

36.

Gupta

and Srivastava

, Parametric Bayesian estimation of differential entropy and relative entropy, Entropy 12(4) (2010), 818–843. doi: 10.3390/e12040818.

37.

and Wong

K.M.

, Riemannian distances for signal classification by power spectral density, IEEE Journal of Selected Topics in Signal Processing 7(4) (2013), 655–669. doi: 10.1109/JSTSP.2013.2260320.

38.

Siuly

and Li

, Designing a robust feature extraction method based on optimum allocation and principal component analysis for epileptic EEG signal classification, Computer Methods and Programs in Biomedicine 119(1) (2015), 29–42. doi: 10.1016/j.cmpb.2015.01.002.

39.

Rai

H.M.

Trivedi

and Shukla

, ECG signal processing for abnormalities detection using multi-resolution wavelet transform and Artificial Neural Network classifier, Measurement 46(9) (2013), 3238–3246. doi: 10.1016/j.measurement.2013.05.021.

40.

Satyanarayana

K.N.V.

, An Approach for finding emotions using Seed Dataset with KNN Classifier, Turkish Journal of Computer and Mathematics Education (TURCOMAT) 12(10) (2021), 2838–2846. doi: 10.17762/turcomat.v12i10.4930.

41.

Zhou

Z.H.

, Ensemble methods: foundations and algorithms, CRC press, 2012.

42.

Zhang

and Ma

, eds., Ensemble machine learning: methods and applications, Springer Science & Business Media, 2012.

43.

Aboneh

Rorissa

and Srinivasagan

, Stacking-based ensemble learning method for multi-spectral image classification, Technologies 10(1) (2022), 17.

44.

Alexandropoulos

S.A.N.

Aridas

C.K.

Kotsiantis

S.B.

and Vrahatis

M.N.

, Stacking strong ensembles of classifiers, in: IFIP International Conference on Artificial Intelligence Applications and Innovations, Springer, Cham, 2019, May, pp. 545–556.

45.

Kwon

Park

and Lee

, Stacking ensemble technique for classifying breast cancer, Healthcare Informatics Research 25(4) (2019), 283–288.

Electroencephalography based human emotion state classification using principal component analysis and artificial neural network

Abstract

Keywords

1. Introduction

3. Principal Component Analysis (PCA) based Artificial Neural Network (ANN) method

Table 1 Experimental results of overall channels in valence on the real time database

Table 4 Experimental results of overall channels in valence on the DEAP database

Funding

Data availability statement

Footnotes

Conflict of interest

Author’s Bios

References

Table 1
Experimental results of overall channels in valence on the real time database

Table 4
Experimental results of overall channels in valence on the DEAP database