Multimodal Emotion Recognition Using Taylor Cooperative Share Hunt Optimized Deep Quantum Neural Network

Abstract

Multimodal emotion recognition comprises speech utterances with corresponding images in a discrete form that strengthens the combined knowledge, captures the importance of image and speech features, and recognizes particular emotions accurately. Several studies have been undertaken in audio–visual model emotion recognition and faced specific challenges such as missing features, misalignment of modalities, inefficient generalization, and inaccurate detection results. To solve these problems, a Taylor Cooperative Share Hunt Optimizer with Deep Quantum neural network (TCoSH–DQNN) is proposed, in which both speech utterance and facial expressions are effectively combined. Furthermore, the proposed model achieves effective multimodal emotion recognition with the combined representation of audio and visual features extracted from videos. The proposed TCoSH–DQNN classifier extracts the human expression in an informative aspect with enhanced performance and improved recognition accuracy. Additionally, the TCOSH–DQNN model generates better communication by resolving conflicts and effectively recognizing feelings. In this perspective, the model reduces the computational complexity and achieves effective recognition with a better convergence rate. The effectiveness of the model is improved by validating the dataset and achieving significant evaluation metric outcomes with an accuracy of 99.97%, a Cohen kappa score of 98.61%, a recall of 99.70%, an F1-score of 99.81%, a f-beta score of 98.67%, and a precision of 99.96%, with the evaluation of the Enterface’05 dataset.

Keywords

multimodal emotion recognition audio–visual input source quantum classifier Taylor Cooperative Share Hunt Optimizer audio–visual feature extraction

1. Introduction

Emotions are multifaceted psychological phenomena for humans to express their feelings in multiple cues (Almulla, 2024). Emotion-related information is usually complementary through multimodal ways, such as visual data that contains the facial expression along with audio characteristics, which contain the voice-frequential components. These components are used to recognize the emotions of humans effectively (Baltrusaitis et al., 2019; Devi & Preetha, 2022). This multimodal detection improves the robustness and performance of the system and recognizes human emotion accurately (Ekman, 1992; Praveen et al., 2023). By identifying emotions, the key components are audio and visual expressions, in which the audio expression comprises the information about the substance of communication, tone, and prosody. On the other hand, the visual expression contains head movement, lighting, visual indication of a person, and so on (Mamieva et al., 2023). By analyzing unimodal behavior with visual expression alone, they faced some obstacles intentionally or unintentionally; some humans may try to differentiate their emotions under the visible phase. Because of these constraints, the visual expression outcome is disregarded and does not achieve recognition. For the analysis of an audio-based recognition system that produced some impact, such as dialects, pronunciation, speech speeds, and accents, adds complexity to observing the important features of audio emotion recognition. To obtain a complete and accurate recognition of people's emotional state, the model is essential to attain the important elements of reliable emotion detection (Mamieva et al., 2023).

Several researchers developed multimodal emotion recognition techniques with various conventional methods such as the Hidden Markov model (Abdusalomov et al., 2022), kernel model (Hsu et al., 2021), artificial neural network (ANN), Gaussian mixture model, support vector machine (SVM), K-nearest neighbor (KNN; Wang et al., 2020), and neural networks (NNs; Makhmudov et al., 2020; Vijayvergia & Kumar, 2021). Together with traditional fusion strategies, some classifiers may be integrated to recognize the emotional dynamic recurrence structure. The long short-term memory (LSTM) network is a transformer model for attaining long-term crucial temporal characters with the more relevant emotional temporal process (Farkhod et al., 2021) that evaluates human multimodal language and generates better performance. An additional attention mechanism was also developed to evaluate the audio–visual stream analysis. For effective multimodal emotion recognition, deep NNs (DNNs; Liu et al., 2023; Mamieva et al., 2023; Pan et al., 2023) were employed to extract relevant features in emotion classification (Mamieva et al., 2023). Convolutional NN (CNN) enables hierarchical function for nonlinear data processing, which analyzes intricate tasks, such as facial recognition and audio identification. Hybrid fusion strategies predict emotions in unimodal expressions, for which the data-level fusion, feature-level fusion, and decision-level fusion approaches are used. The relation among features and decisions of diverse models ensures the system's performance and recognizes human emotional behavior (Almulla, 2024).

The existing methods of multimodal information suffer from interclass variation, missing modality data, intraclass similarities, and so on. In evaluating DNNs, they suffer from system complexity and are computationally challenging during the training process (Mamieva et al., 2023). Additionally, the DNN suffered from an overfitting problem, which resulted in poor data performance and difficulty in recognizing the complex characters of emotions under real-time occurrence (Mamieva et al., 2023). Thus, the exploration of the CNN required pretrained data for identifying corresponding behavior that suffered from overfitting problems to ensure the dynamic characteristics of humans (Kansizoglou et al., 2019). To observe human emotions, SVM and KNN-based network models that possessed high uncertainty and strong computational complexity problems resulted in minimum recognition performance (Wang et al., 2020). In addition, the speech and visual type emotion recognition in LSTM and recurrent neural network (RNN)-based methods notice the gradient disappearance problem and explosion. In the self-supervised learning technique, the features of human emotions are extracted and affected by certain limitations such as high dimensionality, longer sequence length, and mismatches in the model (Siriwardhana et al., 2020). To avoid these problems in the traditional methods, a new model will be designed to recognize human emotions effectively.

To address the drawbacks in the conventional methods, a model is designed with high computational speed and automatically achieves better recognition of high-level emotional features by the Taylor Cooperative Share Hunt Optimizer with Deep Quantum NN (TCoSH–DQNN). The audio–visual data stream input source quality is further enhanced and extracted from the informative feature, and then allowed as the input for the model, which achieves accurate emotional recognition with high-performance accuracy. The major contribution of the research is expressed as follows:

TCoSH: The TCoSH optimization accurately recognizes human emotions effectively by garnering previous experience memory under the validation of the Taylor function, which makes the further recognition process simpler.

TCoSH–DQNN: In a multimodal feature, evaluation contains complex pattern frames, by implementing a quantum classifier that reduced the complexity issues and achieved effective recognition accuracy with improved performance efficacy outcome.

The remaining part of the research is arranged in the following sections. Section 2 provides a detailed explanation of prior methods that recognize human emotion with multimodal expressions. Section 3 describes the proposed TCoSH–DQNN methodology, and the achievements of the proposed model are explained in Section 4. At last, Section 5 serves as the conclusion of the research along with the future scope.

2. Literature Review

The section provides a detailed explanation of various existing methods developed for human emotion recognition.

(i)
Joint Cross-Attention Network

Praveen et al. (2023) developed a joint cross-attentional model for exploring the individual emotional states with the extraction of facial and vocal modalities. The model extracts the salient features to perform effective leverage of inter- and intramodel correlation. The computed cross-attention-based relationships were implemented among the joint feature representation and individual modalities. Furthermore, the joint cross-attentional model was validated by the RECOLA and AffWild2 datasets, and the model attained major benefits such as cost-effectiveness, obtained high-level performance, and encoded intermodal relationships. Because of memory constraints, the size of the network was very complex to perform the training process, which led to computational complexity.
(ii) Multihead Attention Network

Huang et al. (2020) introduced a combined model of transformer and LSTM classifier for identifying the multimodal emotional representation. The proposed model utilized the AVEC 2017 database that observed the fusion strategy of model superiority. The transformer model, along with the self-attention mechanism, observed long-term temporal dependencies and provided better performance for continuous emotion recognition, whereas the LSTM classifier provides better capability for high-level representation. Based on these abilities, the developed combined model attained better performance. The method did not provide better performance for textual modality representation.
(iii) Multihead Self-Attention Network

Ghaleb et al. (2023) established a transformer attention mechanism to evaluate emotion recognition from visual and audio signals. The method explored the temporal information of the signal, and the modality time was explored effectively by the attention mechanism, which reduced the unimodal entropy gap and increased the bimodal predictions. Additionally, the model increased the robustness with higher computational complexity.
(iv) Hybrid Fusion Network

Kumar et al. (2024) employed an interpretability technique to establish the important features of speech and image emotion class, in which the method was validated under the IIT-R SIER dataset, where the duplicate and corrupted samples were removed, and the ground-truth label context was provided for better emotional recognition. The method predicted that one modality emotion was stronger and another modality emotion was weaker; because of this, the model reduced the performance slightly.
(v) Self-Attention Network

Goncalves and Busso (2022) developed an auxiliary network to explore human emotion recognition under audio and visual modalities. An auxiliary network that combined an optimized training mechanism and transformer architecture, to attain high robustness. The major limitation of the auxiliary networks was the high complexity that affected the model's performance.
(vi) Modality Fusion Network

Kansizoglou et al. (2019) established a deep CNN (DCNN) for extracting the features of emotions under speech and visual modality; the DNN model was used to fuse the modality features. In addition, the fused model was trained by the LSTM layer that obtains the temporal quality of the emotions in distinct scenarios. The model exploits large-scale data with misclassified emotions and generates more noticeable features to increase emotion recognition capabilities.
(vii) Attention-Based Fusion Network

Mamieva et al. (2023) introduced an attention-based fusion mechanism to identify human emotional behavior in which the imaging modality was recognized by the ResNet model, whereas the speech modality was recognized by the CNN method. The method was validated under both the IEMOCAP and CMU-MOSEL datasets, which processed the facial and speech features effectively and produced superior system accuracy, but the model possessed data imbalance during the distribution of samples in emotion class recognition.
(viii) Bimodal Fusion Algorithm

Wang et al. (2020) presented a bimodal fusion algorithm to recognize the speech information and facial expressions of humans, in which the speech emotions were recognized by the LSTM and CNN model together with the model of RNN and CNN to obtain the facial expressions information, based on a weighted decision-based bimodal emotion recognition, capturing better performance. The fusion algorithm model could not learn the long-time video frames, which caused the gradient disappearance issue and an explosion of random-length features.
(ix) Hierarchical Fusion Network

Liu et al. (2023) established the cascaded multichannel and hierarchical fusion (CMC–HF) for multimodal emotion recognition, in which the speech, visual, and text signals are concurrently applied as multimodal inputs. Furthermore, the improved hierarchical fusion module was employed for enhancing the intermodality interactions of three modalities. Despite the promising solution, the model requires such models to extract features and strategies to develop fusion approaches for achieving the joint representation between more modalities.
(x) Multimodal Dynamical Fusion

Chen et al. (2023) introduced the multistage multimodal dynamical fusion network (MSMDFN) that applied the cross-modal correlation for achieving the joint representation. In the MSMDFN method, the latent interactions between different features are extracted, and the multistage fusion network splits the fusion procedure into multiple stages. Furthermore, the method improved the generalizability and offered exceptional emotion recognition for different subjects. Nevertheless, the MSMDFN model employs the element-wise product to fuse bimodal signals, which fails to support the adaptable fusion method.
(xi) Feature fusion

Chandraumakantham et al. (2024) presented the large language model (LLM)-based multimodal emotion recognition utilizing feature fusion. Furthermore, the LLM approach utilized the rule-based systems for transforming the nonverbal cues to text without prior knowledge from existing methods. Besides, the method highlights the potential for incorporating additional modalities and converting them into text with the application of rule-based systems to further refine pretrained LLMs. However, the model necessitates the validation of generalizability, exploring different fusion strategies, and potentially integrating larger LLMs for improving the feature extraction.
(xii) Optimization-Based Models

Jain et al. (2025) presented the activational attention layer coupled deep learning model (ALNN-EmR model) using the bald hawks-DCNN for emotion recognition. Using the bald hawk optimization, the hyperparameters, including the weights and biases of the ALNN-EmR model, were effectively tuned and contributed to improved performance. Besides, the ALNN-EmR model effectively addressed the overfitting issue encountered in the conventional methods. Furthermore, the ALNN-EmR model can be extended for analyzing multimodal input in the future.

Devi and Preetha (2022) implemented the self-adaptive model using elephant herding optimization (SA-EHO) for emotion recognition. In the SA-EHO method, the core contribution relies on feature extraction with local vector pattern features and obtains the dense spiral form features. Furthermore, the EHO optimized NN was employed in the method for final classification. Moreover, the optimal weights were selected using the EHO algorithm, which enhanced the effectiveness, and experimental results explicated that the SA-EHO model outperformed the other conventional methods.
(xiii) Multitask Graph NN

Meng et al. (2024) established the class boundary enhanced representation learning (CBERL) model for multimodal emotion recognition. Furthermore, the CBERL method considered the data distribution imbalance problem and resolved the issue by utilizing the three levels of data augmentation. In the CBERL method, a multitask graph NN in conjunction with mask reconstruction and optimization was applied to address both the overfitting and underfitting issues. However, the CBERL required the modal feature fusion for enhancing the overall performance.

2.1 Challenges

In the joint cross-attention fusion, sometimes the possibility of higher attention scores for neutral frames and lower attention scores for more relevant frames occurs, with significant occlusion limiting the overall performance (Praveen et al., 2023).

Attention-based fusion network depends on the attention mechanism for the selection of the most significant features that assist in focusing on informative parts; however, the model introduces a potential vulnerability (Mamieva et al., 2023).

In the audio–visual emotion recognition method using auxiliary networks, a major drawback lies in the high complexity of the network, necessitating the exploration of strategies for reducing the overall complexity without deteriorating the performance (Goncalves & Busso, 2022).

Furthermore, the CMC–HF method necessitates the exploration of such models that extract features and strategies to design fusion methods that effectively learn the joint representation between more modalities (Liu et al., 2023).

The major drawback lies in considering separate descriptors for extracting the features from speech and video input for emotion recognition, which was expected to further enhance the recognition using the ALNN-EmR model (Jain et al., 2025).

For improving the performance, the CBERL method required the modal feature fusion, contributing to providing valuable information for emotion recognition (Meng et al., 2024).

Despite the efforts of the existing techniques to capture the cross-modal relationships using cross-attention-based transformers, the models face difficulty in effectively exploiting the complementary relationship of audio–visual modalities. Besides, the computation of attention weights fails to consider the correlation across the audio and visual features, and hence the models fail to fully capture the complementary relationships between the features. However, the proposed TCoSH–DQNN method addresses the challenges in the existing methods with a combined feature representation integrating the important features of audio extracted using Mel-frequency cepstral coefficient (MFCC), audio-based textual features, and Deep pretrained VGGish features, and video features, including the deep pixel flow-based hybrid textual pattern (HTP) features, and grey level co-occurrence matrix (GLCM) features to improve the multimodal emotion recognition. By applying the combined feature representation, the proposed method eliminates the heterogeneity between audio and video features, thus contributing to robust feature representations. Furthermore, the TCoSH–DQNN model utilizes the quantum convolutional layers to extract features and encodes the data from different modalities into quantum states to improve multimodal emotion recognition. Furthermore, the application of the TCoSH algorithm for optimal tuning contributes to high recognition accuracy with enhanced convergence rate and reduces the computational complexity.
2.2 Problem Statement

Several researchers have gone through the multimodal emotion recognition methods, which perform far better than the unimodal method, but also suffer from certain challenges, such as computational complexity and insufficient performance. The complexity problem is obtained by interpreting a large amount of data for evaluation (Mamieva et al., 2023). Multimodal recognition that is affected by generalizability challenges obtained by diverse contexts and scenarios that are fused in the method for exploration (Greco et al., 2024). While evaluating large video frames that cause an impact on scalability and cause gradient vanishing problems (Wang et al., 2020).

The above-mentioned challenges are overcome by the proposed TCoSH–DQNN model, which recognizes the multimodal feature emotional expressions by a quantum classifier that contains a multidimensional qubit algorithm for evaluating the complex pattern. The input multimodal feature sources are categorized as video and audio, the computational complexity is reduced by a quantum classifier (Enterface’05 Dataset, 2024), and the scalability factor is achieved by tuning the model with optimization techniques. Based on these techniques, the TCoSH–DQNN model attains accurate emotion recognition.

3. Multimodal Emotion Recognition Using TCoSH–DQNN

The research aims to develop the TCoSH–DQNN model to recognize human emotions that are explored under various audio–visual data, which extends numerous applications and advances. The input audio–visual data are obtained from the Enterface’05 Dataset (2024), and the Ryerson Audio–Visual Database of Emotional Speech and Song (RAVDESS, 2024), which are fed into the preprocessing technique in which the visual image is separated by a Gaussian filter. In the audio–visual input source, the Gaussian filter performs well in the frequency domain, which extracts a certain region of the image using the Haar cascade filter. These separated audio and video features are then allowed into the feature extraction phase, where the features of the visual image are extracted by deep flow-based HTP features and GLCM features. The informative features of the audio source are extracted by MFCCs, audio-based textual features, and a deep pretrained VGGish feature. The outcome of the separated features of audio and video is concatenated and allowed as the input to the TCoSH–DQNN model to recognize the desired results, which increases the computational speed with better performance and automatically detects the emotions without any supervision. The potential challenges are exponentially reduced by the integration of the cooperative hunting characteristics of Harris Hawks (Houssein et al., 2021) and the hierarchical sharing characteristics of grey wolves (Zhao et al., 2019). Thus, the proposed TCoSH–DQNN model recognizes human emotions by considering the audio–visual features, and the basic block diagram of TCoSH–DQNN is presented in Figure 1.

Figure 1.

Schematic representation of the TCoSH–DQNN model for multimodal emotion recognition.

3.1 Input

The input audio/visual data source is obtained from Enterface’05 Dataset (2024) and RAVDESS (2024). These datasets contain various expressions of human emotion in the form of audio and visual, in which every expression contains emotional intensity with a neutral category. These garnered audio–visual data input sources are expressed mathematically as,

\begin{aligned} B = {d_{1}, d_{2}, d_{3}, \dots, d_{a}, \dots, d_{t}}, \end{aligned}

(1)

where B represents the input database and

{d_{1}, d_{2}, d_{3}, \dots, d_{a}, \dots, d_{t}}

denotes the audio–visual data characteristics.

3.2 Preprocessing

A preprocessing method is used to improve the quality and extract the important features from the input source, which contains both audio and visual characters of emotional expressions. From these data sources, the images are separated by the Gaussian filter, which performs best in the frequency domain that converts the image into a grey scale and extracts a certain region of images using the Haar cascade filter. The separated audio signal E and video image I are then allowed into the feature extraction phase for obtaining the informative feature.

\begin{aligned} T = \frac{1}{\sqrt{2 π \partial}} \exp (- \frac{{(B - ξ)}^{2}}{2 \partial^{2}}), \end{aligned}

(2)

where

\partial

denotes the standard deviation and

ξ

indicates the mean function.

3.3 Feature Extraction

The separated audio and video stream data are then subjected to the feature extraction phase, in which the important features of audio are extracted by MFCC, audio-based textual features, and deep pretrained VGGish features, whereas the features of the video stream are extracted by deep pixel flow-based HTP features and GLCM features.

3.3.1 Audio feature extraction phase

(a)

MFCC

The separated audio streams are then allowed into the MFCC feature phase, which extracts diverse perception capabilities at distinct frequency ranges. The Mel frequency is more sensitive for lower frequency sound signals and insensitive for high signal frequencies, which extracts the audio stream into discrete overlap frames based on the bandwidth of human ear sound perception, and the derived cepstral type audio depiction in a nonlinear characteristic form. The difference between the normal cepstrum and mel-frequency cepstrum in MFCC is approximately equal to the human hearing system, with a greater linearly spaced occurrence band in the normal cepstrum that extracts the temporal and frequency domain sound information (Prabakaran & Sriuppili, 2021). Thus, the MFCC audio stream extraction produced better sound depiction with improved frequency resolution at low-frequency regions. The outcome of MFCC is mentioned as MFCC(B):

\begin{aligned} MFCC (B) = \sum_{d = 1}^{t} \log (T) \cos [d (t - \frac{1}{2}) \frac{π}{t}] . \end{aligned}

(3)

(b) Audio-Based Textual Features

The variation of sound perception garnered from the MFCC method extracts the substantial texture features based on statistical feature extraction, which provides the statistical information about the audio frames that are then subjected to the deep pretrained features to attain multiple significant enhancements in the performance of extracting the audio signals. The textual audio features are represented by Tex(B).

The VGGish feature extraction model is highly effective for performing audio signal tasks, which provides numerous significant improvements with minimum training time that extract the audio signal by performing tasks including audio classification, audio analysis, and content-based retrieval. The temporal information of the audio signal is obtained by categorizing the signal into segments, which are allowed into the short-time Fourier transform, while the signal is converted into the frequency domain, and the magnitude value for every audio segment is computed, resulting in a high-level representation of audio content. The VGGish method enables better generalization and reduces the overfitting problem in the training process, achieving highly robust features (El-Latif et al., 2024). The attained feature vector dimension for the audio signal is $(1 \times 807)$ . The outcome of the VGGish feature is VGG(B).

The concatenated audio feature extraction is represented as

\begin{aligned} Audio Feature = [MFCC (B) | | Tex (B) | | VGG (B)] . \end{aligned}

(4)

The achieved feature dimension for audio signal extraction is $(1 \times 807)$ , which is not equal to the video extraction feature dimensions. To make the feature dimensions the same size, perform a padding process in which the spatial size of the audio signal is preserved by adding additional filters to adjust and obtain the desired output dimensions. By performing the padding process, the audio signal feature dimensions are obtained $(1 \times 90, 550)$ .

3.3.2 Video Feature Extraction Phase

(a)

Deep Flow-Based HTP Features

The separated video frames are allowed into the video extraction phase that obtains the intricate visual patterns along with the neighboring pixel relation, which integrates the global and local textual information. The pixel labeling and edge response of every image in the neighboring region are achieved in the form of binary integers. To extract the substantial texture features, the image pixels are associated with the binary values 0 and 1 with the threshold constants 1, 0, and −1. Here, the center image sample is considered as the threshold constant, while evaluating the neighboring sample with a greater value is assigned as +1, and with a lower value is denoted as −1. Based on this evaluation, the textual pattern of the image is extracted, and the outcome is represented as

\begin{aligned} HT P_{I} = 0.7 (\sum_{r = 0}^{7} f (p_{r} - p_{ce}) 2^{r}) + 0.1 (\sum_{r = 0}^{7} f (d_{r} - d_{e}) 2^{r}) + 0.2 ({\begin{cases} 1, & ne \geq ce + F \\ 0 & | ne - ce | < F \\ - 1 & ne \leq ce - F \end{cases}}) \end{aligned}

(5)

where f denotes the signal frequency (Hz), p indicates the pixel labeling of the image, d represents the edge response, e defines the directional response, ne indicates the neighboring samples, ce indicates the center constant sample, and F denotes the user-defined threshold.

(b) GLCM Features

The GLCM matrix is widely used to extract the spatial variation and texture features obtained from the probability matrix, along with texture statistical characteristics that include correlation, energy, contrast, and homogeneity (Lohithashva et al., 2020). The GLCM is organized by two structures, such as a histogram with a greyscale pixel pair and a spatial relationship pair, in which the texture discrimination problem was evaluated by co-occurrence matrices that contain rows and columns, which provides high classification accuracy with low time consumption (Dwaich & Abdulbaqi, 2021). The features involved in GLCM are elaborated briefly in the sections below. (i)

Correlation

Correlation measures the occurrence relation of certain pixel pairs to the probability of neighboring pixels over the entire frames. The high correlation dependency is computed between two pixels for positive and negative correlated frames (Kurniati et al., 2024), which is mathematically expressed by

\begin{aligned} Correla = \frac{\sum_{l} \sum_{m} [l m * G (l, m)] - ω_{l} * ω_{m}}{σ_{l} * σ_{m}}, \end{aligned}

(6)

where G(l, m) indicates the element of the co-occurrence matrix of T , l, m denotes the co-occurrence matrix indices,

ω

defines the mean, and

σ

refers to the standard deviation.

(ii) Energy

Energy is calculated by the sum of squared elements to the uniform value elements in the matrix, which is computed as

\begin{aligned} Ener = \sum_{l} \sum_{m} G {(l, m)}^{2} . \end{aligned}

(7)

(iii) Contrast

Contrast is measured by the intensity degree of diverse image pixels along with neighboring pixels. For high contrast, the variation of pixel intensity is increased, whereas for low contrast, the variation of pixel intensity is less.

\begin{aligned} Cont = \sum_{l} {\sum_{m} (l - m)}^{2} * G (l, m) . \end{aligned}

(8)

(iv) Homogeneity

Homogeneity is evaluated by the degrees of matrix elements that are placed near the probability matrix diagonal. If the matrix elements have high values, which are placed near the diagonal, the obtained homogeneity value is also high. The mathematical expression for homogeneity is represented as

\begin{aligned} Homo = \sum_{l} \sum_{m} \frac{G (l, m)}{1 + {(l - m)}^{2}} . \end{aligned}

(9)

The extracted textual features of video frames by the GLCM matrix are represented by

\begin{aligned} GLCM = [Correla | | Ener | | Cont | | Homo] . \end{aligned}

(10)

The feature extraction of video frames obtained from deep pixel flow-based HTP features and GLCM is concatenated, and the attained outcome feature dimension is $(28 \times 29)$ . The concatenated video feature is mentioned as

\begin{aligned} Video Feature = {HT P_{I} | | GLCM} . \end{aligned}

(11)

3.3.3 Feature Concatenation

The extracted features of audio and video frames from the feature extraction phase are concatenated and obtained as a feature vector, which is the input for the significant model. The obtained dimension of the feature vector is $(1 \times 90, 550)$ . The output of concatenated audio and video features is represented as D and is expressed as

\begin{aligned} D = {Audio feature | | Video feature} . \end{aligned}

(12)

3.4 Emotional Recognition Using DQNN

The concatenated feature vector is considered as the input for the TCoSH–DQNN model, which effectively recognizes human emotion characters. Specifically, the TCoSH–DQNN model encodes the data associated with multiple modalities, and the feature vector is mapped to the amplitudes of a quantum state, providing the unified representation of multimodal data. The major benefits of the TCoSH–DQNN model are enhanced computation speed with better performance and garnered high-level emotional features without any human supervision, which is based on a quantum-parameterized circuit that contains a DCNN followed by a quantum-classified layer (Li et al., 2020). The TCoSH–DQNN model acquires data encoding, variation, and quantum measurement layers. In data encoding, the input feature vector data are encoded to form a quantum state, which is mathematically expressed by

\begin{aligned} \sum_{{q D_{1}, q D_{2}, \dots, q D_{n}}} {‖ μ_{q D_{1}, \dots, q D_{n}} ‖}^{2} = 1, \end{aligned}

(13)

where

q D_{1}, q D_{2}, \dots, q D_{n}

defines the quantum state of the input feature vector and

μ

denotes the amplitude of every quantum state. The total probability of the encoding should be 1.

The encoded data are flattened and converted into rotation angles by the arc tangent function. For every feature vector, two rotation angles were generated with a quantum state, which is then allowed into a unitary transformation series that contains the entanglement part and rotation part. Here, every qubit rotation, which provides the basic unit of information of quantum computing, was parameterized by an optimization procedure. Then the quantum value for every qubit rotation was measured, which reduced the complexity issues based on quantum measurement (Chen et al., 2022). These parameterized qubit values are then extracted into the layer of the TCoSH–DQNN model containing a convolutional layer, pooling layer, and fully connected layer in which the convolutional layer extracts the input feature into the kernel and computes the dot product of the input image pixel to the kernel slide values, the obtained feature maps are subjected into pooling layer, which reduces the size of the feature map for effective computation that acts as a connector for both convolution and fully connected layer. In a fully connected layer, the resultant pooling layer is compared with the input image feature to perform the classification function. Based on these evaluations, the TCoSH–DQNN recognizes human emotions effectively. To avoid overfitting problems, dropout is used, and the activation function decides which informative feature is being received or terminated for attaining the recognition function. By considering these informative features, the weight and bias of the model are effectively tuned by the TCoSH algorithm, which provides high recognition accuracy with enhanced convergence rate and computational complexity. The basic architecture of the model is illustrated in Figure 2.

Figure 2.

Architecture of the proposed model.

3.5 TCoSH Optimization

The TCoSH optimization is obtained by the hybridization of cooperative hunting and sharing characters of Harris hawks (Makhadmeh et al., 2024), and the hunting hierarchy behavior of the grey wolf (Al-Betar et al., 2023). These characters are merged with the inclusion of the Taylor series (Vinitha & Rukmini, 2022) to recognize human emotions from audio/visual streams. The hierarchy hunting and cooperative sharing characters with the integration of the Taylor series may provide a better determination of emotional expressions, along with the previous expressions. This determination process is achieved by the Taylor series, which obtains the previous experience memory of the grey wolf and Harris hawks to achieve effective hunting with an increased convergence rate and reduce the computational complexity. Based on these previous experiences, the collection memory is compared with the current iteration and provides a better outcome. By this memory recollection experience analysis, the entire model accurately recognizes human emotion and provides highly effective outcome results.

Inspiration: Generally, the Canidae family lives as a pack with social hierarchy and hunting behavior. Additionally, contains the decision-making principle for effective hunting, mating, migration, and so on. In nature, Harris Hawks have the character of a dynamic chasing style with cooperative attack and sharing character, which collects information about the prey and makes decisions based on the previous attacking method to achieve a better hunting process. By this evaluation, the method contains collective information that is very complex to solve, and makes the process simple by implementing the Taylor function to evaluate the computational series. Based on this multifactorial consideration, the TCoSH–DQNN model determines the emotional expression effectively and provides effective declaration results.

Solution Representation: Initialize the population, which is mathematically expressed as

\begin{aligned} s = {s_{1}, s_{2}, \dots, s_{i}, \dots, s_{n}} \end{aligned}

(14)

For all “n” solution/parameter sets, the best solutions are updated based on the strength degree score.

Objective Function: The evaluation of the strength degree in the experimental research is represented as H. The entire experiment requires the maximum value to achieve better recognition results

\begin{aligned} H = max (Accuracy) . \end{aligned}

(15)

Solution Update: The solution of TCoSH optimization is based on the strength degree, which obtains exploration and exploitation phases that are briefly expressed in the sections below.

Phase 1: Exploration Phase [ $ρ < ξ$ ]

Evaluate the strength degree for “n” number of solutions, based on this strength factor, sort the solutions with a higher degree, and declare one solution as best agent “ $s_{v}^{b e s t}$ ,” which recollects the preceding memory about the area boundaries and experience. By this recollected memory, select one search boundary based on the strength factor and initialization factor. In this boundary, the location and position of the solution are updated, and the hunting process is performed.

The recollected boundaries are denoted as

\begin{aligned} Q_{u 1}, Q_{u 2}, Q_{u j}, Q_{u w}, \end{aligned}

(16)

where w indicates the total recollected positions.

\begin{aligned} Q_{u j} = s_{v}^{best} + α \cdot u_{1} (s_{avg} - s_{i}), \end{aligned}

(17)

where

α

indicates the control parameter and is expressed as

1.5 \leq α \leq 2

, and

u_{1}

indicates the random number that ranges as [0,1]

s_{avg}

representing the recollected previous experiences.

ρ

defines the random probability and

ξ

denotes the probability threshold, which attains the value 1 [

ξ = 1

Based on the strength factor and initialization factor, the best search boundary is selected in the j^th search space. In this boundary, the location and position of the solution are determined. Locate the target point as $Q_{u j}^{*}$ .

\begin{aligned} δ_{i} = | b_{1} s^{ref} - s_{i} |, \end{aligned}

(18)

where

s^{ref}

means the reference location.

Based on the target set, the position and angle of attack are illustrated as

\begin{aligned} s^{v + 1} & = s^{ref} - b_{2} δ + DI, \end{aligned}

(19)

\begin{aligned} b_{1} & = 2 u_{2}, \end{aligned}

(20)

\begin{aligned} b_{2} & = 2. k u_{3} - k, \end{aligned}

(21)

\begin{aligned} u_{3} & = \frac{δ_{i} - δ_{min}}{δ_{max} - δ_{min}} \in [0, 1] . \end{aligned}

(22)

At the beginning of optimization, the value of $u_{3} = 1$ , and which reduces during iteration.

Assume that $δ_{max} = δ_{min}$ , then $u_{3} = 1$ , $u_{2}$ is a random number that ranges as $[0, 1]$ , where $DI$ indicates the direction indicator that shows the direction of the attack.

Based on these positions and the location of the solution, hunting mechanisms are performed effectively by selecting the best position to achieve the attack. By deciding the best position, the attack is attained and performs the defeating process.

\begin{aligned} s^{v + 1} & = \frac{s_{1}^{v + 1} + s_{2}^{v + 1} + s_{3}^{v + 1} + s_{4}^{v + 1} + s_{5}^{v + 1}}{5}, \end{aligned}

(23)

\begin{aligned} s_{1}^{v + 1} & = s_{v}^{best} - b_{4} \cdot δ_{1, ref}, \end{aligned}

(24)

\begin{aligned} s_{2}^{v + 1} & = s_{2}^{best} - b_{5} \cdot δ_{5, ref}, \end{aligned}

(25)

\begin{aligned} s_{3}^{v + 1} & = s_{3}^{best} - b_{6} δ_{3, ref}, \end{aligned}

(26)

\begin{aligned} s_{4}^{v + 1} & = s_{ran} - u_{4} | s_{ran} - 2 \cdot u_{5} s_{i} |, \end{aligned}

(27)

\begin{aligned} s_{5}^{v + 1} & = s_{ref}^{v} - s_{avg} - V^{v + 1} (L + u_{6} (U - L)), \end{aligned}

(28)

where $s_{2}^{best}, s_{3}^{best}$ represent the second and third best solution, $δ_{1, ref}$ defines the distance between the $s_{v}^{best}$ and $s_{1}^{v}$ , $δ_{2, ref}$ indicates the distance between the second best based on strength degree, and $s_{ref}$ . $δ_{3, ref}$ denotes the distance between the third best solution decided based on strength indication, and $s_{ref}$ . $L, U$ represents the lower and upper bound, $s_{i}$ indicates the i^th solution, and $s_{ref}$ means the reference solution.

\begin{aligned} b_{4} = γ_{1} (s_{max, 1} - s_{min, 1}) . \end{aligned}

(29)

The above equation represents the maximum and minimum positions $s_{1}^{best}$ .

\begin{aligned} b_{5} = γ_{1} (s_{max, 2} - s_{min, 2}), \end{aligned}

(30)

which represent the maximum and minimum positions

s_{2}^{best}

. And, the maximum and minimum positions

s_{3}^{best}

are defined as

\begin{aligned} b_{6} = γ_{1} (s_{max, 3} - s_{min, s}), \end{aligned}

(31)

where

γ_{1} \in (0, 1)

u_{6} \in (0, 1)

V^{v + 1}

denotes the construction parameter, which is mathematically depicted as

\begin{aligned} V^{v + 1} = V^{v} + g_{1} u_{8} (s_{v}^{best} - s^{v}) + g_{2} u_{9} (s_{h} - s^{v}), \end{aligned}

(32)

where

g_{1}, g_{2}, u_{8}, u_{9}

denote the optimal coefficients that range as

[0, 1]

u_{4} = u_{5} = 2

that enables the search to cover the region that is centered at

s_{h}

and

s_{v}^{best}

The direction indicator explores the search for the solution to attain an effective attacking mechanism in a spiral movement, which is expressed as

\begin{aligned} DI & = \frac{C_{1} (i) * \sin θ (i) + C_{2} (i) * \cos θ (i)}{2}, \end{aligned}

(33)

\begin{aligned} C_{1} (i) & = ϕ (i) + Y * ran, \end{aligned}

(34)

where Y is defined as the search cycle counter, which takes the value between 0.5 and 2

\begin{aligned} ϕ (i) = x * π * ran, \end{aligned}

(35)

where x indicates the degree of search that ranges as

(0, 10)

and the random movement,

ran

ranges as

[- 1, 1]

Phase 2: Exploitation phase [ $ρ > ξ$ ]

In this phase, the chaser attacks the solution based on the current position compared with the previous position. The phase has higher strategies of chasing techniques to achieve effective attacking successfully by generating new, updated solution positions. The higher degree of chasing strategies is briefly elaborated in the cases below.

Case (i) $S \geq 0.5$ and $P \geq 0.5$

In this case, the escaping degree is high, which makes the situation convenient for escape, while the chaser aims at successful hunting with the current position information of the runner, the chaser is insufficient to perform the hunting process. To make the hunt successful, update the position with the standard equation, which is derived as follows:

\begin{aligned} s^{v + 1} & = Δ s^{v} - P [s_{c}^{v} - s_{i}^{v}], \end{aligned}

(36)

\begin{aligned} Δ s^{v} & = (s_{c}^{v} - s_{i}^{v}) . \end{aligned}

(37)

Now, the fractional theory is applied to the standard equation to update the position, which is modified as

\begin{aligned} s^{v + 1} = β^{v} s^{v - 1} + \frac{1}{2} β^{v} . s^{v - 2} + \frac{1}{6} (1 - β^{v}) s^{v - 3} + \frac{1}{24} β^{v} (1 - β^{v}) (2 - β^{v}) s^{v - 3} - P (s_{c}^{v} - s_{i}^{v}), \end{aligned}

(38)

where

s^{v + 1}

defines the chaser position,

s_{c}^{v}

denotes the runner position, S means the control factor, P computes the escape parameter, and

β

indicates the fractional degree that is expressed as

\begin{aligned} β = \frac{β_{max}^{v} - β_{min}^{v}}{β_{max}^{v}} . \end{aligned}

(39)

Case (ii) $S \geq 0.5$ and $P < 0.5$

In this case, the runner gets exhausted because of the improved adaptability character of the chaser, which is less due to the updated position of the runner.

\begin{aligned} s^{v + 1} & = s_{c}^{v} - P (s_{c}^{v} - s_{i}^{v}), \end{aligned}

(40)

\begin{aligned} s^{v + 1} & = s_{c}^{v} - P [s_{c}^{t} - \frac{1}{2} (s_{i}^{v} + η_{0} + s_{i}^{v - 1} η_{1} + η_{2} s_{i}^{v - 2} + η_{1} H (s_{i}^{v - 1}) + η_{2} H (s_{i}^{v - 2}))], \end{aligned}

(41)

where H indicates the strength degree.

Case (iii) $S < 0.5$ and $P \geq 0.5$

In this case, the chaser has improved sensitive behavior to detect the runner's movement for every updated iteration to make an effective attack. Additionally, the runner has sufficient energy to escape, but still, the chaser makes a soft besiege to achieve the best possible attack

\begin{aligned} s^{v + 1} & = s_{c}^{v} - P (s_{c}^{v} - s_{i}^{v}) + ran [0, 1] * V^{v + 1}, \end{aligned}

(42)

\begin{aligned} V^{v + 1} & = 0.5. V_{0}^{v} + 1.34 V^{v - 1} - 1.35 V^{v - 2} + 0.69 V^{v - 3} - 0.23 V^{v - 4} + 0.56 V^{v - 5} - 0.01 V^{v - 6} + 1.38 V^{v - 7} - 9.9 V^{v - 8} . \end{aligned}

(43)

Case (iv) $S < 0.5$ and $P < 0.5$

In this case, the chaser attacks the runner with a high degree of strength factor because the energy level of the runner is reduced in the updated movement, which makes the chaser reduce the distance in the circumference, and the runner makes a sudden attack effectively.

\begin{aligned} s^{v + 1} = s_{c}^{v} - P (s_{c}^{v} - s_{avg}^{v}) + ran [0, 1] * [s_{best}^{v} - s_{i}^{v}] . \end{aligned}

(44)

Declaration of objective function: To attain the best solution, which is obtained by analyzing the previous recollected memory experiment.

Termination condition: When $ran > H$ , the best search boundaries are obtained through the position updation stage, which tunes the weights and bias of the TCoSH–DQNN model and enhances the recognition performance. The flowchart for the TCoSH optimization is depicted in Figure 3.

Figure 3.

Flowchart of TCoSH optimization.

4. Results

The section elaborates on the achieved result for emotion recognition by the TCoSH–DQNN model, along with the performance evaluation and comparative evaluation of various conventional methods.

4.1 Experimental Setup

The experiment is implemented in Python running on Windows 11, with available memory of RAM 16 GB and ROM 128 GB.

4.2 Dataset Description

The research recognizes human emotion based on the input dataset, such as Enterface’05 audio–visual dataset (Enterface’05 Dataset, 2024), and RAVDESS (2024), which are defined as follows:

Enterface’05 Audio–Visual Dataset (Enterface’05 Dataset, 2024): The Enterface’05 dataset is an audio–visual emotion database mainly used in multimodal tasks for testing and assessing the video, audio, or joint audio–visual emotion recognition methods. Specifically, the dataset contains 42 subjects from 14 diverse nationalities, of which 81% are men, and the remaining 19% are women. Furthermore, the recordings are collected for two weeks, and all the experiments are conducted in English. Each participant was told to listen to six consecutive short stories, each of which provoked a specific emotion. Hence, each subject unambiguously portrays various expressions; in this case, the informative expression sample is stored in a database, and others will be discarded.

RAVDESS dataset (RAVDESS, 2024): The RAVDESS dataset is an audio–visual database containing 1,440 files obtained from 24 professional actors, each claiming 60 vocalizing trials. Furthermore, the 24 professional actors involved 12 female and 12 male subjects vocalizing two lexically matching statements in the accent of neutral North America. Furthermore, the dataset provides only speech signals with various expressions such as sad, fear, happy, angry, calm, disgust, and surprise. Every expression is computed with emotional intensity as strong, normal, or neutral.

4.3 Experimental Results

The image result obtained from every step of the human emotion recognition is evaluated under the TCoSH–DQNN model, and the achieved results are represented in Figure 4.

Figure 4.

Experimental analysis of the TCoSH–DQNN model.

4.4 Performance Evaluation

The performance estimation of the TCoSH–DQNN model for recognizing the emotional expressions of humans under the utilization of the Enterface’05 dataset achieves certain evaluation metrics such as F1-Score, Precision, Recall, f-beta score, accuracy, and Cohen kappa score by attaining the training percentage (TP) as 90% with maximum epoch as 500, the obtained evaluation value of F1-score is 99.81%, the attained value of precision is 99.95%, the achieved performance evaluation value of Recall is 99.70%, the attained f-beta score is 98.67%, the achieved value for accuracy is 99.96% and the achieved score of Cohen kappa is 98.60%. The overall performance assessments for the model under the Enterface’05 dataset are schematically represented in Table 1.

Table 1.

Performance Estimation of the TCoSH–DQNN.

	Enterface’05 Dataset			RAVDESS Dataset
Metrics	TP = 50%	TP = 70%	TP = 90%	TP = 50%	TP = 70%	TP = 90%
F1-Score	96.09	98.71	99.81	96.55	97.54	98.63
Precision	96.31	99.25	99.96	96.18	97.59	98.63
Accuracy	97.04	98.71	99.97	97.54	98.13	98.36
Cohen's kappa score	97.92	97.70	98.61	97.67	98.01	98.27
F-beta score	97.72	97.71	98.67	96.45	96.93	98.25
Recall	96.84	99.13	99.70	96.57	97.45	98.74

Note. TCoSH–DQNN = Taylor Cooperative Share Hunt Optimizer with Deep Quantum Neural Network; RAVDESS = Ryerson Audio–Visual Database of Emotional Speech and Song; TP = training percentage.

The performance assessment of the TCoSH–DQNN model for recognizing the emotional expressions of humans under the utilization of the RAVDESS dataset, which achieves certain evaluation metrics, by considering the TP value as 90% with maximum epoch as 500, the obtained performance evaluation value of F1-Score metric is 98.62%, the attained value of precision is 98.63%, the achieved performance evaluation value of Recall is 98.73%, the achieved value of accuracy is 98.36%, the obtained value f-beta score is 98.25%, and the attained value of Cohen kappa score is 98.28. The overall performance assessment for the TCoSH–DQNN under the RAVDESS dataset is illustrated in Table 1.

4.5 Comparative Evaluation

The performance of TCoSH–DQNN is compared with other preliminary methods of emotional recognition of humans, such as SVM (Yan et al., 2023), multilayered perceptron (MLP; Joy et al., 2020), ALNN-EmR (Chen et al., 2023), SA-EHO (Chandraumakantham et al., 2024), DNN (Kim & Shin, 2019), DCNN (Haque & Valles, 2018), Harris Hawk optimization algorithm (HHOA)–DCNN (Al-Betar et al., 2023), grey wolf optimization algorithm (GWOA)–DCNN (Makhadmeh et al., 2024), Hybrid DCNN (Verma & Verma, 2020), HHOA–quantum DCNN (QDCNN; Halawani et al., 2023), and GWOA–QDCNN (Nirmala Sreedharan et al., 2018).

4.5.1 Comparative Evaluation for Enterface’05 dataset

The comparative assessment of the TCoSH–DQNN model using the Enterface’05 dataset with maximum TP at 90% is shown in Figure 5. With 90% of training, the TCoSH–DQNN attained an accuracy of 99.97%, indicating the relative improvement of 23.88% over SVM, 18.69% over MLP, 13.23% over DNN, 11.16% over ALNN-EmR, 8.82% over SA-EHO, 7.65% over DCNN, 4.96% over HHOA–DCNN, 2.11% over GWOA–DCNN, 0.10% over Hybrid DCNN, 1.67% over Harris Hawk optimization (HHO)–quantum CNN (QCNN), and 0.29% over grey wolf optimization (GWO)–QCNN. Subsequently, the proposed method attained the F1-score of 99.82%, while compared with the respective methods, which provides the performance improvement as 23.97%, 21.76%, 17.83%, 11.17%, 7.78%, 2.55%, 0.100%, 1.07%, and 2.54% compared to SVM, MLP, DNN, DCNN, HHOA–DCNN, GWOA–DCNN, Hybrid DCNN, HHO–QCNN, and GWO–QCNN, respectively. Similarly, the proposed method attained a high Cohen kappa score, demonstrating the relative improvement of 31.99%, 27.71%, 25.52%, 11.49%, 5.45%, 1.63%, 0.10%, 1.02%, and 1.12% compared to SVM, MLP, DNN, DCNN, HHOA–DCNN, GWOA–DCNN, Hybrid DCNN, HHO–QCNN, and GWO–QCNN, respectively. Furthermore, the TCoSH–DQNN achieved values of the f-beta score are 98.67%, whereas the relative improvements of 19.89%, 19.04%, 17.90%, 8.77%, 4.89%, 1.69%, 0.11%, 1.49%, and 1.61% are attained over SVM, MLP, DNN, DCNN, HHOA–DCNN, GWOA–DCNN, Hybrid DCNN, HHO–QCNN, and GWO–QCNN, respectively. For the recall metric, the proposed method obtained 99.7%, and the improved value of recall when compared with SVM, MLP, DNN, DCNN, HHOA–DCNN, GWOA–DCNN, Hybrid DCNN, HHO–QCNN, and GWO–QCNN are attained as 17.03%, 14.83%, 12.95%, 7.95%, 6.35%, 4.71%, 0.10%, 0.14%, and 4.64%, respectively. Furthermore, the proposed method attained the high precision of 99.95%, indicating the improvements of 22.02%, 19.91%, 9.16%, 4.05%, 1.08%, 0.71%, 0.10%, 013%, and 0.38% against SVM, MLP, DNN, DCNN, HHOA–DCNN, GWOA–DCNN, Hybrid DCNN, HHO–QCNN, and GWO–QCNN, respectively. Moreover, the proposed method outperformed the other baseline methods utilized for the comparison. The comparative estimation of the Enterface’05 dataset is depicted in Figure 5.

Figure 5.

Comparative evaluation for Enterface’05 dataset.

4.5.2 Comparative Assessment of the RAVDESS Dataset

The comparative assessment for the TCoSH–DQNN model under the RAVDESS dataset with maximum TP at 90% is shown in Figure 6. From the evaluation, the proposed model attained a high accuracy of 98.5%, outperforming the existing methods with a substantial improvement of 2.44% over SVM, 2.35% over MLP, 2.11% over DNN, 2.117% over ALNN-EmR, 2.11% over SA-EHO, 2.11% over DCNN, 1.97% over HHOA–DCNN, 1.74% over GWOA–DCNN, 0.69% over Hybrid DCNN, 0.72% over HHO–QCNN, and 0.80% over GWO–QCNN. The achieved value of F1-score is 97.93%, while the improved values are 1.93%, 0.95%, 0.68%, 0.67%, 0.64%, 0.51%, 0.212%, 0.216%, and 0.255% compared to SVM, MLP, DNN, DCNN, HHOA–DCNN, GWOA–DCNN, Hybrid DCNN, HHO–QCNN, and GWO–QCNN, respectively. The attained values of the Cohen kappa score is 97.4%, while compared with the respective traditional methods, the improved performance values are 4.59%, 2.22%, 2.14%, 2.09%, 1.56%, 1.30%, 0.59%, 0.62%, and 0.876% compared to SVM, MLP, DNN, DCNN, HHOA–DCNN, GWOA–DCNN, Hybrid DCNN, HHO–QCNN, and GWO–QCNN, respectively. Subsequently, the proposed method achieved a f-beta score of 98.42%, while the improved values are 4.20%, 3.67%, 3.11%, 2.11%, 1.86%, 1.14%, 0.10%, 0.41%, and 0.70% over SVM, MLP, DNN, DCNN, HHOA–DCNN, GWOA–DCNN, Hybrid DCNN, HHO–QCNN, and GWO–QCNN, respectively. Furthermore, the proposed method achieved a high recall value of 98.20%, attaining the performance improvement of 2.68%, 1.98%, 1.49%, 1.46%, 1.35%, 0.80%, 0.08%, 0.33%, and 0.71% against SVM, MLP, DNN, DCNN, HHOA–DCNN, GWOA–DCNN, Hybrid DCNN, HHO–QCNN, and GWO–QCNN, respectively. Furthermore, the proposed method achieved a high precision of 98.31%, while the improved values of precision are 2.67%, 2.27%, 1.14%, 1.07%, 1.00%, 0.99%, 0.16%, 0.44%, and 0.98% compared to SVM, MLP, DNN, DCNN, HHOA–CNN, GWOA–DCNN, Hybrid DCNN, HHO–QCNN, and GWO–QCNN, respectively. Overall, the proposed method, incorporating the combined feature representation and QCNN, contributed to high recognition accuracy and outperformed the other baseline methods utilized for the comparison. The overall comparative estimation of the RAVDESS dataset is depicted in Figure 6.

Figure 6.

Comparative estimation for the RAVDESS dataset.

4.6 Comparative Discussion

The TCoSH–DQNN model achieves better recognition accuracy when compared with the conventional methods. SVM-based emotion recognition systems suffer from inefficiency in performance and do not generate accurate prediction results (Yan et al., 2023). The multilayer perceptron classifier contains numerous hidden layers to perform the prediction function; thus, it does not capture the dynamic emotional expression effectively, and also very difficult to handle the input variables, which results in overfitting and computational complexity problems (Joy et al., 2020). For large dependencies in emotional recognition, it is very complex to perform the mathematical calculation, and it increases the computational cost and memory requirements (Kim & Shin, 2019). To achieve effective results, a large amount of data is required in the training phase, and requires an additional attribute function to recognize the dark and bright images, resulting in inaccurate efficiency in expression prediction (Haque & Valles, 2018). The HHOA–DCNN classifier does not capture the fine-grained emotions effectively for bimodal emotion analysis (Halawani et al., 2023). Based on facial emotional expressions, the GWOA–DCNN classifier does not determine the emotional state of the person (Nirmala Sreedharan et al., 2018). Besides, the ALNN-EmR model was not validated for analyzing the multimodal input, limiting the effectiveness of the model (Jain et al., 2025). However, the challenges of traditional emotional recognition methods are addressed by the TCoSH–DQNN model, which achieves effective recognition results. In the method, a TCoSH optimization algorithm is proposed, which reduces the computational complexity issues. Furthermore, the proposed TCoSH–DQNN classifier reduced the overfitting issue, memory requirements problems, and generated accurate emotion recognition. Table 2 depicts the comparative evaluation of the TCoSH–DQNN model with the prior methods using the Enterface’05 dataset. Table 3 demonstrates the comparative discussion of the TCoSH–DQNN model with the existing methods on the RAVDESS dataset.

Table 2

Comparative Assessment of the TCoSH–DQNN Model With the Enterface’05 Dataset.

	90% of Training
Metrics/ Methods	Accuracy (%)	Cohen's kappa score (%)	F1-score (%)	Recall (%)	f-beta score (%)	Precision (%)
SVM (Yan et al., 2023)	76.09	67.06	75.88	82.72	79.04	77.95
MLP (Joy et al., 2020)	81.28	71.28	78.09	84.91	79.88	80.05
DNN (Kim & Shin, 2019)	86.73	73.44	82.02	86.79	81.00	90.80
ALNN-EmR (Jain et al., 2025)	88.80	80.04	87.81	87.68	83.18	92.56
SA-EHO (Devi and Preetha, 2022, May)	91.15	83.10	88.33	88.24	87.95	94.71
DCNN (Haque & Valles, 2018)	92.32	87.27	88.67	91.77	90.01	95.91
HHOA–DCNN (Al-Betar et al., 2023 Feb 1)	95.00	93.23	92.05	93.36	93.84	98.88
GWOA–DCNN (Makhadmeh et al., 2023 Aug 14)	97.85	97.00	97.27	95.00	96.99	99.25
Hybrid DCNN (Verma & Verma, 2020)	99.86	98.50	99.71	99.60	98.56	99.86
HHO–QCNN (Halawani et al., 2023 Oct 1)	98.29	97.60	98.75	99.56	97.20	99.83
GWO–QCNN (Nirmala Sreedharan et al., 2018)	99.67	97.50	97.28	95.07	97.08	99.58
TCoSH–DQNN	99.97	98.61	99.82	99.70	98.67	99.96

Note. TCoSH–DQNN = Taylor Cooperative Share Hunt Optimizer with Deep Quantum Neural Network; SVM = support vector machine; MLP = Multilayer Perceptron; DNN = deep neural network; SA-EHO = self-adaptive model using elephant herding optimization; DCNN = deep convolutional neural network; HHOA = Harris Hawk optimization algorithm; GWOA = grey wolf optimization algorithm; HHO–QCNN = Harris Hawk optimization with quantum convolutional neural network; GWO–QCNN = grey wolf optimization with quantum convolutional neural network.

Table 3

Comparative Assessment of the TCoSH–DQNN Model With the RAVDESS Dataset.

	90% of Training
Metrics/Methods	Accuracy (%)	Cohen's kappa score (%)	F1-score (%)	Recall (%)	f-beta score (%)	Precision (%)
SVM (Yan et al., 2023)	96.09	92.93	96.03	95.57	94.28	95.68
MLP (Joy et al., 2020)	96.18	95.24	97.00	96.25	94.81	96.07
DNN (Kim & Shin, 2019)	96.42	95.32	97.26	96.73	95.36	97.18
ALNN-EmR (Jain et al., 2025)	96.42	95.33	97.26	96.75	95.42	97.20
SA-EHO (Devi and Preetha, 2022)	96.42	95.36	97.26	96.75	96.31	97.21
DCNN (Haque & Valles, 2018)	96.42	95.36	97.26	96.77	96.34	97.25
HHOA–DCNN (Al-Betar et al., 2023)	96.56	95.88	97.30	96.87	96.58	97.32
GWOA–DCNN (Makhadmeh et al., 2023)	96.78	96.14	97.43	97.41	97.29	97.33
Hybrid DCNN (Verma & Verma, 2020)	97.82	96.82	97.72	98.12	98.32	98.15
HHO–QCNN (Halawani et al., 2023)	97.78	96.79	97.72	97.87	98.01	97.87
GWO–QCNN (Nirmala Sreedharan et al., 2018)	97.70	96.55	97.68	97.50	97.73	97.34
TCoSH–DQNN	98.50	97.40	97.93	98.20	98.42	98.31

Note. TCoSH–DQNN = Taylor Cooperative Share Hunt Optimizer with Deep Quantum Neural Network; RAVDESS = Ryerson Audio–Visual Database of Emotional Speech and Song; SVM = support vector machine; MLP = multilayer perceptron; DNN = deep neural network; SA-EHO = self-adaptive model using elephant herding optimization; DCNN = deep convolutional neural network; HHOA = Harris Hawk optimization algorithm; GWOA = grey wolf optimization algorithm; HHO–QCNN = Harris Hawk optimization with quantum convolutional neural network; GWO–QCNN = grey wolf optimization with quantum convolutional neural network.

5. Conclusion

The diverse emotional expressions of humans are recognized by TCoSH–DQNN, which extracts the features from multimodal feature streams. Initially, the input video streams are categorized into audio and visual frames for performing further evaluation. The extracted informative high-level features are evaluated by the proposed TCoSH–DQNN model, which improves the generalization ability by reducing the traditional limitations of prior methods, such as complexity issues and interpretability issues. In addition, the model enhances the convergence rate by implementing the TCoSH optimization techniques that reduce the gradient vanishing problem, overfitting issues, and global optima issues effectively through the integration of the Taylor series function. The TCoSH–DQNN model acquires robust recognition of human emotions and achieves efficient recognition accuracy with various performance evaluation metrics. The TCoSH–DQNN model utilized both the Enterface’05 and RAVDESS datasets, while considering the performance evaluation of the model, the Enterface’05 dataset achieves better evaluation results when compared with the other dataset. The achieved performance evaluation of the Enterface’05 dataset with accuracy is 99.97%, recall is 99.70%, Cohen's kappa score is 98.61%, F1-score is 99.81%, f-beta score is 98.67%, and precision is 99.96%. To improve the performance of the proposed model, additional attention functions or distinct classifiers together with diverse optimization techniques are implemented in future directions.

Footnotes

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

ORCID iDs

Sweta Nishant Padman

Dhiraj Magare

References

Abdusalomov

A. B.

Safarov

Rakhimov

Turaev

Whangbo

T. K.

(2022). Improved feature parameter extraction from speech signals using machine learning algorithm. Sensors, 22, 8122. https://doi.org/10.3390/s22218122

Al-Betar

M. A.

Awadallah

M. A.

Makhadmeh

S. N.

Doush

I. A.

Zitar

R. A.

Alshathri

Abd Elaziz

(2023). A hybrid Harris hawks optimizer for economic load dispatch problems. Alexandria Engineering Journal, 64, 365–389. https://doi.org/10.1016/j.aej.2022.09.010

Almulla

M. A.

(2024). A multimodal emotion recognition system using deep convolution neural networks. Journal of Engineering Research, 13(4).

Baltrusaitis

Ahuja

Morency

L.-P.

(2019). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423–443. https://doi.org/10.1109/TPAMI.2018.2798607

Chandraumakantham

Gowtham

Zakariah

Almazyad

(2024). Multimodal emotion recognition using feature fusion: An LLM-based approach. IEEE Access, 12, 108052–108071. https://doi.org/10.1109/ACCESS.2024.3425953

Chen

Tang

Zhu

Kong

(2023). A multi-stage dynamical fusion network for multimodal emotion recognition. Cognitive Neurodynamics, 17(3), 671–680. https://doi.org/10.1007/s11571-022-09851-w

Chen

S. Y.

Wei

T. C.

Zhang

Yoo

(2022). Quantum convolutional neural networks for high energy physics data analysis. Physical Review Research, 4(1), 013231. https://doi.org/10.1103/PhysRevResearch.4.013231

Devi

Preetha

M. M. S. J.

(2022). Impact of self adaptive-elephant herding optimization towards neural network for facial emotion recognition. Web Intelligence, 20(1), 67–81. https://doi.org/10.3233/WEB-210481

Dwaich

H. A.

Abdulbaqi

H. A.

(2021) Signature texture feature extraction using the GLCM approach in Android Studio. Journal of Physics: Conference Series, 1804(1), 012043.

10.

Ekman

(1992). An argument for basic emotions. Cognition and Emotion, 6(3–4), 169–200. https://doi.org/10.1080/02699939208411068

11.

El-Latif

E. I.

El-Sayad

N. E.

Mohammed

K. K.

Darwish

Hassanien

A. E.

(2024). VGGish transfer learning model for the efficient detection of payload weight of drones using Mel-spectrogram analysis. Neural Computing and Applications, 36, 12883–12899.

12.

Enterface’05 Dataset (2024). https://enterface.net/enterface05/main.php?frame=emotion

13.

Farkhod

Abdusalomov

Makhmudov

Cho

Y. I.

(2021). LDA-based topic modeling sentiment analysis using topic/document/sentence (TDS) model. Applied Science, 11, 11091. https://doi.org/10.3390/app112311091

14.

Ghaleb

Niehues

Asteriadis

(2023). Joint modeling of audio-visual cues using attention mechanisms for emotion recognition. Multimedia Tools and Applications, 82(8), 11239–11264. https://doi.org/10.1007/s11042-022-13557-w

15.

Goncalves

Busso

(2022). Robust audiovisual emotion recognition: Aligning modalities, capturing temporal information, and handling missing features. IEEE Transactions on Affective Computing, 13(4), 2156–2170. https://doi.org/10.1109/TAFFC.2022.3216993

16.

Greco

Barra

D’Errico

Staffa

. (2024). Multimodal interfaces for emotion recognition: Models, challenges and opportunities. In International conference on human–computer interaction (pp. 152–162). Springer Nature.

17.

Halawani

H. T.

Mashraqi

A. M.

Badr

S. K.

Alkhalaf

(2023). Automated sentiment analysis in social media using Harris Hawks optimization and deep learning techniques. Alexandria Engineering Journal, 80, 433–443. https://doi.org/10.1016/j.aej.2023.08.062

18.

Haque

M. I.

Valles

(2018). A facial expression recognition approach using DCNN for autistic children to identify emotions. In 2018 IEEE 9th annual information technology, electronics and mobile communication conference (IEMCON) (pp. 546–551). IEEE.

19.

Houssein

E. H.

Abohashima

Elhoseny

Mohamed

W. M.

(2021). An efficient binary Harris Hawks optimization based on quantum SVM for cancer classification tasks. In The 2nd international conference on distributed sensing and intelligent systems (ICDSIS 2021) (Vol. 2021, pp. 247–258). IET.

20.

Hsu

J. H.

M. H.

C. H.

Chen

Y. H.

(2021). Speech emotion recognition considering nonverbal vocalization in affective conversations. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 1675–1686. https://doi.org/10.1109/TASLP.2021.3076364

21.

Huang

Tao

Liu

Lian

Niu

(2020). Multimodal transformer fusion for continuous emotion recognition. In ICASSP 2020–2020 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 3507–3511). IEEE.

22.

Jain

Shaikh

Tanwani

(2025). An efficient emotion recognition model using Bald Hawk optimization based on deep learning. Web Intelligence, 23(3), 319–337. https://doi.org/10.1177/24056456251320118

23.

Joy

Kannan

Ram

Rama

(2020). Speech emotion recognition using neural network and MLP classifier. International Journal of Engineering and Computer Science, 2020, 25170–25172.

24.

Kansizoglou

Bampis

Gasteratos

(2019). An active learning paradigm for online audio-visual emotion recognition. IEEE Transactions on Affective Computing, 13(2), 756–768. https://doi.org/10.1109/TAFFC.2019.2961089

25.

Kim

Shin

J. W.

(2019). DNN-based emotion recognition is based on bottleneck acoustic features and lexical features. In ICASSP 2019–2019 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 6720–6724). IEEE

26.

Kumar

Malik

Raman

(2024). Interpretable multimodal emotion recognition using a hybrid fusion of speech and image data. Multimedia Tools and Applications, 83(10), 28373–28394. https://doi.org/10.1007/s11042-023-16443-1

27.

Kurniati

F. T.

Manongga

D. H.

Sediyono

Prasetyo

S. Y.

Huizen

R. R.

(2024) GLCM-based feature combination for extraction model optimization in object detection using machine learning. arXiv preprint arXiv:2404.04578.

28.

Zhou

R. G.

Luo

(2020). A quantum deep convolutional neural network for image recognition. Quantum Science and Technology, 5(4), 044003. https://doi.org/10.1088/2058-9565/ab9f93

29.

Liu

Huang

(2023). Multimodal emotion recognition based on cascaded multichannel and hierarchical fusion. Computational Intelligence and Neuroscience, 2023(1), 9645611. https://doi.org/10.1155/2023/9645611 .

30.

Lohithashva

B. H.

Aradhya

V. N.

Guru

D. S.

(2020). Violent video event detection based on integrated LBP and GLCM texture features. Revue d'Intelligence Artificielle, 34(2), 179–187. https://doi.org/10.18280/ria.340208

31.

Makhadmeh

S. N.

Al-Betar

M. A.

Doush

I. A.

Awadallah

M. A.

Kassaymeh

Mirjalili

Zitar

R. A.

(2024). Recent advances in Grey Wolf Optimizer, its versions, and applications. IEEE Access, 12, 22991–23028. https://doi.org/10.1109/ACCESS.2023.3304889

32.

Makhmudov

Mukhiddinov

Abdusalomov

Avazov

Khamdamov

Cho

Y. I.

(2020). Improvement of the end-to-end scene text recognition method for “text-to-speech” conversion. International Journal of Wavelets, Multiresolution and Information Processing, 18, 2050052. https://doi.org/10.1142/S0219691320500526

33.

Mamieva

Abdusalomov

A. B.

Kutlimuratov

Muminov

Whangbo

T. K.

(2023). Multimodal emotion detection via attention-based fusion of extracted facial and speech features. Sensors, 23(12), 5475. https://doi.org/10.3390/s23125475

34.

Meng

Shou

Yin

(2024). Deep imbalanced learning for multimodal emotion recognition in conversations. IEEE Transactions on Artificial Intelligence, 5(12), 6472–6487. https://doi.org/10.1109/TAI.2024.3445325

35.

Nirmala Sreedharan

N. P.

Ganesan

Raveendran

Sarala

Dennis

Boothalingam

R. R.

(2018). Grey wolf optimization-based feature selection and classification for facial emotion recognition. IET Biometrics, 7(5), 490–499. https://doi.org/10.1049/iet-bmt.2017.0160

36.

Pan

Fang

Zhang

Chen

Zhang

Wang

(2023). Multimodal emotion recognition based on facial expressions, speech, and EEG. IEEE Open Journal of Engineering in Medicine and Biology, 5, 396–403. https://doi.org/10.1109/OJEMB.2023.3240280

37.

Prabakaran

Sriuppili

(2021). Speech processing: MFCC based feature extraction techniques investigation. Journal of Physics: Conference Series, 1717(1), 012009. https://doi.org/10.1088/1742-6596/1717/1/012009

38.

Praveen

R. G.

Cardinal

Granger

(2023). Audio–visual fusion for emotion recognition in the valence-arousal space using joint cross-attention. IEEE Transactions on Biometrics, Behavior, and Identity Science, 5(3), 360–373. https://doi.org/10.1109/TBIOM.2022.3233083

39.

Ryerson Audio-Visual Database of Emotion Speech and Song (RAVDESS) (2024). https://www.kaggle.com/datasets/uwrfkaggler/ravdess-emotional-speech-audio

40.

Siriwardhana

Kaluarachchi

Billinghurst

Nanayakkara

(2020). Multimodal emotion recognition with transformer-based self-supervised feature fusion. IEEE Access, 8, 176274–85. https://doi.org/10.1109/ACCESS.2020.3026823

41.

Verma

(2020). Hybrid-deep learning model for emotion recognition using facial expressions. The Review of Socionetwork Strategies, 14(2), 171–180. https://doi.org/10.1007/s12626-020-00061-6

42.

Vijayvergia

Kumar

(2021). Selective shallow models strength integration for emotion detection using GloVe and LSTM. Multimedia. Tools Appl, 80, 28349–28363. https://doi.org/10.1007/s11042-021-10997-8

43.

Vinitha

Rukmini

M. S.

(2022). Secure and energy-aware multi-hop routing protocol in WSN using Taylor-based hybrid optimization algorithm. Journal of King Saud University-Computer and Information Sciences, 34(5), 1857–1868. https://doi.org/10.1016/j.jksuci.2019.11.009

44.

Wang

Chen

Cao

(2020). Human emotion recognition by optimally fusing facial expression and speech feature. Signal Processing: Image Communication, 84, 115831. https://doi.org/10.1016/j.image.2020.115831

45.

Yan

Lin

Vucetic

(2023). A novel exploitative and explorative GWO–SVM algorithm for smart emotion recognition. IEEE Internet of Things Journal, 10(1), 9999–10011. https://doi.org/10.1109/JIOT.2023.3235356

46.

Zhao

Wang

Jia

Wang

(2019). Feature selection of grey wolf optimizer based on quantum computing and uncertain symmetry rough set. Symmetry, 11(12), 1470. https://doi.org/10.3390/sym11121470