Localization and classification of human facial emotions using local intensity order pattern and shape-based texture features

Abstract

Facial emotion recognition system (FERS) recognize the person’s emotions based on various image processing stages including feature extraction as one of the major processing steps. In this study, we presented a hybrid approach for recognizing facial expressions by performing the feature level fusion of a local and a global feature descriptor that is classified by a support vector machine (SVM) classifier. Histogram of oriented gradients (HoG) is selected for the extraction of global facial features and local intensity order pattern (LIOP) to extract the local features. As HoG is a shape-based descriptor, with the help of edge information, it can extract the deformations caused in facial muscles due to changing emotions. On the contrary, LIOP works based on the information of pixels intensity order and is invariant to change in image viewpoint, illumination conditions, JPEG compression, and image blurring as well. Thus both the descriptors proved useful to recognize the emotions effectively in the images captured in both constrained and realistic scenarios. The performance of the proposed model is evaluated based on the lab-constrained datasets including CK+, TFEID, JAFFE as well as on realistic datasets including SFEW, RaF, and FER-2013 dataset. The optimal recognition accuracy of 99.8%, 98.2%, 93.5%, 78.1%, 63.0%, 56.0% achieved respectively for CK+, JAFFE, TFEID, RaF, FER-2013 and SFEW datasets respectively.

Keywords

Facial emotion recognition histogram-of-oriented-gradients local intensity order pattern support vector machine texture features

1 Introduction

Emotions are associated with the brain’s nervous system and provide useful information about a person’s state of mind. These are the particular reactions taken by human beings in the form of variations in their speech tone, body gestures, written text, or facial expressions as a result of the change in their internal thoughts or external events that take place in the outside world. Emotions can be represented based on the 2D valence-arousal or circumplex model [1] and basic models. Ekman et al. [2, 3] analyzed the emotional variations of the subjects to propose a basic emotion model and concluded that facial expressions of the six basic emotions namely fear (Fe), anger (An), sadness (Sa), disgust (Di), surprise (Su) and happiness (Ha) remain constant and can be observed in cross-cultures. FERS have a wide range of applications in real-life such as in distance learning [4], surveillance [5], affective computing [6], healthcare [7 –11], etc.

Facial expressions describe the changes that take place on the face according to the person’s intentions. In interpersonal communication, a person’s facial expressions play an essential role. It is shown by a research study that the speaker’s facial expressions convey a 55% impact of the spoken message while voice intonation contributes 38% and spoken words contribute merely 7% of the overall message effect [12]. Automatic ER from facial expressions has been a challenging and interesting task for over 30 years. The main challenges in the development of the FERS are changes in illumination conditions, face pose, emotion variability among individuals, facial image viewpoint (frontal or profile view), image scaling, and skin color complexion (refer to Fig. 1). Several approaches have been proposed in the literature to overcome these challenges but still, no universal algorithm exists that can perform effectively in complex realistic scenarios along with constrained scenarios.

Fig. 1

Variation in face texture.

FERS can be developed based upon the action units [13]; feature-based approaches including geometric features [14 –16], texture features [17 –22], a combination of both geometric and texture features [23]; model-based approaches i.e. active shape models (ASM) [24], active appearance models (AAM) [25] and deep learning-based approaches [26 –28]. Deep transfer learning is an advanced approach proposed in deep learning in which information from one model transfers to another model. It enables the researchers to solve any particular task by utilizing the full or a part of the model pre-trained on a different task. It’s also been used for classification [29, 30] as well as in different classification tasks like detection and classification of COVID-19 patients [31 –33]. All the mentioned emotion recognition approaches provide optimal performance in particular scenarios. Deep learning-based approaches provide promising results but require quite a large size dataset for network training. While those based on both geometric features and facial models recognize emotions by measuring the distance variation among facial parts e.g. eyes, mouth, etc. It’s quite challenging to train geometric feature-based approaches for person independent ER as everyone’s facial deformation is different depending upon the size and shape of a face. Geometric features are also more likely to ignore the changes in facial texture while appearance or texture base features encode pixel values and are more sensitive to the changes in facial texture. Whitehill et al. [34] reported that appearance-based approaches provide superior performance than geometric features. These descriptors have proved very successful for the image processing applications like object recognition, image matching, and object categorization [35]. Several researchers reported that these descriptors result in better recognition accuracy for the development of FERS [36 –38]. However, all these descriptors have varying advantages and shortcomings. That is why the complementary feature vectors obtained by the combination of more than one descriptor help to overcome their shortcomings.

In the proposed research work, a model for FERS is presented based on facial texture-based features to recognize the emotions in both constrained and realistic environment. Initially, color images are converted to grayscale to minimize the computation cost. The facial area is then extracted from the input image with the help of the viola-jones algorithm [39]. As face size varies from person to person causing unequal feature extraction, so it is further resized through various interpolation techniques and discrete wavelet transformation (DWT). We extracted the image visual information with the help of LIOP being a local and HoG being a global descriptor. We complimented them together to get the global facial structural information as well as the deformation caused in local facial components like eyes, mouth, etc. as a result of emotional changes. The complemented features are invariant to several geometric and photometric transformations. The HoG [40] descriptor is originally proposed for the pedestrians’ detection and is a shape descriptor. It counts the gradient orientations in a localized image region for edge analysis. As the deformations caused in facial muscles due to changing emotions can also be analyzed by edges making HoG a suitable descriptor for emotion classification [41 –44]. On the other hand, the LIOP descriptor [45] describes the image based on the pixels’ intensity order. It is invariant to several image transformations as a change in viewpoint, illumination conditions, JPEG compression, and image blurring as well. Inspiring by the distinguishing nature of both these texture descriptors, we complemented them together for classifying emotions. HoG considers the individual pixels for computing the orientation and magnitude of gradient while LIOP based on pixels’ statistical intensity order also considers the relationship among all neighborhoods. So both HoG and LIOP descriptors have proved very useful to recognize the facial expressions based on the images captured in constrained as well as in complex realistic scenarios having varying illumination conditions, viewpoint, blurring effects, etc. Features classified by SVM classifier by varying kernel functions and optimal results achieved by the polynomial kernel.

The major contributions of this study are as follows:

A novel model is proposed to recognize facial emotions based on complementary features.

Feature level fusion of pixel intensity order based descriptor and histogram of oriented gradients based descriptor is performed for the extraction of distinguishing features being invariant to illumination conditions, image viewpoint, rotation, blurring, and JPEG compression.

The performance of the proposed model is evaluated on the publicly available datasets, which contain images of both lab constrained and realistic unconstrained environments. Enhancement in recognition accuracy under a realistic environment having large variations in illumination conditions, viewpoints, etc. is also highlighted.

Image scaling is performed by discrete wavelet transformation and various interpolation techniques to minimize the computational resources.

The remaining part of the paper is organized such that section 2 presents an overview of the related work. The proposed FERS model is discussed in section 3, section 4 provides a detailed description of the results and evaluation of the proposed model and section 5 concludes it.

1.1 Motivation and justification of the proposed model

The motivation behind the development of the ER framework is that such systems have a lot of applications in real life as in distance learning, medical, etc. These systems can be developed by analyzing the spoken words, written text, body gestures, or facial expressions. From all modalities, facial expressions are widely used means for social communication and have the highest contribution to convey the message [12, 46].

Facial expressions can be recognized through traditional machine learning (TML) and modern deep learning-based approaches (DLBA). DLBA provides promising results but requires quite a large size dataset for the successful training of the network. TML approaches results promising accuracy rate even for small datasets. Such approaches can be developed based on facial texture features or geometric features. It’s quite challenging to train geometric feature-based approaches for person independent ER as everyone’s facial deformation is quite different depending upon the size and shape of one’s face. Geometric features are also more likely to ignore the changes in facial texture while appearance features encode pixel values and are more sensitive to the changes in facial texture. However, various appearance descriptors have varying advantages and shortcomings. That is why we complemented global and local feature descriptors to overcome their shortcomings. Features are classified by SVM that is a widely used supervised learning classifier.

2 Related works

In this section, an overview of state-of-the-art ER approaches is presented. Such systems evolved with the time to tackle the challenges faced in a realistic environment. Many years ago, darwin gave the concept of emotional existence in humans and animals [47]. Ekman and frisen extended his idea and developed a facial action coding system (FACS). The visual movement of the facial components e.g. eyes, mouth, etc. known as action units (AU) is monitored in these systems. The contractions or expansions caused in AUs are further encoded by FACS to recognize the varying emotions. Each emotion is represented by a group of particular AUs e.g. happiness is represented by two AUs that are lip corner puller and cheeks raiser. The benefit of FACS is that each expressive image can be categorized to a particular emotion by encoding the corresponding AUs while the major shortcoming is that it requires a lot of time to analyze each image one by one [48].

Recently, texture feature-based approaches are preferred for FERS because of being invariant to various image transformations as image scaling, illumination conditions, etc. [35]. Several attempts are carried out for the development of FERS based on texture features. Luo et al. [37] proposed a FERS framework based on the texture feature descriptors named improved completed local ternary patterns (ICLTP). A facial area having 150×110 dimensions is extracted from the original image by determining the eyes’ position on the frontal face image. Initially, the image gradient is computed with the help of a Scharr operator that provides better image details than a Sobel operator. Secondly, CLTP features are extracted from 7×6 gradient image patches by consideration of two groups of neighborhoods i.e. 8 neighbors with radius one and 16 neighbors with radius 3. Features based on both neighborhoods are then concatenated together to generate the final feature vector to be used as an input to KNN and sparse representation classifier for 10-fold cross-validation. Their proposed framework resulted in optimal recognition accuracy based on frontal face images captured in a constrained environment. A FERS based on local phase quantization (LPQ) feature descriptor is proposed by kherchaoui et al. [20]. LTP coding and thresholding concept proposed for quantization of local phases better than conventional LPQ. Face and eyes’ region was detected by the viola-jones algorithm, core facial area then normalized by 128×128 pixels upon which later on histogram equalization applied. Facial features extracted from frontal face images by texture-based adapted gradient LPQ descriptor that further classified by SVM classifier. Hu et al. [22] proposed a novel feature descriptor based on local binary pattern (LBP) and center symmetric LBP named center-symmetric local octonary pattern (CS-LOP) for feature extraction. Eight neighbors were considered for computing feature descriptors similar to LBP including four center-symmetric pixels for reduced feature vector computation based on CS-LBP. Unlike LBP and CS-LBP, they also consider the contribution of the central pixel. Detailed CS-LOP features extracted by feature map of Gabor and gradient magnitude features that further fused as a result of feature-level fusion. SVM with polynomial kernel trained for classification based on person-independent (N-person) and person-dependent (N-fold) cross-validation for frontal face images captured in a constrained environment. FERS with the help of a 3D facial model proposed by qi et al. [49]. Initially, the facial area was detected based on the convex hull formed by LBP codes of four facial regions i.e. left, right part of the forehead, and zygomatic to the chin area. Secondly, the 3D facial model was established to divide the facial area into six sub-regions including eyes, nose, forehead, mouth, and cheeks area based on cognition. Promising results were achieved by consideration of mouth and eyes region. Dimensionality of extracted LBP features then reduced by multi-dimensional scaling (MDS) technique based upon earth movers distance. Finally, emotions were recognized based on SVM and convolutional neural network (CNN) classifier with softmax activation function by employing two emotion models i.e. basic discrete model and circumplex model. Higher recognition accuracy is achieved by the circumplex model and softmax classifier for frontal face images captured in a constrained environment.

Several researchers explored the HoG descriptor individually as well as by combining it with other descriptors to recognize facial expressions. Donia et al. [42] presented a framework to recognize the posed and spontaneous facial expressions based on HoG features extracted from frontal face images and SVM classifiers with a linear kernel. Face detected and extracted by continuously adaptive mean shift (CAMshist) algorithm. Facial feature points were marked manually for extraction of six facial parts i.e. eyebrows, an area between eyes, left and right eye, mouth, and nose for feature extraction. It requires a lot of time for manual feature region marking and extraction for a dataset comprising a large number of images. Later on, Carcagni et al. [43] presented a detailed review regarding HoG’s suitability for recognizing facial expressions and proposed an algorithm for frontal face images detected by the viola-jones algorithm. Experiments were performed using 10-fold cross-validation with the SVM classifier. Experimental results were evaluated by varying the number of bins and cell size and optimal accuracy achieved based on 7-pixel cell size and 7 orientation bins. Sajjad et al. [44] proposed the FERS framework based on HoG, uniform local ternary pattern (U-LTP) features, and multiclass SVM classifier. Initially, noise and blurring effects were removed from the image by applying 3×3 median filters and histogram equalization. Face detected based on the viola-jones algorithm further resized to 128×128 dimensional image by adaptive interpolation technique. Optimal results were achieved based on 9 bins histogram per cell, 16×16 cell size, and 2×2 block size having a 50% overlapping area for HoG features. While 59 bins based two histograms built for LTP separately for upper and lower binary patterns resulting 118 dimensional feature vector. One-vs-rest multi-class SVM with linear kernel used for classification purpose. The proposed framework resulted promising results for frontal faces while profile faces were not considered for experimentation purposes.

3 Methodology

In this section, a detailed description of the proposed FERS is presented that is developed based on facial texture features (refer to Fig. 2). It comprises of major three steps i.e. image collection discussed in subsection 4.1, image pre-processing preceded by emotion classification described in the following subsections.

Fig. 2

A framework of the proposed model of FER.

3.1 Image pre-processing

Nowadays, pixels per image are increasing day by day for the incorporation of more detail that becomes possible with the advancement in image acquisition techniques and technology. Various devices capture color images having a distinct size and contrast. As color images having three components require large computation power so initially in this step, images are converted into grayscale having one color component and also normalized. Originally captured images comprised of a lot of complex background areas along with the facial part that contributes to emotions. So facial area detected based on the viola-jones algorithm [39] and extracted from an input image for further feature extraction. As detected faces are of varying sizes resulting in an unequal feature vector length, also the LIOP descriptor requires a square image having an odd side length. So DWT and various interpolation techniques are employed to resize all images to an average size as we show in Fig. 3. Interpolation can both scale down and scale up the image’s dimension. While DWT low-low (LL) band image part is comprised of a lot of image detail and reduce the image size to one-fourth. Nearest interpolation simply copies the value of the nearest pixel to assign to the new pixel while bilinear and bicubic interpolation consider the average value of 2 and 3 pixels respectively. Experiments are performed by scaling images directly to an average size as well as applying DWT first and then resizing to an average size based on various interpolation techniques. Experimental results show that images directly resized based on bicubic interpolation resulted in comparatively high recognition accuracy than those resized after DWT (refer to Tables 3 –6).

Fig. 3

Face detected based on the viola-jones algorithm and resized based on bicubic interpolation.

Table 1

Statistical measures for performance analysis of the proposed model

True positive rate (TPR) /	$\frac{TP}{TP + FN}$	False-positive rate (FPR)	$\frac{FP}{FP + TN}$
sensitivity/ Recall
True negative rate (TNR)/	$\frac{TN}{TN + FP}$	False-negative rate (FNR)	$\frac{FN}{TP + FN}$
specificity
Positive prediction value	$\frac{TP}{TP + FP}$	Accuracy (ACC)	$\frac{TP + TN}{TP + TN + FP + FN}$
(PPV)/precision

Table 2

Optimal values of the proposed model hyper-parameters

Descriptor/classifier	Dataset	Values of hyper-parameters
HoG descriptor	CK+ and JAFFE	Number of bins = 7;
		Cell Size = 32×32;
		Block size = 2×2 with 50% overlapping area;
		Descriptor’s length = 400.
	TFEID, SFEW, and RaF	Number of bins = 9;
		Cell Size = 24×24;
		Block size = 2×2 with 50% overlapping area;
		Descriptor’s length = 775.
	FER-2013	Number of bins = 9;
		Cell size = 16×16;
		Block size = 2×2 with 50% overlapping area;
		Descriptor’s length = 279.
LIOP descriptor	For all datasets	Number of neighbors = 4;
		Number of sub-regions = 6;
		Descriptor’s length = 144.
Complementary descriptor	FER-2013	Descriptor’s length = 423
	CK+ and JAFFE	Descriptor’s length = 544
	TFEID, SFEW, and RaF	Descriptor’s length = 919
SVM classifier	For all datasets	C = 1, Gamma = 1;
		A polynomial kernel with degree 3.

Table 3

Classification results for CK+ dataset based on LIOP descriptors and SVM polynomial kernel

Descriptor’s size	Interpolation	Images directly resized to 129×129			DWT then resized
		Linear	RBF	Polynomial	Polynomial
144 (N = 4)	Nearest	68.3	78.9	92.0	74.8
	Bilinear	70.5	79.1	93.4	76.0
	Bicubic	70.6	79.5	92.7	77.5
4320 (N = 6)	Nearest	86.4	93.3	99.4	95.4
	Bilinear	88.1	94.5	99.7	96.0
	Bicubic	88.4	95.1	99.7	95.3

Table 4

Recognition accuracy for the JAFFE dataset based on LIOP descriptors and SVM polynomial kernel

Descriptor’s size	Interpolation	Images resized directly to 129×129	DWT then resized
144 (N = 4)	Bilinear	70.2	52.6
	Bicubic	78.9	56.1
4320 (N = 6)	Bilinear	86.0	77.2
	Bicubic	87.7	73.7

Table 5

Recognition accuracy for the TFEID dataset based on LIOP descriptors and SVM polynomial kernel

Descriptor’s size	Interpolation	Images resized to the shorter side	Images resized to a longer side	Images resized to 129×129
				Directly resized	DWT then resized
144 (N = 4)	Bilinear	73.9	76.1	67.4	63.0
	Bicubic	73.9	76.1	69.6	60.9
4320 (N = 6)	Bilinear	73.9	78.3	80.4	73.9
	Bicubic	73.9	78.3	84.8	84.6

Table 6

Recognition accuracy for SFEW dataset based on LIOP descriptors and SVM polynomial kernel

Descriptor’s size	Interpolation	Images resized to the shorter side	Images resized to a longer side	Images resized to 129×129
				Directly resized	DWT then resized
144 (N = 4)	Bilinear	35.8	34.9	36.4	28.7
	Bicubic	33.0	33.6	37.1	30.5
4320 (N = 6)	Bilinear	31.5	33.0	32.7	30.5
	Bicubic	31.5	33.3	28.0	34.3

3.2 Facial feature extraction and fusion

In this subsection, we discuss the feature extraction algorithms. Texture based features are extracted from the preprocessed facial part based on the LIOP and HoG feature descriptors that further complemented together by feature level fusion. We discuss the feature extraction and fusion algorithms in subsections 3.2.1–3.2.3. We fine-tune the descriptors’ parameters by repetitive experiments to enhance recognition accuracy (refer to Tables 3 –6).

3.2.1 LIOP based facial feature extraction

For feature extraction, we initially resized the image to a square size image having an odd side length i.e. 129. The facial region is then divided into sub-regions based on the pixels’ intensity order. The parameters that we defined for LIOP are R and N, where R shows the number of sub-regions (r₁, r₂, r₃..., r_R) to which the facial part will be divided (refer to Fig. 4) and N represents the number of neighbors for each pixel P_i in r_j region (refer to Fig. 5).

Fig. 4

Facial region division into sub-regions based on pixels intensity value.

Fig. 5

LIOP feature vector computation.

Consider a set P^N of N-dimensional vectors like (p₁, p₂,... p _N) defined as P^N = (p₁, p₂,... p _N): p_iεR where each element Fig_1 εP^N is a vector representing the selected N neighbors of a pixel ‘P’ and is arranged in non-descending order such that: $p_{i_{1}} ⩽ p_{i_{2}} ⩽ \dots ⩽ p_{i_{N}}$ (1)

In Equation (1), a particular pixel p_{i
₁}≤p_{i
₂} if and only if $\overset{⌢}{I}$ (p_{i
₁})< $\overset{⌢}{I}$ (p_{i
₂}) or $\overset{⌢}{I}$ (p_{i
₁}) = = $\overset{⌢}{I}$ (p_{i
₂}) but i₁ <i₂. Here $\overset{⌢}{I}$ represents the pixel’s intensity value. All neighbors are arranged circularly with radius r around a pixel P, pixels’ non-descending order remains the same after the rotation so both p_{i
₁} and its rotated pixel $p_{i_{1}}^{'}$ belongs to the same partition Fig_1 in P^N (refer to (1)).

Let suppose an index table δ^N for N = 3 having N! rows and two columns pointing to the unique ordered arrangement of Fig_1 elements and corresponding index respectively. A mapping function Ω: P^N → δ^N is defined as Ω (Fig_1) =λ; Fig_1 ε P^N ; λεδ^N to map each Fig_1 ε P^N to a particular permutation λεδ^N according to the intensity-based order. λ= (i₁, i₂, ... ,i_N) in the index table shows the permutation of N neighbors.

The procedure for the LIOP feature vector computation of a particular pixel is represented in Fig. 5. LIOP (P) represents the feature vector of a particular pixel P, similarly feature vectors of all the pixels in r_i sub-region computed that resulted in des (r_i). LIOP of the entire facial area LIOP_I is computed by concatenation of all des(r_i) as given in the following equation. $\begin{matrix} {LIOP}_{I} = (des (r_{1}), des (r_{2}), \dots ., des (r_{R})) \\ des (r_{i}) = \sum_{P ɛ r} LIOP (P) \end{matrix}$ (2)

3.2.2 HoG feature vector computation

HOG descriptor is primarily used for the extraction of shape-based features, originally proposed for pedestrian detection [40]. It describes the image based on texture information. Major steps that we performed for the computation of the HoG descriptor are gradient computation, histogram binning, block normalization and the last one is feature vector computation as briefly discussed below.

3.2.2.1. Gradient computation

The HoG feature vector is computed based on the orientation and magnitude of a gradient in the selected image patch. Gradient magnitude represents the rate of change in the pixels’ values both in a horizontal and vertical direction, it’s computed by convolving the image with a kernel function horizontally to compute Fig_1_x and vertically to compute Fig_1_y: ${Fig_1}_{x} = {- 1 0 1} {Fig_1}_{y} = {\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix}},$

Overall image gradient Fig_1 then computed based on Fig_1_x and Fig_1_y as shown in (3). $Fig_1 = \sqrt{{Fig_1}_{x}^{2} + {Fig_1}_{y}^{2}}$ (3)

Gradient orientations represent the change in edge direction caused because of facial components’ deformation during emotional change. It’s computed with the help of Fig_1_x and Fig_1_y as defined in Equation (4): $θ = \arctan \frac{{Fig_1}_{y}}{{Fig_1}_{x}}$ (4)

Gradient orientations can be 0–360⁰ known as signed gradient and 0–180⁰ called unsigned gradients. We achieved optimal results based on an unsigned gradient. Gradient magnitude varies depending upon the expressed emotion e.g. it’s quite large for a surprise while quite small for sad emotion.

3.2.2.2. Histogram binning

The number of histogram bins à needs to be defined based on gradient orientations in an image cell. Cell size is inversely proportional to the feature vector length. Too small cell size extract irrelevant features and increase the dimensionality requiring high computation power. While too large cell size lack in image detail (refer to Figs. 6-7). Experimental results evaluated by varying value of à and optimal results achieved by 7 bins histogram for CK+ and JAFFE while 9 bins histogram for TFEID, SFEW, RaF, and FER-2013 datasets. The value of a particular bin à_i is computed by gradient magnitude value against orientation angle θ lying in its range.

Fig. 6

HoG feature descriptors computation based on varying cell size (CS).

Fig. 7

Bar diagram of the HoG feature vector for varying cell sizes and histogram bins.

3.2.2.3. Block normalization

1-D histograms computed based on edge orientations in multiple cells forming a block are further concatenated together resulting in 1×D vector ƛ. The dimension of ƛ is 1×(à * ß) where à represents the number of bins as discussed in subsection 3.2.2.2 and ß denotes the number of cells in a block. Vector ƛ is then normalized by L₂ (euclidean) norm to make it invariant to illumination conditions and image rotation (refer to (5)). $ƛ = \sqrt{ƛ_{1}^{2} + ƛ_{2}^{2} + . . . . + ƛ_{(a * ß)}^{2}}$ (5)

Block and cell size affect recognition accuracy. We perform experiments by varying the discussed parameters. It is shown that best recognition results achieve by 24×24 cell size, 2×2 block size with 50% block overlapping area resulting in 775-dimensional feature vector for 129×129 dimensional TFEID, SFEW, and RaF dataset images; cell size reduced to 16×16 for 48×48 dimensional FER-2013 dataset images which resulted in a 279-dimensional feature vector. While 32×32 cell size, 2×2 block size with 50% overlapping area resulting in 400-dimensional feature vector generated optimal results for lab-constrained CK+ and JAFFE dataset.

3.2.2.4. Image feature vector

The HoG feature vector of an entire image HoG_I is computed by concatenation of 1-D ƛ vector of all blocks having 1×(ƛ*x*y) dimension where x denotes the number of horizontal and y denotes the number of vertical blocks in an image.

3.2.3 Feature level fusion

LIOP_I and HoG_I features are extracted separately as discussed in section 3.2.1 and 3.2.2 respectively from the resized facial area (refer to Fig. 3). Initially, we perform emotion classification based on LIOP feature descriptors by varying descriptor parameters and multiclass SVM kernel functions. Optimal results are achieved for CK+, JAFFE, and TFEID dataset by quite a large 4320-dimensional feature vector as shown in Tables 3 –5. While for the SFEW dataset 144-dimensional feature vectors resulted in better recognition accuracy as presented in Table 6. From the literature study, we came to know that complementary feature descriptors can result in better recognition accuracy. Carcagni et al. [41] showed that the HoG descriptor is quite useful for FER. As more detail is extracted from the image is sometimes able to increase the recognition accuracy. So inspiring from [41], we then extracted HoG features as well by varying parameters and concluded that a 400-dimensional feature vector resulted in optimal results for CK+ and JAFFE datasets. Comparatively high dimensional HoG feature vector i.e. 775 require for realistic datasets as RaF, TFEID, and SFEW while the 279-dimensional feature vector requires for comparatively smaller size FER-2013 images. We then performed feature level fusion of LIOP_I and HoG_I feature vectors extracted based on fine-tuned parameters. Experimental results show that a low dimensional hybrid descriptor ᵮ^{(LIOPi +HoGi)} resulted in recognition accuracy comparable to quite larger 4320-dimensional LIOP feature vectors that are computationally expensive. Finally, the classification performed based on a training matrix having (Ñ x n) dimensions where Ñ represents the number of images in a dataset while n is 544 for CK+, JAFFE, 423 for FER-2013 dataset and its value is 919 for more challenging datasets including RaF, TFEID and SFEW dataset.

3.3 Classification

In this section, we discussed the detail of the classification mechanism based on the SVM classifier. Experiments are performed based on supervised learning. We used training matrices with corresponding emotion labels as an input to the classifier. We repeated the experiments based on various SVM kernel functions. It is shown by experimental results that optimal recognition accuracy achieved for polynomial kernel having degree 3 as represented in Table 3.

3.3.1 Support vector machine

SVM is a classification algorithm originally proposed for binary classification where data lies in a 2-D plane [50]. It separates the linearly separable data belonging to distinct classes by finding a hyperplane based on w . x - b = 0, where $\frac{b}{| | w | |}$ represents the hyper-plane offset. SVM binary classification problem can be define according to (6). ${\begin{matrix} ω . x - b ⩾ 1 \forall x_{i} \in C_{1} \\ ω . x - b ⩽ - 1 \forall x_{i} \in C_{2} \end{matrix} = y_{i} (ω . x - b) ⩾ 1 \forall x_{i}$ (6)

In (6) C₁, C₂ represents the classes and x_i shows instances. Two possible extensions of the SVM classifier for multiclass classification problems are one-vs-rest and one-vs-one [51]. In one-vs-rest, ‘N’ binary classifiers (N shows the number of classes) are trained. While in the case of one-vs-one, $\frac{N (N - 1)}{2}$ classifiers are trained for each pair of classes. It maps the data that is not linearly separable to higher dimensional space based on the non-linear kernel functions including radial basis function (RBF) and polynomial kernel. In this research study, we trained ⁶C₂ SVM models by varying kernel functions. The linear kernel is the simplest one and its recognition accuracy increases linearly with the increase in feature vector’s size at a cost of an increase in computation time. While RBF and polynomial kernel outperformed linear kernel for the problem under consideration as demonstrated in Table 3. Optimal results are achieved with the help of polynomial kernel (refer to (7)) for all the selected datasets as shown in Tables 3 –12. $k (y, z) = {(y^{T} . z + c)}^{d}$ (7)

Table 7

Statistical analysis for CK+ dataset based on hybrid descriptors and SVM polynomial kernel

Emotion category	TPR	TNR	PPV	FPR	FNR	ACC
An	1	1	1	0	0	1
Di	1	1	1	0	0	1
Fe	1	0.99	0.99	0.0018	0	0.998
Ha	0.99	1	1	0	0.0086	0.998
Sa	1	1	1	0	0	1
Su	1	1	1	0	0	1
Average recognition accuracy 99.8%

Table 8

Statistical analysis for JAFFE dataset based on hybrid feature descriptor and SVM polynomial kernel

Emotion category	TPR	TNR	PPV	FPR	FNR	ACC
An	1	1	1	0	0	1
Di	1	1	1	0	0	1
Fe	0.90	1	1	0	0.1	0.98
Ha	1	1	1	0	0	1
Sa	1	0.98	0.91	0.02	0	0.98
Su	1	1	1	0	0	1
Average recognition accuracy 98.2%

Table 9

Statistical analysis for the TFEID dataset based on hybrid descriptors and SVM polynomial kernel

Emotion category	TPR	TNR	PPV	FPR	FNR	ACC
An	0.57	1	1	0	0.428	0.93
Di	1	1	1	0	0	1
Fe	1	1	1	0	0	1
Ha	1	1	1	0	0	1
Sa	1	0.92	0.72	0.078	0	0.93
Su	1	1	1	0	0	1
Average recognition accuracy 93.5 %

Table 10

Statistical analysis for SFEW dataset based on hybrid descriptors and SVM polynomial kernel

Emotion category	TPR	TNR	PPV	FPR	FNR	ACC
An	0.64	0.85	0.566	0.15	0.35	0.80
Di	0.14	0.96	0.21	0.037	0.85	0.90
Fe	0.39	0.93	0.43	0.067	0.61	0.86
Ha	0.70	0.90	0.68	0.098	0.29	0.85
Sa	0.62	0.88	0.59	0.11	0.37	0.83
Su	0.43	0.92	0.48	0.074	0.56	0.86
Average recognition accuracy 56.0%

Table 11

Statistical analysis for FER-2013 dataset based on hybrid descriptors and SVM polynomial kernel

Emotion category	TPR	TNR	PPV	FPR	FNR	ACC
An	0.51	0.89	0.51	0.10	0.48	0.83
Di	0.51	0.99	0.75	0.0039	0.48	0.98
Fe	0.47	0.91	0.49	0.09	0.52	0.84
Ha	0.79	0.88	0.77	0.11	0.20	0.85
Sa	0.46	0.89	0.46	0.10	0.53	0.83
Su	0.726	0.95	0.71	0.053	0.27	091
Average recognition accuracy 63.0%

Table 12

Statistical analysis for RaF dataset based on hybrid feature descriptors and SVM polynomial kernel

Emotion category	TPR	TNR	PPV	FPR	FNR	ACC
An	0.61	0.97	0.61	0.028	0.388	0.94
Di	0.47	0.96	0.48	0.037	0.525	0.93
Fe	0.43	0.99	0.63	0.008	0.567	0.97
Ha	0.91	0.87	0.87	0.13	0.089	0.89
Sa	0.74	0.93	0.72	0.07	0.25	0.89
Su	0.67	0.97	0.78	0.03	0.32	0.93
Average recognition accuracy 78.1%

In (7), y and z are feature vectors extracted from dataset images, c is a constant to trade-off the effect of the lower order and higher-order terms; d shows kernel degree which is 3 in our case.

The hyper-parameters of SVM are gamma (kernel scale) and C (box-constraint). SVM classifier draws a decision boundary to separate the instances belonging to different classes. Trade-off analysis is required to be performed while adjusting the position of this boundary line. The parameter C is used to control this tradeoff. A too-small value (close to zero) of C causes under-fitting while too large cause over-fitting that decreases the generalization power of the classifier. Gamma parameter represents the distance to which the influence of individual training instance reaches. Its value is inversely proportional to the radius of the influence of those training samples which are selected as support vectors. Too small gamma can’t capture the shape or complexity of the model while too large gamma value only includes the support vector itself and unable to prevent the overfitting even by decreasing the C value. In the proposed model, optimal results are achieved for C = 1 and gamma = 1.

4 Experiments and discussions

In this section, we presented the detail of the datasets selected for the experimentation purpose, performance evaluation measures, model’s hyperparameters, and experimental results.

4.1 Detail of experimental datasets

We evaluated the performance of the proposed FERS model based on realistic datasets i.e. SFEW [52], FER-2013 [53], RaF-DB [54] and also based on those collected in lab constrained environment i.e. TFEID [55], JAFFE [56], and CK+ [57]. All the selected datasets are briefly reviewed below in the following subsections.

4.1.1 Static facial expressions in the wild (SFEW) dataset

This dataset is comprised of static facial expression images extracted from acted facial expressions in the wild (AFEW) dataset [58]. AFEW is a video-based 2D expression dataset collected from 37 movie clips for the transition from the constrained lab environment to the real world. Subjects belonging to various races, age groups, and genders are considered for the collection of more realistic images having varied head poses, occlusions, etc. SFEW dataset consists of 700 images belonging to six basic emotions. The testing set is not labeled; so for results comparison, we used a validation set consisting of 321 images of six basic emotions.

4.1.2 FER-2013 dataset

The most famous dataset for recognizing emotions is based on 48×48 dimensional images belonging to six basic and one neutral emotion. It was published in ICML-2013 representation learning challenges. We have performed experiments based on the training-testing division of the dataset as well as by 10-fold cross-validation. The recognition accuracy of 56% achieved by training-testing division and 63.0% achieved by cross-validation.

4.1.3 Real-world affective faces database (RAF-DB) dataset

It is one of the larger datasets comprising 15,339 aligned images belonging to six basic and one neutral expression. It is quite challenging having images belonging to different ethnicity, gender, and ages range. It includes images with different head poses and captured in a challenging environment having diverse illumination conditions and occlusions. We selected images belonging to six basic emotions and performed experimentation based on the 80-20 rule.

4.1.4 Taiwanese facial expression database (TFEID) dataset

It comprised 7200 expression images of 20 males showing eight emotions including six basic, neutral, and contempt emotions. Two CCD cameras were used for capturing images at 0⁰ and 45⁰ viewing angles having slight and high-intensity values. For consistency with previous datasets, a total of 229 high-intensity grayscale images of six basic emotions are selected for experimentation purposes.

4.1.5 Extended cohn kanade (CK+) dataset

It is an extended form of the Cohn Kanade database. It comprises 593 image sequences collected by 123 subjects for the representation of seven emotions including six basic and neutral emotions. We selected images belonging to six basic emotions for classifier training and testing purposes.

4.1.6 Japanese female facial expression (JAFFE) dataset

It consists of a total of 213 images captured from 10 females’ facial expressions including six basic and one neutral expression. 60 Japanese subjects rated every single image belonging to a particular emotion; at the end, a highly voted emotion label was assigned to the image under consideration.

We divided the selected images of RaF, TFEID, CK+, and JAFFE datasets into training-testing sets according to the 80–20 rule as 80% used for classifier training and 20% apart from the training set used for the testing purposes. We used a validation set of SFEW for testing purposes which are 31% of the overall dataset size and 69% selected for classifier training purposes. The recognition accuracy of 39.9% and 56.1% achieved for SFEW and FER-2013 respectively by training-testing division and it’s improved by performing 10-fold cross-validation for both datasets to 56.0% and 63.0% respectively.

As in videos, facial expressions change from a neutral state occupying three expressive phases including onset, apex, and offset phase. Among the three phases, the apex phase shows the peak expression while the onset and offset phases are closer to the neutral state. So we selected apex phase images from the above-mentioned datasets for experimentation purposes.

4.2 Performance evaluation measures based on statistical analysis and model’s experimental parameters

We briefly discussed the statistical analysis measures and the model’s hyper-parameters in the following subsections.

The confusion matrix describes the overall performance of the model. It shows TP, TN, FP, and FN cases. The statistical analysis of the proposed model is performed based on this matrix. The statistical terminologies that we considered to assess the performance of the proposed model are:

True-positive (TP): Number of instances belonging to the positive class which correctly predicted as positive.

True-negative (TN): Number of instances belonging to the negative class which correctly predicted as negative.

False-positive (FP): Number of negative instances that are incorrectly classified as positive.

False-negative (FN): Number of positive instances that are incorrectly classified as negative.

The measures that we selected for statistical analysis are presented in Table 1. Recall shows the portion of the relevant images that are retrieved by the classifier. Precision shows the portion of retrieved images that are also relevant. Accuracy value represents the correct predictions either positive or negative of the classifier. Specificity shows the portion of the negative images that are correctly predicted. While FPR and FNR both show a portion of incorrect predictions. The values of all these measures vary from 0-1 where zero shows the smallest and one shows the highest possible value. The higher values of recall, precision, TNR, and accuracy are desirable while small values equal to zero or closer to zero for FPR and FNR are desirable.

4.2.1 Model’s hyper-parameters

The hyper-parameters of the proposed model are C and gamma for SVM as discussed in section 3.3.1; several neighbors and number of bins for LIOP descriptor as discussed in section 3.2.1; cell size, block size, and several histogram bins for HoG descriptor as explained in section 3.2.2. The optimal values of the hyper-parameters for the selected datasets are shown in Table 2.

4.3 Results and discussions

We examined the performance of the proposed model based on the statistical analysis measures presented in Table 1. The statistical analysis results of the selected datasets are presented in Tables 7 –12. The experiments are repeated several times by varying the following parameters.

The scaling of the extracted facial area is based on the various interpolation methods and DWT.

The feature vector’s dimension is based on HoG and LIOP feature descriptors.

Kernel functions for a one-vs-one multiclass SVM classifier.

We carefully selected these parameters after extensive experiments based on the pre-processed images. Image resizing is a major pre-processing step as the HoG descriptor’s size is dependent on the image’s dimension which varies from dataset to dataset and also from person to person for the same dataset. SFEW aligned faces are of 143×181 dimension, TFEID rimmed images are of 480×600, RAF-DB-aligned faces are of 100×100, FER-2013 images are 48×48, JAFFE faces have around 150×150, and CK+ images minimum face size is around 200×200 dimension. Thus all images can either be scaled up or scaled down to an average size to result in an equal size feature vector. Two commonly used image resizing techniques are interpolation and DWT. We repeated experiments based on nearest, bilinear, bicubic interpolation, and DWT as shown in Tables 3 –6. We show with the help of experimental results that optimal recognition accuracy is achieved by resizing images based on bicubic interpolation for all the selected datasets.

LIOP descriptor’s size is independent of the image dimension. It needs a square image having an odd side length that can be achieved by scaling all selected dataset images:

To an average size i.e. 129×129 dimension with the help of DWT and interpolation techniques.

Equivalent to either shorter or longer image-side based on the interpolation techniques.

In JAFFE and CK+ dataset, the facial part detected based on the viola-jones algorithm is a square image having either odd or even side length and we scaled them based on option (a). RAF-DB and FER-2013 comprised of square aligned faces of 100×100 and 48×48 dimensions. While SFEW and TFEID datasets are comprised of rectangular aligned faces that can be resized based on (a-b) options. We performed experiments for all mentioned resizing options and our results show that optimal accuracy achieved for all datasets based on images directly resized to 129×129 through bicubic interpolation. FER-2013 is dataset accuracy decreased by scaling up its 48×48 dimensional images to 129×129 which is more than double the original size. We converted FER-2013 images to 49×49 dimensional square images and an increase in recognition accuracy is noted. We also performed classification for complemented feature vectors ᵮ and achieved promising results for all datasets as shown in Tables 7 –12.

Experimental results show that emotions like surprise cause more facial deformation and can be easily recognized while emotions like sadness, fear results in a high false-positive rate because of similar deformations (refer to Fig. 8).

Fig. 8

Sample images of six emotions from left to right 1st column represent angry, 2nd disgust, 3rd fear, 4th happy, 5th sad and 6th shows surprise emotion from selected datasets.

The comparative analysis of the recognition accuracy, precision, and recall for the datasets collected in a lab-constrained and lab-unconstrained environment is shown in Figs. 10-11. The accuracy rate for lab-constrained datasets is much higher than lab-unconstrained datasets because these are captured in constant illumination conditions, head pose, etc.

Fig. 9

Bar diagram of the SFEW dataset performance based on the variable length of a hybrid feature vector.

Fig. 10

Bar diagram of average recognition accuracy, recall, and precision for the lab-constrained datasets.

Fig. 11

Bar diagram of average recognition accuracy, recall, and precision for the realistic datasets.

4.3.1 Performance evaluation for CK+ dataset

It is publically available and the most commonly used benchmark dataset. So initially, we selected it for experimentation purposes by varying kernel functions for image resizing option (a) to fine-tune the resizing techniques and SVM kernel functions. For option (a), we resized images directly to 129×129 as well as by first applying DWT to extract the LL band image part and then resized them to 129×129 with the help of interpolation techniques. Table 3 shows that bilinear and bicubic interpolation performs comparatively better than the nearest one, so later on, we performed further experiments for all dataset images based on bicubic and bilinear interpolation. We show that the SVM polynomial kernel outperforms linear and RBF kernel functions resulting in 99.7% for images directly resized to 129×129 and 4320-dimensional LIOP features as shown in Table 3. Table 7 shows that 99.8% accuracy is achieved based on a 544-dimensional hybrid feature descriptor.

4.3.2 Performance evaluation for JAFFE dataset

Initially, we performed experiments for LIOP features extracted by considering 4 and 6 neighbors resulting in 144 and 4320-dimensional feature vectors respectively. The recognition accuracy of 87.7% achieved for 4320-dimensional LIOP features for images directly resized to 129×129 dimension through bicubic interpolation (refer to Table 4). Experiments are then performed for hybrid features ᵮ^{(LIOPi +HoGi)}. The 544-dimensional hybrid features with SVM polynomial kernel resulted in optimal recognition accuracy of 98.2% as represented in Table 8.

4.3.3 Performance evaluation for TFEID dataset

Rimmed images having 480×600 dimensions (refer to Fig. 8) are considered for experimentation purposes. LIOP requires square images having odd side length, so experiments performed by resizing images equivalent to shorter side length, longer side length, and to an average size i.e. 129×129 (refer to Table 5). Images directly resized to 129×129 resulted in high recognition accuracy of 84.8% based on 4320-dimensional LIOP descriptor and SVM polynomial kernel. While recognition accuracy of 93.5% achieved based on 919-dimensional hybrid features (144-LIOP and 775-HoG feature vector dimension) as shown in Table 9.

4.3.4 Performance evaluation for SFEW dataset

We perform experiments base on 143×181 dimensional aligned facial images as shown in Fig. 8. Maximum accuracy achieved for images directly resized to 129×129 based on bicubic interpolation. The accuracy rate of 37.1% achieved for 144-dimensional LIOP descriptors extracted base on four neighbors as shown in Table 6. Then we repeat experiments by varying the size of the hybrid feature vector. We concatenated 144-dimensional LIOP descriptor with variable size HoG descriptor achieved by varying cell size and number of bins as shown in Fig. 7. The performance-based on varied feature vector length is shown in Fig. 9. The highest accuracy rate of 39.9% is achieved for 769 and 919-dimensional features with the same recall and different precision values i.e. 31.8, 34.4 respectively. Accuracy started to decrease after that because of overfitting caused by a high dimensional feature vector. We then performed 10-fold cross-validation to enhance the classifier’s generalization power which resulted in an accuracy rate of 56.0%. Statistical analysis based on 919-dimensional hybrid feature descriptors and SVM 10-fold cross-validation is presented in Table 10.

4.3.5 Performance evaluation for FER-2013 dataset

This dataset is comprised of 48×48 dimensional grayscale images (refer to Fig. 8). As the LIOP descriptor requires a square image with odd side length so we resized all images to 49×49 dimensions based on bicubic interpolation. Here images are quite smaller than other selected datasets because of which optimal results are also achieved for smaller descriptor size that is 423 (144-LIOP and 279-dimensional HoG descriptor). Initially, we performed experiments by dividing the dataset into a training-testing part and an optimal accuracy rate of 56.1% achieved with SVM polynomial kernel and penalty value of 0.4. As it’s clear from Fig. 8 that in most of the images, the major facial region is hidden that affects badly the classifier’s generalization ability. To optimize the classifier generalization ability we repeated the experiments for cross-validation. The highest accuracy rate of 63.0% is achieved for 10-fold cross-validation. The statistical analysis is shown in Table 11.

4.3.6 Performance evaluation for RAF-DB

We selected aligned facial images having 100×100 dimensions (refer to Fig. 8). All images are resized to 129×129 with the help of bicubic interpolation for experimentation purposes. We evaluated the performance based on both training-testing and cross-validation. The recognition accuracy of 78.1% achieved by separately training the classifier on the training set and testing on the dataset images apart from training. Statistical analysis is performed to measure the performance of the proposed (refer to Table 12).

4.4 Performance comparison

We compared the performance of the proposed model with state-of-the-art approaches that are discussed in the literature to recognize the emotions based on facial expressions. The comparison based on each dataset is shown in Tables 13 –18. In [42], the author extracted HoG features from the facial area detected by the CAMshift algorithm that was further classified by the SVM classifier. A 95% accuracy rate was reported for the CK+ dataset. While in [41], an improvement in accuracy rate (i.e. 98.8% for CK+) is reported because of HoG features extraction from the facial area detected by the viola-jones algorithm. In [44], the author combined HoG and U-LTP features extracted from frontal face images and classified them by the SVM classifier. They extracted HoG features based on 16×16 cell size resulting in a comparatively large size feature vector. While in our case, more compact features extracted from processed facial area detected by the viola-jones algorithm. We extracted HoG features by selecting cell size 24 (for TFEID, SFEW) and 32 (for CK+, JAFFE), and an improvement in recognition accuracy is achieved by concatenating them with LIOP features. Sadeghi et al. [59] convolved images with Gabor filters which are then coded based on maximum and minimum responses. Final feature vectors computed based on the histogram for 16 and 8 Gabor filters having a length of 240 and 56 respectively. Ten-fold cross-validation was performed for classification by the SVM classifier with a linear kernel. An approach [36] proposed for ER based on facial patches indicating salient features. LBP-based concatenated histogram features extracted from the selected salient patches that were further used as an input for the training and classification by SVM classifier. An approach [60] using meta probability codes based framework is proposed to recognize the facial expressions. Four feature extractors i.e. LBP, Gabor-wavelet, facial fiducial point, and Zernike moment are selected for images’ feature extraction while SVM, KNN, sparse representation-based classifier, and RBF neural network selected for classification purpose. Zeng et al. [38] initially detected 51 facial landmark points by AAM. A single feature vector generated by concatenation of HoG, LBP, and gray value-based features extracted from the selected local patches centered at each landmark. The dimensionality of the high dimensional single feature vector generated by concatenation of all three features is then reduced by PCA. Final feature vector is then used as an input to deep sparse auto-encoders having four hidden layers each with 100 hidden nodes. Local mean binary pattern (LMBP) that is an appearance based local descriptor is proposed in [61]. Extracted features further classified by template matching as well as least-square SVM.

We compared the proposed model with the deep learning-based approaches as well. Jain et al. [28] initially employed contrastive equalization for contrast enhancement. Pre-processed images are further classified by DCNN as having convolutional and deep residual layers. Sun et al. [62] proposed a method to identify the optimized active region from three regions including regions of the left eye, right eye, and mouth. Histogram equalization, spatial normalization, and rotation correction were performed on the expression images to get more suitable features. The final emotion label was then obtained by majority voting of three CNNs’ classification results. In [63], the author combined two smaller networks to minimize the overfitting caused by big networks for small size datasets. A CNN model developed by a combination of two models, a temporal appearance features based network and temporal geometric features based network. Both types of extracted features are then jointly fine-tuned for the generation of small dimensional feature space to be classified. An algorithm for recognizing facial expressions with the help of biologically plausible features is presented in [64]. Initially, homogeneity of the facial encoded contours checked by a self-organizing network. In the end, several classifiers including SON, KNN, and multi-layer perceptron applied for classification.

A multithreading cascade of SURF (McSURF) was implemented by configuring the area under the curve (AUC) for ER in [19]. Three components of the proposed work were SURF feature descriptors’ extraction from local patches, weak classifiers based on logistic regression combined with ROC curve for testing cascade convergence, and last one multi-threading cascade that was able to process multiple categories to boost the training process. Weak classifiers build for each local patch based on SURF descriptors and logistic regression and optimum patches selected during boosting iterations, which repeated until AUC converges, or a specified number of iterations reached.

It is evidenced in comparison Tables 13 –18 that our proposed model based on texture features is highly suitable to recognize the emotions with promising accuracy in both constrained as well as in unconstrained environments. We compared the results with FER frameworks developed with the help of both texture features as well as with deep learning-based approaches. It is shown that the proposed model outperforms several state-of-the-art FER approaches that demonstrate its superiority.

Table 13
Comparison of the average recognition accuracy of the CK+ dataset

Study Method Avg. accuracy (%)

Luo et al. [37] ICLTP + KNN and sparse representation 99.26

Jain et al. [28] DNN 93.24

Zeng et al. [38] HoG, LBP, gray value descriptors + DSAE 95.79

Sajjad et al. [44] HoG, U-LTP + SVM 99.68

Sun et al. [62] CNN 95.36

Chen et al. [19] McSURF 91.2

Carcagni et al. [41] HoG + SVM 98.8

Jung et al. [63] DTAGN 97.25

Happy et al. [36] LBP + SVM 94.09

Proposed model LIOP, HoG + SVM 99.8

Study	Method	Avg. accuracy (%)
Luo et al. [37]	ICLTP + KNN and sparse representation	99.26
Jain et al. [28]	DNN	93.24
Zeng et al. [38]	HoG, LBP, gray value descriptors + DSAE	95.79
Sajjad et al. [44]	HoG, U-LTP + SVM	99.68
Sun et al. [62]	CNN	95.36
Chen et al. [19]	McSURF	91.2
Carcagni et al. [41]	HoG + SVM	98.8
Jung et al. [63]	DTAGN	97.25
Happy et al. [36]	LBP + SVM	94.09
Proposed model	LIOP, HoG + SVM	99.8

Table 14

Comparison of the average recognition accuracy of the RaF-DB dataset

Study	Method	Avg. accuracy (%)
Li et al. [54]	Deep Locality-Preserving CNN + SVM	74.2
Liu et al. [65]	Boosting-POOF	73.19
Proposed model	LIOP, HoG + SVM	78.1

Table 15

Comparison of the average recognition accuracy of the JAFFE dataset

Study	Method	Avg. accuracy (%)
Luo et al. [37]	ICLTP + KNN and sparse representation	95.5
Jain et al. [28]	DNN	95.23
Sajjad et al. [44]	HoG, U-LTP + SVM	95.72
Sun et al. [62]	CNN	96.57
Happy et al. [36]	LBP + SVM	91.8
Farajzadeh et al. [60]	MPC + KNN, SVM, RBFNN, SRC	88.3
Proposed model	LIOP, HoG + SVM	98.2

Table 16

Comparison of the average recognition accuracy of the TFEID dataset

Study	Method	Avg. accuracy (%)
Goyani et al. [61]	LMBP + SVM	90.4
Farajzadeh et al. [60]	MPC + KNN, SVM, RBFNN, SRC	92.5
Gu et al. [64]	SON	85.45
Proposed model	LIOP, HoG + SVM	93.5

Table 17

Comparison of the average recognition accuracy of the SFEW dataset

Study	Method	Avg. accuracy (%)
Sadeghi et al. [59]	Gabor filters + SVM	36.92
Li et al. [54]	Deep Locality-Preserving CNN + SVM	51.05
Liu et al. [65]		46.10
Chen et al. [19]	McSURF	32.2
Liu et al. [66]	STM-ExpLet	31.73
Proposed model	LIOP, HoG + SVM	56.0

Table 18

Comparison of the average recognition accuracy of the FER-13 dataset

Study	Method	Avg. accuracy (%)
Liu et al. [67]	CNN (subnet3)	62.44
Proposed model	LIOP, HoG + SVM

4.5 Computational performance

The proposed model is developed on a DELL core i3 system with 4 GB RAM, operating system windows 8.1 (64-bit), and MATLAB R2018a (64-bit). Either empirical or theoretical analysis can be performed to analyze the computational performance of an algorithm. The empirical analysis is based on the specifications of computer systems used for experimentation purposes but is varies from time to time depending upon the computational load. Whereas the input size determines the required computational resources in the theoretical analysis [68]. We have performed a theoretical analysis to analyze the computational time required for recognizing the emotions based on 129×129 processed facial images having 16,641 pixels. Initially, we perform experiments for the LIOP descriptor and optimal recognition rate achieved based on quite a high dimensional LIOP descriptor having 4320 lengths. We then performed feature level fusion of LIOP and HoG descriptor that resulted in 423 (from FER-2013), 544 (from JAFFE, CK+ dataset), and 919 (from TFEID, SFEW dataset) dimensional compact and distinguishing feature vectors. As hybrid descriptors’ length is much smaller than 4320-dimensional LIOP descriptors with comparable recognition accuracy, thus computational resources are minimized to a great extent. We performed this analysis independently of the system used to execute the program code. The computation time in seconds required for feature extraction from an image of the CK+ dataset based on the mentioned system specifications is shown in Table 19.

Table 19

Comparison of computational cost (in seconds) required for the per image facial features extraction from CK+ dataset

Standalone LIOP descriptor	Standalone HoG descriptor	Hybrid descriptor
0.0181	0.0056	0.0169

The comparison of the computation time for classifier training based on whole datasets and per image testing time is shown in Table 20 for Ck+ and JAFFE datasets. The computation time is reduced to a great extent because of the robust selected descriptors. Conclusion and future work

Table 20

Running time in seconds for CK+ and JAFFE datasets

Study	Training time		Testing time
	Proposedmodel	[37]	Proposed model	[37]
CK+	30.66	48.44	0.34	0.45
JAFFE	3.93	38.20	0.31	0.43

In this research study, we proposed a model to recognize facial expressions based on facial texture information. The core idea behind the developed framework is to enhance the discrimination power of the texture feature descriptors. The major challenge in the development of ER systems nowadays is the change in the geometrical structure of facial area from person to person and also environmental conditions like illumination conditions etc. We extracted facial texture information being invariant to face geometrical structure. The information extracted with the help of descriptors that are also invariant to several image transformations like illumination conditions, face viewpoint, and image blurring. It is evidenced from the experimental results that our selected complementary features proved useful to recognize the emotions in both lab-constrained as well as in realistic datasets. Local intensity order and histogram of oriented gradients based complemented descriptors increased the inter-class distance simultaneously decreasing the intra-class distance to enhance the recognition accuracy that is reflected in our results. We have performed a supervised classification based on a one-vs-one SVM classifier with varying kernel functions. The experimental results show that polynomial kernel results in an optimal accuracy rate of 99.8%, 98.2%, 93.5%, 56.0%, 78.1%, 63.0% for Ck+, JAFFE, TFEID, SFEW, RAF-DB and FER-2013 dataset respectively. We have not hardcoded the constraints related to intensity values so the proposed model can be fine-tuned for distinct datasets captured in a varying environment. In the future, we aim to extend the proposed framework by combining it with a deep learning model. The idea can be implemented by concatenating the selected complementary features with deep learning features. The resulted feature vectors can then be classified either by a local learning or a deep learning model. Another possible extension is to apply any deep transfer learning approach like VGG-19 to fine-tune the pre-trained model on the selected realistic datasets to optimize its classification accuracy.

Contributions

All the authors contributed equally to this research work.

Footnotes

Acknowledgments

We would like to thank Patrick Lucey, Li-Fen Chen, et al., Abhinav Dhall et al., and Michael J. Lyons et al. for providing facial emotion datasets for experimental purposes.

References

Posner

, Russell

J.A.

and Peterson

B.S.

, The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology, Development and Psychopathology 17 (2005), 715–734.

Ekman

and Friesen

W.V.

, Constants across cultures in the face and emotion, Journal of Personality and Social Psychology 17 (1971), 124.

Ekman

, Are there basic emotions? (1992).

Ayvaz

, Gürüler

and Devrim

M.O.

, Use of facial emotion recognition in e-learning systems, Information Technologies and Learning Tools 60 (2017), 95–104.

Arunnehru

and Geetha

M.K.

, Automatic human emotion recognition in surveillance video, in Intelligent Techniques in Signal Processing for Multimedia Security, ed: Springer, (2017), 321–342.

Meudt

, Schmidt-Wack

, Honold

, Schüssel

, Weber

, Schwenker

, et al., Going further in affective computing: how emotion recognition can improve adaptive user interaction, in Toward Robotic Socially Believable Behaving Systems-Volume I, ed: Springer, (2016), 73–103.

Fieker

, Moritz

, Köther

and Jelinek

, Emotion recognition in depression: An investigation of performance and response confidence in adult female patients with depression, Psychiatry Research 242 (2016), 226–232.

Tanaka

, Adachi

, Ukita

, Ikeda

, Kazui

, Kudo

, et al., Detecting dementia through interactive computer avatars, IEEE Journal of Translational Engineering in Health and Medicine 5 (2017), 1–11.

Argaud

, Vérin

, Sauleau

and Grandjean

, Facial emotion recognition in Parkinson’s disease: A review and new hypotheses, Movement Disorders 33 (2018), 554–567.

10.

Liu

T.-L.

, Wang

P.-W.

, Yang

Y.-H.C.

, Hsiao

R.C.

, Su

Y.-Y.

, Shyi

G. C.-W.

, et al., Deficits in facial emotion recognition and implicit attitudes toward emotion among adolescents with high functioning autism spectrum disorder, Comprehensive Psychiatry 90 (2019), 7–13.

11.

Griffiths

, Jarrold

, Penton-Voak

I.S.

, Woods

A.T.

, Skinner

A.L.

and Munafò

M.R.

, Impaired recognition of basic emotions from facial expressions in young people with autism spectrum disorder: Assessing the importance of expression intensity, Journal of Autism and Developmental Disorders 1–11. (2017).

12.

Pantic

and Rothkrantz

L.J.M.

, Automatic analysis of facial expressions: The state of the art, IEEE Transactions on pattern analysis and Machine Intelligence 22 (2000), 1424–1445.

13.

Tian

Y.-I.

, Kanade

and Cohn

J.F.

, Recognizing action units for facial expression analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence 23 (2001), 97–115.

14.

Sharma

, Singh

and Gautam

, Automatic Facial Expression Recognition Using Combined Geometric Features, 3D Research 10 (2019), 14.

15.

Mao

Q.-r.

, Pan

X.-y.

, Zhan

Y.-z.

and Shen

X.-j.

, Using Kinect for real-time emotion recognition via facial expressions, Frontiers of Information Technology & Electronic Engineering 16 (2015), 272–282.

16.

Xiong

and De la Torre

, Supervised descent method and its applications to face alignment, in Proceedings of the IEEE conference on computer vision and pattern recognition (2013), 532–539.

17.

Kalsum

, Anwar

S.M.

, Majid

, Khan

and Ali

S. M.

, “Emotion recognition from facial expressions using hybrid feature descriptors,”, IET Image Processing 12 (2018), 1004–1012.

18.

Happy

and Routray

, Automatic facial expression recognition using features of salient facial patches, IEEE transactions on Affective Computing 6 (2015), 1–12.

19.

Chen

, Luo

, Takiguchi

and Ariki

, Multithreading cascade of SURF for facial expression recognition, EURASIP Journal on Image and Video Processing 2016 (2016), 37.

20.

Kherchaoui

and Houacine

, Facial expression identification using gradient local phase, Multimedia Tools and Applications (2019), 1–17.

21.

Chen

, Chen

, Chi

and Fu

, Facial expression recognition in video with multiple feature fusion, IEEE Transactions on Affective Computing 9 (2016), 38–50.

22.

, Zheng

, Yang

, Wang

, He

and Ren

, Facial Expression Recognition Using Fusion Features Based on Center-Symmetric Local Octonary Pattern, IEEE Access (2019).

23.

Mistry

, Zhang

, Neoh

S.C.

, Jiang

, Hossain

and Lafon

, Intelligent Appearance and shape based facial emotion recognition for a humanoid robot, in The 8th International Conference on Software Knowledge Information Management and Applications (SKIMA 2014), (2014), 1–8.

24.

Shbib

and Zhou

, “Facial expression analysis using active shape model, Int J Signal Process Image Process Pattern Recognit 8 (2015), 9–22.

25.

Lee

Y.-H.

, Han

, Kim

and Kim

C.-G.

, Robust emotion recognition algorithm for ambiguous facial expression using optimized AAM and k-NN, International Journal of Security and Its Applications 8 (2014), 203–212.

26.

Moldoveanu

R.A.

, Facial emotion expressions recognition with brain activites using kinect sensor V2, (2015).

27.

Alabbasi

H.A.

, Moldoveanu

and Moldoveanu

, Real time facial emotion recognition using kinect v2 sensor, IOSR J Comput Eng Ver II 17 (2015), 2278–2661.

28.

Jain

D.K.

, Shamsolmoali

and Sehdev

, Extended deep neural network for facial emotion recognition, Pattern Recognition Letters 120 (2019), 69–74.

29.

Ngo

Q.T.

and Yoon

, Facial Expression Recognition Based on Weighted-Cluster Loss and Deep Transfer Learning Using a Highly Imbalanced Dataset, Sensors 20 (2020), 2639.

30.

Rescigno

, Spezialetti

and Rossi

, Personalized models for facial emotion recognition through transfer learning, Multimedia Tools and Applications (2020), 1–18.

31.

Jaiswal

, Gianchandani

, Singh

, Kumar

and Kaur

, Classification of the COVID-19 infected patients using DenseNet201 based deep transfer learning, Journal of Biomolecular Structure and Dynamics (2020), 1–8.

32.

Das

N.N.

, Kumar

, Kaur

, Kumar

and Singh

, Automated deep transfer learning-based approach for detection of COVID-19 infection in chest X-rays, Irbm (2020).

33.

Gianchandani

, Jaiswal

, Singh

, Kumar

and Kaur

, Rapid COVID-19 diagnosis using ensemble deep transfer learning models from chest radiographic images, Journal of Ambient Intelligence and Humanized Computing (2020), 1–13.

34.

Whitehill

, Bartlett

M.S.

and Movellan

J.R.

, Automatic facial expression recognition, Social Emotions in Nature and Artifact 88 (2013).

35.

Mikolajczyk

and Schmid

, A performance evaluation of local descriptors, IEEE Transactions on Pattern Analysis and Machine Intelligence 27 (2005), 1615–1630.

36.

Happy

and Routray

, Automatic facial expression recognition using features of salient facial patches, IEEE Transactions on Affective Computing 6 (2014), 1–12.

37.

Luo

, Liu

X.-y.

, Zhang

, Chen

X.-f.

and Chen

, Facial expression recognition based on improved completed local ternary patterns, Optoelectronics Letters 15 (2019), 224–230.

38.

Zeng

, Zhang

, Song

, Liu

, Li

and Dobaie

A.M.

, Facial expression recognition via learning deep sparse autoencoders, Neurocomputing 273 (2018), 643–649.

39.

Viola

and Jones

, Rapid object detection using a boosted cascade of simple features, in Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001, (2001), I–I.

40.

Dalal

and Triggs

, Histograms of oriented gradients for human detection, in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), (2005), 886–893.

41.

Carcagnì

, Del Coco

, Leo

and Distante

, Facial expression recognition and histograms of oriented gradients: a comprehensive study, Springer Plus 4 (2015), 645.

42.

Donia

M.M.

, Youssif

A.A.

and Hashad

, Spontaneous facial expression recognition based on histogram of oriented gradients descriptor, Computer and Information Science 7(31) (2014).

43.

Carcagnì

, Coco

, Leo

and Distante

, Facial expression recognition and histograms of oriented gradients: a comprehensive study, Springer Plus 4(1) (2015).

44.

Sajjad

, Shah

, Jan

, Shah

S.I.

, Baik

S.W.

and Mehmood

, Facial appearance and texture feature-based robust facial expression recognition framework for sentiment knowledge discovery, Cluster Computing 21 (2018), 549–567.

45.

Wang

, Fan

and Wu

, Local intensity order pattern for feature description, in 2011 International Conference on Computer Vision (2011), 603–610.

46.

Jack

R.E.

and Schyns

P.G.

, Toward a social psychophysics of face communication, Annual Review of Psychology 68 (2017), 269–297.

47.

Darwin

, 1965. The expression of the emotions in man and animals, London, UK: John Marry (1872).

48.

Ekman

, Friesen

W.V.

and Hager

J.C.

, Facial action coding system (FACS), A technique for the Measurement of Facial Action. Consulting, Palo Alto 22 (1978).

49.

, Li

, Wang

, Zhang

, Xing

, Gao

, et al., Facial expressions recognition based on cognition and mapped binary patterns, IEEE Access 6 (2018), 18795–18803.

50.

Boser

B.E.

, Guyon

I.M.

and Vapnik

V.N.

, A training algorithm for optimal margin classifiers, in Proceedings of the fifth annual workshop on Computational Learning Theory (1992), 144–152.

51.

Mayoraz

and Alpaydin

, Support vector machines for multi-class classification,” in International Work-Conference on Artificial Neural Networks (1999), 833–842.

52.

Dhall

, Goecke

, Lucey

and Gedeon

, Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark, in 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) (2011), 2106–2112.

53.

Goodfellow

I.J.

, Erhan

, Carrier

P.L.

, Courville

, Mirza

, Hamner

, et al., Challenges in representation learning: A report on three machine learning contests, in International Conference on Neural Information Processing (2013), 117–124.

54.

, Deng

and Du

, Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild, in Proceedings of the IEEE conference on computer vision and pattern recognition (2017), 2852–2861.

55.

Chen

L.-F.

and Yen

Y.-S.

, Taiwanese facial expression image database, Brain Mapping Laboratory Institute of Brain Science National Yang-Ming University Taipei Taiwan (2007).

56.

Lyons

M.J.

, Akamatsu

, Kamachi

, Gyoba

and Budynek

, The Japanese female facial expression (JAFFE) database, in Proceedings of third international conference on automatic face and gesture recognition (1998), 14–16.

57.

Lucey

, Cohn

J.F.

, Kanade

, Saragih

, Ambadar

and Matthews

, The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression, in 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops (2010), 94–101.

58.

Dhall

, Goecke

, Lucey

and Gedeon

, Collecting large, richly annotated facial-expression databases from movies, IEEE Multimedia 19 (2012), 34–41.

59.

Sadeghi

and Raie

A.-A.

, Human vision inspired feature extraction for facial expression recognition, Multimedia Tools and Applications 78 (2019), 30335–30353.

60.

Farajzadeh

, Pan

and Wu

, Facial expression recognition based on meta probability codes, Pattern Analysis and Applications 17 (2014), 763–781.

61.

Goyani

M.M.

and Patel

, Recognition of facial expressions using local mean binary pattern, ELCVIA: electronic letters on computer vision and image analysis 16 (2017), 54–67.

62.

Sun

, Li

, Huang

Y.-M.

, Li

and Lu

, Facial expression recognition using optimized active regions, Human-Centric Computing and Information Sciences 8(33) (2018).

63.

Jung

, Lee

, Yim

, Park

and Kim

, Joint fine-tuning in deep neural networks for facial expression recognition, in Proceedings of the IEEE international conference on computer vision (2015), 2983–2991.

64.

, Venkatesh

and Xiang

, A novel application of self-organizing network for facial expression recognition from radial encoded contours, Soft Computing 14 (2010), 113–122.

65.

Liu

, Li

and Deng

, Boosting-POOF: boosting part based one vs one feature for facial expression recognition in the wild, in 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), (2017), 967–972.

66.

Liu

, Shan

, Wang

and Chen

, Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014), 1749–1756.

67.

Liu

, Zhang

and Pan

, Facial expression recognition with CNN ensemble, in 2016 international conference on cyberworlds (CW), (2016), 163–166.

68.

Kao

M.-Y.

, Encyclopedia of algorithms: Springer Science & Business Media, (2008).