ICRL: Using landmark ratios with cascade model for an accurate age estimation system using deep neural networks

Abstract

Over the past decade, the computer vision community has given increased attention to the development of age estimation systems. Several approaches to more accurate and robust facial age estimation have been introduced. Apparent age datasets are typically collected from uncontrolled environments, leading to a number of challenges. In this paper, a cascade model system, which we called the ‘Integrated Classification and Regression with Landmark Ratios (ICRL), is introduced. Our system uses a classification model in order to learn the age label distribution, then uses this knowledge as an auxiliary input to a regression model. ICRL is based on context facial information and label distribution analysis. Facial context information is introduced through the extraction of precise facial landmark ratios. Extracted landmark ratios allow the system to distinguish each age label. The ICRL system uses a classification model to train the CNN network to learn the in-between relation of age labels. ICRL sufficiently models the aging process in the form of ordered and continuous imagery. The ICRL system minimizes the number of parameters needed as well as overall computational costs whilst maintaining robust and accurate results. Despite its simplicity, our system has outperformed other state-of-the-art approaches when applied onto the MORPH II, CLAP2015, AFAD and UTKFace datasets. ICRL achieved an overall superior predictive performance, reaching 99.67% with MORPH II, 99.51% with AFAD, 96.52 with CLAP2015, and 96.28% with UTKFace.

Keywords

Age estimation ordinal regression facial context information age label distribution

1 Introduction

Age estimation is not a simple problem of classifying images. It invariably requires an accurate regression task as part of any age estimation system. Facial features extracted from an input image are essential to an accurate estimation process. Aging eventually leads to changes in one’s facial features in the form of stretches, wrinkles, and other changes. There are often differences between actual age and apparent age [3]. Actual age is the actual number of years a person had lived. Apparent age is an annotation based on facial features. The cornerstones of any age-estimation system involve computational learning techniques to develop the estimated age. The use of apparent age datasets presents challenges, as some annotators decide on apparent age solely based on the appearance of a person’s face in a given image [21]. Aging datasets also present similar challenges, as there are typically no complete age series for identities nor variance in race and metadata. In addition, samples may be of poor quality, having possibly been scanned or collected from the internet. The ununiform nature of the aging process further complicates the task of age estimation, as aging is influenced by lifestyle and genetics as well as the surrounding environment. The aging process can be studied as ordered labels beside the in-between relation of labels [17].

Deep Networks have emerged as powerful tools in machine learning. Of these networks, the computer vision community most comply employs the convolution neural network (CNN), due to its suitability to the task of facial recognition and age estimation. Instead of handcrafted features, CNN extracts key features from the input im-age data. Network architecture invariably affects system performance. There is a trade-off between increasing network depth and avoiding the problem of gradient vanishing. The deeper the network architecture, the better the feature utilization and the more efficacious the system performance. Residual neural networks, or ResNets [38] represent the current best solution to the problem of gradient vanishing. The proposed ICRL age estimation system is thus based on ResNet architecture.

Based on deep network privileges, we designed our ICRL system that would address both ordered and continuous properties of age labels. Label distribution learning is used to analyze face images and address the relation between chronological age and adjacent labels [18]. The ICRL system consists of three phases: extracting facial context information, learning label distribution, and estimating age using the local regressors. ICRL increases facial context information by extracting more deduced features through measuring the distance ratios of traditional facial landmarks. In addition, the proposed ICRL system studies the nature of the label distribution by introducing a classification CNN network. Then, this knowledge is transferred to a regression CNN network through which local regressors would more accurately estimate age. The contributions of this paper can be summarized as follows:

Label distribution learning enhances features utilization by detecting relations between image labels and adjacent labels.

ICRL improves network learning, as label distribution learning makes the predicted output depend on the output of the most correlated classifiers.

ICRL extracts more facial context information from face images by introducing different distance ratios drawn from facial landmarks. These ratios contribute to the ability of the system to distinguish different age labels which enhances estimation performance.

The experiments were conducted on four datasets using the ResNet-34, VGG-16 and ResNet-50 models, where ICRL outperformed other state-of-the-art systems.

For the purposes of this study, different evaluation measures were used. These included the Mean Absolute Error (MAE), the Root Mean Square Error (RMSE), the confusion matrix, and the Cumulative Score (CS). Experiments were conducted on four datasets: the MORPH II, CLAP2015, AFAD, and UTKFace. The remaining sections are divided as follows: section 2 discusses related work; section 3 introduces the ICRL system and its proposed architecture; section 4 details the experiments and analysis of results; and section 5 details our conclusions and recommendations for future work.

2 Related work

Age estimation technology has become an integral part of multiple types of applications, including age-based computer interaction systems, information retrieval and security systems. The estimation process has typically been divided into two tasks: features extraction and age prediction through the analysis of features extracted [17]. Age labels are treated as either classes or discrete exact values. Some recent approaches tend to use the classification concept to separate age labels. Other approaches consider the exact value of age labels using a regression network.

In [19], the ‘DEX system,’ based on convolution neural networks (CNN), was designed to estimate age without relying on facial landmarks. DEX reportedly detects the age group in order to narrow the range of regression decision. DEX was pretrained by an IMDB-WIKI dataset to achieve an MAE of 2.68 on MORPH II. The most commonly used approach in the estimation process was converting the ordinal regression problem to K binary classifiers, with K being the number of age labels [1 , 35]. The other approach was to integrate CNNs with other modules in a parallel or sequential manner [10 , 28].

In [27], a ‘CORAL-CNN system’ was proposed. A one-linear bias parameter was added to the output layer of a ResNet-34. The default value of the one-linear bias parameter was set to zero and was to be adjusted during the training and learning processes. Researchers based their CORAL-CNN on ordinal regression with K output classifiers where the estimation process depended on the ordinal information of labels regardless of the in-between relation of labels.

Using Res-Net 101, Xie et al. proposed a ‘DOEL’ age estimation system [8]. An ordinal ensemble was created in order to divide the estimation task into several partitions. There are a number of possible classification strategies (i.e., two groups, three groups, etc.). Researchers used a two-stage aggregation strategy in order to combine the achieved probability predictions of different age groups. This was done by transforming these predictions into value distributions – rather than combining the votes – to obtain the final decision.

In [6], a scattering transform was used to extract features and a support vector machine was used in the estimation process. This framework firstly classifies samples by gender using a binary-class SVM, then detects age class through the multi-class SVM with a linear kernel. This framework determines the exact age value from the age class using the Sequential Forward Selection (SFS) algorithm in the fusion stage. To compute the number of age groups, several possible scenarios, resulting from 252 systems, were employed. Despite these complex scenarios, the system achieved an MAE of 2.7 years with the MORPH II dataset as had previous studies. In the same manner, BridgeNet [9] was used to take the extracted facial features from CNN and predict the output based on local regressors. In addition, the BridgeNet reportedly transfers these extracted features to a bridge-tree to produce weights to the local regressors’ result. This model presented is complicated and seemingly increases network parameters without a noticed enhancement of performance.

The C3AE system [2] was modeled upon ordinal regression using a compact plain model. C3AE was designed to avoid the depth-wise convolution in order to reduce the number of parameters needed whilst using a standard convolution. The C3AE system neglects the relation between the neighbor labels which affects the estimation decision.

CR-MT [16] employs a classification and a regression model to estimate age using a multiple-task network. The classification model is independent of the regression model where there is no error propagating from classification to regression. The adjacent information of labels is represented in the final regression output. DLDL-v2 [1] uses Gaussian distribution with a constant standard deviation to represent the label distribution. Results of age label distribution varies according to samples, so the estimation system does not require a constant standard deviation.

In [32], the authors proposed a new ranking method called ‘Soft-ranking’. Soft-ranking was designed to combine the LDL [29] and ordinal ranking methods [35]. For each sample, soft-ranking produces a vector that specifies the probability if the input age exceeding the specified classifier rank. Soft-ranking employs Gauss error function to encoding the age label. Gauss error function misrepresents the label distribution, as it produces the same probability vector to different age labels. They achieved an MAE of 2.7 years with the MORPH II dataset. A summary of recent approaches is shown in Table 1.

Table 1
Summary of recent age estimation approaches

Approach name Stand for Based on Mean absolute error

DEX [19] Deep EXpectation VGG-16 + Softmax 2.68 on MORPH II

OR-CNN [35] Ordinal Ranking-CNN ResNet-34 + Softmax 3.27 on MORPH II and 3.65 on AFAD

CORAL-CNN [27] COnsistent RAnk Logits-CNN ResNet-34 + Sigmoid 2.64 on MORPH II

DAG-CNNs [22] Directed Acyclic Graph Convolutional Neural Networks Combines VGG-16 and GoogLeNet architectures 2.81 on MORPH II

C3AE [2] Compact Cascade Context-based Age Estimation model CNN with a multiple-scale regression model 2.75 years on MORPH II and 2.95 years on FGNET

ScatNet [6] Scattering Network Scattering transform+support vector machine 3.92 years on FGNET and 2.7 years on MORPH II

BridgeNet [26] – ResNet-34 + BridgeNet 2.3 years on MORPH II and 2.68 years on FGNET

DOEL [8] Deep and Ordinal Ensemble Learning Res-Net 101 2.7 on MORPH II and 2.53 on FGNET, 5.69 on AgeDB

CR-MT [16] Classification and regression multi-task learning Alexnet 2.3 on MORPH II, 5.6 on Webface and 4.87 on CACD datasets.

Soft-ranking [32] – Soft-ranking with ResNet-34 2.7 on MORPH II, 5.58 on AgeDB

DLDL-v2 [1] Deep label distribution learning –version 2 Gaussian distribution with TinyAgeNet and ThinAgeNet models 2.291, 1.969 on MORPH II by TinyAgeNet and ThinAgeNet models respectively

Approach name	Stand for	Based on	Mean absolute error
DEX [19]	Deep EXpectation	VGG-16 + Softmax	2.68 on MORPH II
OR-CNN [35]	Ordinal Ranking-CNN	ResNet-34 + Softmax	3.27 on MORPH II and 3.65 on AFAD
CORAL-CNN [27]	COnsistent RAnk Logits-CNN	ResNet-34 + Sigmoid	2.64 on MORPH II
DAG-CNNs [22]	Directed Acyclic Graph Convolutional Neural Networks	Combines VGG-16 and GoogLeNet architectures	2.81 on MORPH II
C3AE [2]	Compact Cascade Context-based Age Estimation model	CNN with a multiple-scale regression model	2.75 years on MORPH II and 2.95 years on FGNET
ScatNet [6]	Scattering Network	Scattering transform+support vector machine	3.92 years on FGNET and 2.7 years on MORPH II
BridgeNet [26]	–	ResNet-34 + BridgeNet	2.3 years on MORPH II and 2.68 years on FGNET
DOEL [8]	Deep and Ordinal Ensemble Learning	Res-Net 101	2.7 on MORPH II and 2.53 on FGNET, 5.69 on AgeDB
CR-MT [16]	Classification and regression multi-task learning	Alexnet	2.3 on MORPH II, 5.6 on Webface and 4.87 on CACD datasets.
Soft-ranking [32]	–	Soft-ranking with ResNet-34	2.7 on MORPH II, 5.58 on AgeDB
DLDL-v2 [1]	Deep label distribution learning –version 2	Gaussian distribution with TinyAgeNet and ThinAgeNet models	2.291, 1.969 on MORPH II by TinyAgeNet and ThinAgeNet models respectively

Currently, a number of age estimation systems depend on ordinal information between age labels [2 , 35], while others depend on the correlation between adjacent labels using label distribution learning [1 , 18]. The ICRL system studies ordinal information and in-between relations between age labels. The system employs context facial information and label distribution analysis. ICRL system has been able to extract more facial context information by measuring distance ratios of traditional facial landmarks. Learning label distribution helps in the detection of the relation between the image label and adjacent labels as it distributes the neighborhood degree of the age label. In other words, the label distribution technique is used to determine the importance of an age label in relation to the target label. Knowledge of the label distribution is transferred to an ordinal regression CNN network in order to estimate age on the bases of degrees of neighborhood importance.

3 The proposed ICRL system

In this section, the proposed system of Integrated Classification and Regression with Landmarks ratio (ICRL) is discussed. The ICRL system consists of three phases: extracting facial context information, learning the label distribution, and estimating age using local regressors. ICRL system studies the nature of label distribution by introducing a classification CNN network. Then, this knowledge is transferred to a regression CNN network in order for local regressors to make the most accurate age estimation possible. The architecture of the ICRL age estimation system is shown in Fig. 1.

Fig. 1

The model architecture of the ICRL age estimation system.

3.1 Extracting facial context information

Accurate age estimation greatly depends on the precision feature extraction from face images. Using the proposed system, we focused on the effects of aging on facial features. After detecting the main five facial landmarks, which are the left eye, right eye, nose, mouth and forehead ‘regions,’ our system computes the distance ratios between each center point of facial landmarks. The most important ratios are then used as input for the CNN network, which learn them as added features. Through the learning algorithm, these ratios assist in differentiating each age label from the others. These ratios apparently increased the accuracy of classification decisions, as will be shown in the results section.

To achieve its goals, the system firstly extracts facial landmarks from the input image (left eye, right eye, nose, mouth and forehead). The regions of these landmarks are detected as outlined in Fig. 2b. The distance between each pair of the center points is calculated using the Euclidean distance equation [7], detailed in Equation (1).

Fig. 2

From the left a) the five regions of the facial landmarks are shown with green rectangles and the midpoint between the two eyes are drawn with blue circle, b) the distance between the center points is determined as the added features (best showed in color).

$D_{i} = \sqrt{{(x_{2} - x_{1})}^{2} + {(y_{2} - y_{1})}^{2}}$ (1) where (x1, y1) and (x2, y2) is a pair of center points. Table 2 defines the ten important distances that were drawn from seven main points of the face image: (left_eye_center, right_eye_center, mouth_center, nose_center, forehead_center, p), where p is the middle point between the left eye and the right eye.

Table 2

Definitions of the ten important distances

Distance label	Two points used in computation
D1	left_eye_center, right_eye_center
D2	left_eye_center, nose_center
D3	left_eye_center, mouth_center
D4	right_eye_center, nose_center
D5	right_eye_center, nose_center
D6	mouth_center, forehead _center
D7	nose_center, forehead _center
D8	p, mouth_center
D9	p, nose_center
D10	mouth_center, nose_center

These ten distances produce 90 possible ratios by dividing each distance by the others. The Recursive Feature Elimination algorithm [31] was used to select the most important ratios that affected the predicted output. Through experimentation, the first seven ratios detected are determined to be the most affected features. Increasing the number of ratios to 30 ratios improved performance, albeit with a small effect. Using all 90 ratios did not improve performance, but rather introduced additional computational overhead. Therefore, thirty of these ratios were deemed better than using only seven ratios in terms of accuracy and are used in the rest of this work.

For each image, these thirty ratios are computed and entered to the CNN network as input along with the face image. A comparison between the use of seven ratios and thirty ratios was conducted (for details, see section 4.3.3).

3.2 Learning the label distribution

In our proposed system, label distribution learning treats age labels as discrete values, allowing for the correlation between labels to be learnt according to input facial features. We used label distribution to distribute the importance probability among the age labels according to neighborhood degrees. ICRL system uses a ResNet-34 network to learn the label distribution of the training dataset. The thirty ratios are added to the features extracted from the input image as shown in Fig. 1. The ground truths of label distributions are then computed as the inverse of the cumulative distribution function (CDF) [12] that is closer to the binary representation of ordinal ranking in comparison to the CDF function, as shown in Fig. 3. The cumulative distribution function (CDF) is given by Equation (2).

Fig. 3

The output of the inverse of CDF function (1-CDF) is closer to the binary representation of ordinal ranking than the CDF function (best showed in color).

$CDF (x) = \frac{1}{2} (1 + \frac{2}{\sqrt{Π}} o^{\int_{e}^{x} - t^{2} dt}) .$ (2)

The importance vector of the age labels is defined by Equation (3). ${imp_v ec}_{i} = \sum_{k = 0}^{k - 1} 1 (1 - CDF (\frac{k - L_{i}}{\bar{δ} * \sqrt{2}}))$ (3) where imp_vec_i is the ground truth label distribution for the ith image, k is the classifier rank as k ∈ (0, 1, . . . , K - 1), L_i is the ground truth age label of the ith image and $\bar{δ}$ is a parameter to define the relation between adjacent labels. The value of imp_vec_i ranges from 0 to 1 and is defined by Equation (4). ${imp_v ec}_{i} = {\begin{matrix} ⩾ 0.5, if k ⩽ L_{i} \\ < 0.5, if k > L_{i} \end{matrix}$ (4) where imp_vec equals 0.5 only if the classifier rank k is equal to the target age label L_i. Each classifier checks if the input image label (L_i) exceeds its rank or not. Output > 0.5 indicates that classifier rank is less than the target label. The predicted imp_vec is computed by the fully connected layer as per Equation (5). $im {\hat{p_vec}}_{i} = S (w_{\dot{i}}^{J} * O_{\dot{i}} + b_{i}^{J}),$ (5) where S is a sigmoid function $S (x) = \frac{1}{1 + e^{- x}}$ , w_i and b_i are the fully connected layer parameters (weights and biases, respectively), $O_{\dot{i}}$ is the output of the previous layer. The loss function is used to compute the cost and adjust the parameters of the network. The loss function is adopted to use the sigmoid cross-entropy according to Equation (6). $\begin{matrix} cos t_{1} = - \frac{1}{T} * \sum_{i = 1}^{T} \sum_{k = 0}^{k - 1} [log (im \hat{p_v {ec}_{i}^{k}}) * \\ imp_v {ec}_{i}^{k} + log (1 - im \hat{p_v {ec}_{i}^{k}}) * (1 - imp_v {ec}_{i}^{k})] \end{matrix}$ (6) where $im \hat{p_v {ec}_{i}^{k}}$ is the predicted importance vector label for input image ith with classifier rank kth and $imp_v {ec}_{i}^{k}$ is the desired encoding label.

One of the merits of the proposed ICLR system is that the label distribution learning does not produce constant mean and standard deviation values for all labels, as in [1], but it varies according to input image features. This helps to improve the detection of each class characteristics especially with the use of extracted ratios.

3.3 Estimating age using the local regressors

In the proposed system, knowledge of the label distribution is transferred to an ordinal regression CNN network in order to estimate an accurate age through the use of local regressors. ResNet-34 with K outputs is used for the regression task, with K representing age label numbers. The ordinal regression model then transforms the regression task into K-1 binary classification problems, as was proposed in [35]. As in Fig. 1, the input of the network is the face image with its ratios.

The model is pretrained using the label distribution task then finetuned with the regression task. For an input sample (X_i, L_i), where X_i is the face image and L_i is the input label, the ground truth ranking label of classifier with rank k is determined by Equation (7). $B_{i}^{k} = {\begin{matrix} 1, if k < L_{i} \\ 0, if k ⩾ L_{i} \end{matrix}$ (7)

Each classifier checks if the input label exceeds its rank (k) or not. A sigmoid function (S) was chosen for local regressor function as follows in Equation (8). $\overset{´}{ρ} = S (z_{i})$ (8) where z_i is the output of the fully connected layer. The final estimated label ${\hat{L}}_{i}$ is determined by computing the number of classifiers whose output (z_k) exceeds the value 0.5 as in Equation (9): ${\hat{L}}_{i} = \sum_{k = 0}^{k - 1} I {z_{k} > 0.5})$ (9) where the indicator function $I {.}$ equals 1 if $(z_{\dot{k}}^{J} > 0 . 5)$ is true and 0 otherwise. As a result of learning transfer, the cost function considers the importance probability of age classes. Therefore, loss is affected by near-neighborhood classes of the target label. The sigmoid cross-entropy loss is used as the cost function, defined by Equation (10). $\begin{matrix} cos t_{2} = - \frac{1}{T} * \sum_{i = 1}^{T} \sum_{k = 0}^{k - 1} [log (S (z_{i}^{k})) * B_{i}^{k} \\ + log (S (1 - z_{i}^{k})) * (1 - B_{i}^{k})] \end{matrix}$ (10) where T is the number of the training images, $B_{i}^{k}$ is the ground truth binary label, and $z_{i}^{k}$ is the output of fully connected layer at rank k for the image ith. In the proposed system, the label distribution learning considers neighborhood relations between labels where ordinal ranking information is used by the regression task. Thus, the ICRL system sufficiently models both ordering and continuity properties of the aging process.

4 Experiments and discussions

In this subsection, a brief discussion of evaluation datasets is presented. The training experiments are explained in detail. Comprehensive comparisons between the proposed system and other recently proposed systems are discussed. In addition, the evaluation metrics used to evaluate the results are stated. Furthermore, an analysis of each phase of the proposed system, including complexity analysis, is presented.

4.1 Datasets

The proposed ICRL system is tested on four datasets that can be defined as follows:

MORPH II is a large aging dataset that includes 13,617 identities [9]. There are 55,134 samples of superior quality. There is variance in brightness and background. It may be considered a representative dataset as different genders and races are included. There is a sufficient number of images for each identity to represent an age series. Randomly Split (RS) [5] and five-fold subject-exclusive (SE) [26] splitting are alternatingly used for the purposes of dataset splitting. Using RS, 80% of the dataset is randomly allocated to the training phase where the remaining 20% of the dataset is allocated to the testing phase. The SE protocol is more complicated than RS as it divides the dataset into 80–20 subsets with no overlapping. As in [5, 32], we choose the MORPH II dataset to be subjected to splitting by both protocols.

CLAP2015, which stands for ‘Chalearn LAP 2015’ [20], encompasses 4699 images along with their apparent ages. Some annotators evidently had voted for age according to the general appearance of the face in each given image. The mean and the standard deviation of the annotators’ votes have been provided. Samples vary in resolution requiring an additional step of resizing and aligning images. There are variations in illumination and background. This dataset is divided into three subsets: training, validation and testing. We combined training and validation subsets to obtain large training set as proposed in [18 , 32].

AFAD is larger than MORPH II, but the images have an inferior quality [35]. The dataset includes about 160 thousand face images that had been collected from different settings and environments. There is variance in brightness, facial pose and background. An additional challenge is the variability in facial expressions. Aligned and cropped faces are provided as ‘helpful’ versions. The dataset includes a wide range of ages (15–75). The images are mainly of Asian students who had shared their images through school social networking sites. There are two groups of images for each age set: one for females and the other for males. As in [27, 35], the SE protocol was used with the AFAD dataset.

UTKFace is smaller than AFAD as it contains only about 20,000 images [36], however, it contains wider range of ages (0–116). In addition, aligned and cropped faces are provided as a ‘helpful’ version. Despite that, UTKFace is not a representative dataset as there are no complete age series for the identities. Due to the uncontrolled environments, facial age appearances are not accurate, and the quality is poor. Gender and race are provided next to the age of each sample. As in [27, 35], the SE protocol was used with the UTKFace dataset.

4.2 Evaluation metrics

In age estimation systems, the most commonly employed evaluation metrics are the Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) metrics [5 , 35]. MAE is the absolute average difference between the actual age and the predicted output, where RMSE is the square root of the mean square between the actual age and the predicted one. Equations (11) and (12) define MAE and RMSE, respectively [5]: $MAE = \frac{1}{T} \sum_{J = 1}^{T} | L_{j} - \hat{L_{j}} |$ (11) $RMSE = \sqrt{\frac{1}{T} \sum_{J = 1}^{T} {(L_{j} - \hat{L_{j}})}^{2}}$ (12) where T is the total number of testing images, L_j and $\hat{L_{j}}$ are the predicted age and the ground truth label, respectively [21]. In addition, the cumulative score (CS) is typically used to calculate the proportion of test images that have mean absolute errors below a threshold error E as defined in Equation (13). $CS (E) = \frac{t^{E}}{T} * 100 %$ (13) where t^E is the number of testing images that have MAEs below a threshold error E and T is the total number of testing images.

In apparent age datasets, images are labeled by several annotators, who based their labels on the general appearances of faces in the images [20, 21]. The mean and the standard deviations of the annotators’ votes are typically provided. Thus, the normal score (ɛ) is better than MAE in evaluating the apparent age dataset. It is calculated by Equation (14): $ɛ = \frac{1}{T} \sum_{J = 1}^{T} [1 - e^{- (\frac{{(\hat{L_{j}} - μ_{j})}^{2}}{2 o \cdot_{\dot{j}}^{2}})}],$ (14) where j refers to the tested image, μ_j refers to the mean label of the image j and σ_j refers to standard deviation. The value of ɛ score ranges from 0 to 1.

4.3 Experiments

Experiments were run on four datasets of images with different environments and characteristics. MORPH II, AFAD and UTKFace are a real age dataset, whilst CLAP2015 is an age apparent dataset. In the following, the preprocessing step and more technical details are detailed. In addition, the training phase and the interpretation of results are presented.

4.3.1 Experimental setup

In the system initialization, we firstly prepared each dataset by aligning the face in the image. The Mlxtend package was used to trace facial landmarks. Each image was aligned according to the average facial landmarks of all images. After that, we aligned noses to the center and cropped each face image. Each image was resized to 200*240 pixels. In CLAP2015, the images have different resolutions, so we firstly cropped faces then resized all images to a standard size (1024, 1024). The irrelevant images without full faces or clear poses were removed. Examples of the preprocessing of face images are shown in Fig. 4.

Fig. 4

The predicted age of some face images that are evaluated by ICRL system. (A) MORPH II (B) CLAP2015 (C) AFAD and (D) UTKFace. A black rectangle outlines the low prediction images.

The ICRL system uses one ResNet-34 model that is trained for 400 epochs: 200 epochs with label distribution cost functions and 200 epochs with regression task cost functions. Because of the absence of individual age series, our network was pre-trained on MS-Celeb-1M [34] and IMDB-WIKI [19] datasets. Using the learning transfer technique, we fine-tuned the pre-trained model. Different data augmentation methods were randomly used during the training phase. Scale-up, rotate, center crop, random grayscale and color normalization were randomly chosen to produce different patterns of the training images in each epoch in order to enrich the learning process.

Network transform was used to resize all images to 128*128 then a 120*120 random cropping was conducted. The experiments were run on NVIDIA Titan X GPU for the large datasets and NVIDIA GeForce GTX 1660 Ti for the small datasets. The code was written in Python from scratch and based on PyTorch 1.7.1 [38].

Each dataset was trained for 400 epochs with batch sizes of 256. Adam was used for the purposes of model optimization with momentum set to 0.9, weight decay set to 0.0002, and the learning rate initialized to 0.0001 with a 10% decreasing rate every 30 epochs. As [32], $\bar{δ}$ was set to 4.

4.3.2 Results and discussion

In this part, we show the effectiveness of our ICRL age estimation system, which, as aforementioned, is based on the context facial information and label distribution analysis. In Table 3, a complexity analysis of ICRL system is detailed, with CLAP2015 as the example dataset. We divided the code into four parts: face alignment, extracting the facial context information, training, and testing. Facial alignment was conducted using one loop. Facial landmarks were detected, followed by cropping of face regions. Following this step, noses were aligned in the middle. Finally, images were resized to a standard size (1024, 1024), as images had vastly different resolutions. The face alignment process was O(N). Total time elapsed equals the sum of the time for training, validation and testing sets.

Table 3
Complexity analysis of the ICRL system on CLAP2015

The Algorithm O() Consumed Memory (MB) Elapsed time (minute)

Face alignment O(n) 5 42.7

Facial context information O(n) 5 42.13

Training (minutes) O(n²) 341.1284 129.38

Testing O(n) 341.1298 0.71

The Algorithm	O()	Consumed Memory (MB)	Elapsed time (minute)
Face alignment	O(n)	5	42.7
Facial context information	O(n)	5	42.13
Training (minutes)	O(n²)	341.1284	129.38
Testing	O(n)	341.1298	0.71

The facial context information was extracted and saved in a CSV file with O(N) program. Facial landmarks were used to compute the thirty facial ratios of each image, needing a single round in the whole dataset. This is the extra part of the ICRL system which affects system performance. The time elapsed to extract the facial context information equaled 42.13 minutes with 5 MB memory, so the added computational cost is nothing to mention.

The training phase consists of 400 epochs: 200 epochs for label distribution and 200 epochs for the regression task. Two nested loops were used. At this stage, the program was O(n²). Finally, system evaluation was conducted on the testing set by computing MAE between the actual age and the predicted one. At that point, the testing script consisted of one loop, and the program was O(n). The added computational cost of using thirty ratios is marginable in comparison to the level of enhanced performance achieved.

In Table 4, comparisons with the state-of-the-art systems in relation to the four datasets are provided. Experiments were repeated several times using ResNet-34, VGG-16 and ResNet-50 models. As shown in Table 4, system implementation using ResNet-34 with thirty ratios gave the best performance followed by system implementation using VGG-16. The increased number of distance ratios enhanced performance only slightly.

Table 4

Comparison to show MAE of different implementations of ICRL system using SE protocol

Age system	MORPH II	CLAP2015	AFAD	UTKFace
ICRL (with ResNet-50)	2.72	0.2768	3.78	5.64
ICRL (with VGG-16)	2.67	0.2747	3.65	5.76
ICRL (with ResNet-34)	2.64	0.2553	3.46	5.29
ICRL (ResNet-34 with seven ratios)	2.58	0.2468	3.44	5.14
ICRL (ResNet-34 with thirty ratios)	2.57	0.2453	3.42	5.13

Table 5 shows the results of testing the proposed ICRL system as well as other recently proposed system on MORPH II using an SE protocol. The DEX system is based on a CNN of VGG-16 architecture for the classification task, followed by a Softmax function. The DEX system treats age estimation as a regression problem. There are very large gradients which result in unstable predictions. The OR-CNN system transforms the regression task with k labels into a simple k-1 classification problem. OR-CNN is based on ordinal ranking of information be-tween labels. ICRL achieved the best performance of 2.57 MAE, as it functions by combining the capabilities of ordinal ranking of information and neighborhood relations between labels. The CORAL-CNN system uses the ordinal regression without introducing the in-between relation of labels.

Table 5

Comparison on MORPH II between ICRL system and state-of-the-art systems using SE protocol

System name	MAE
OH-Rank [11]	6.07
DEX [19]	2.68
CNN+ELM [14]	3.44
OR-CNN [35]	3.27
M-LSDML [4]	2.89
Mean-variance loss [5]	2.79
CORAL-CNN [27]	2.64
C3AE [2]	2.75
RAGN [15]	2.61
BridgeNet [26]	2.63
AGEn [37]	2.93
Soft encoding [32]	2.78
ICLR	2.57

In Table 6, the ICRL system is shown to have outperformed other state-of-the-art systems on MORPH II using an RS protocol with 1.62 MAE. DLDL-v2 uses the Gaussian distribution with a constant standard deviation to represent label distribution. However, age label distribution varies from different samples, and it is not essential that the system follows a constant standard deviation. Our ICRL system addresses both ordered and continuous properties of age labels. Label distribution learning is used to analyze face images and address the relation between chronological age and adjacent labels. In ICRL, label distribution probabilities vary according to input image features.

Table 6

Comparison on MORPH II between ICRL system and state-of-the-art systems using RS protocol

System name	MAE
SAF [24]	2.97
Ranking-CNN [23]	2.96
MSFCL-KL [13]	2.73
SSR-Net [25]	2.52
C3AE [2]	2.75
Mean-variance loss [5]	2.16
TinyAgeNet [1]	2.291
ThinAgeNet [1]	1.969
LDL-oneHot [18]	1.905
Soft encoding [32]	1.74
ICLR	1.62

The confusion matrix of the ICRL system on MORPH II is shown in Figs. 5 6, comparing actual ages to predicted results. MORPH II is an unbalanced dataset as the last age labels are represented by small numbers of samples. The samples mostly concentrate in the first age labels, so ICRL system sufficiently estimate the first age labels with more than 150 match cases with the RS protocol and 100 with the SE protocol.

Fig. 5

ICRL Confusion matrix with SE protocol on MORPH II.

Fig. 6

ICRL Confusion matrix with RS protocol on MORPH II.

In Table 7, the experiments demonstrate that the label distribution and the added distance ratios had a significant effect on the performance. CLAP2015 is a more complicated dataset as some annotators voted for age on the bases of the general appearance of faces in the given images.

Table 7

Comparison on CLAP2015 between ICRL system and state-of-the-art systems using RS protocol

Age system	Pretrained model	ɛ-error	No of network
Human	–	0.34	–
SEU NJU [33]	FGNET,MORPH,et al.	0.3057	6
DEX [19]	IMDB-WIKI	0.265	20
ICT-VIPL [30]	MORPH, CACD, et al.	0.2706	8
AGEn [37]	IMDB-WIKI	0.2635	8
BridgeNet [26]	IMDB-WIKI	0.2551	1
DLDL-v2 [1]	MS-Celeb-1M	0.272	1
CEN [18]	IMDB-WIKI	0.274	1
DOEL [8]	IMDB-WIKI	0.247	1
ICLR(with thirty ratios)	MS-Celeb-1M, IMDB-WIKI, MORPH II	0.2453	1

ICRL enhances ɛ-error to 0.2453 as it enriched network training by the added features that distinguish between different age classes. Among the other state-of-the-art systems, DOEL’s [8] performance has been superior to the others. DOEL’s superior performance is depended on the additional samples added to the training set and an ensemble of multi-models with some distributed weights. In ICRL, a single CNN is used for the purposes of estimating apparent ages of images from a publicly available dataset.

As detailed by Table 8, the ICRL system performed better with thirty ratios than only seven ratios as the extra features evidently enriched network training. There are few systems that operate on AFAD and UTKFace. Face images are provided with an aligned and cropped version, and as a result, distance ratios are accurate and enhance the specific class features. OR-CNN and CORAL-CNN are based on the ordinal ranking information between labels. The CORAL-CNN system uses an ordinal regression with output layer of k-1 classifiers. We believe that ICRL achieves the best performance by considering the relative relation between adjacent class labels. The ICRL system’s functionality is based on context facial information and label distribution analysis.

Table 8

Comparison on AFAD, and UTKFace using SE protocol

Age system	AFAD	UTKFace
OR-CNN [35]	3.65	5.76
CORAL-CNN [27]	3.48	5.39
ICRL (with seven ratios)	3.44	5.14
ICRL (with thirty ratios)	3.42	5.13

In Fig. 7, statistical analysis is detailed. Cumulative Score (CS) curves were used to define predictive performance under different threshold values. Areas under the curve (AUC) were computed as per the composite Simpson’s rule. From the curves, ICRL appears to have achieved the best results, with the AUC equaling 13.83 on MORPH II, 13.08 on AFAD, and 11.44 on UTKFace. CORAL-CNN had the second-best results, with the AUC equaling 13.73 on MORPH II, 12.96 on AFAD, and 11.22 on UTKFace. There are no public results of CS curves with CLAP2015. The plotted performance of CORAL-CNN, OR-CNN, and CORAL-CE were reported by [27]. The best CS were recorded with MORPH II with its superior quality images. The lowest CS was recoded with the UTKFace dataset.

Fig. 7

MORPH II, AFAD, and UTKFace CS curves of ICRL system.

In Fig. 7, a statistical analysis is demonstrated using Cumulative Score (CS) curves that defines the predictive performance under different threshold values. The area under the curve (AUC) is computed by the composite Simpson’s rule. From the curves, ICRL achieves the best result of large AUC equals 13.83 on MORPH II, 13.08 on AFAD, and 11.44 on UTKFace. CORAL-CNN is the second-best results of AUC equals 13.73 on MORPH II, 12.96 on AFAD, and 11.22 on UTKFace. There are no public results of CS curves on CLAP2015. The plotted performance of CORAL-CNN, OR-CNN, and CORAL-CE were reported in the published paper [27]. Morph II has the best CS as it has good quality images where UTKFace has the worst CS.

5 Conclusion

In this paper, an age estimation system (ICRL) was proposed, based on pretrained ordinal regression by label distribution learning, is proposed. ICRL sufficiently models the aging process in the form of ordered and continuous imagery. Firstly, ICRL learns label distributions in order to demonstrate relations between neighboring labels. Then, ICRL uses the ordinal ranking of labels to generate the final estimated decision. Label distribution learning enhances features utilization by detecting the relation between the image label and the adjacent labels as it distributes the neighborhood degree of the age label among the other labels. ICRL improves network learning by adding some computed features, which are the distance ratios, drawn from facial landmarks. These ratios contribute to distinguishing between the different age class labels which enhances estimation performance. The experiments showed that the ICRL system improved estimation performance on four facial age datasets: MORPH II, CLAP2015, AFAD, and UTKFace. ICRL achieved an MAE of 1.62 on MORPH II, 3.42 on AFAD, and 5.12 on UTKFace. ICRL enhances ɛ-error to 0.2453 on CLAP2015 as it enriched network training by the added features that distinguish the different age classes. The predictive performance of ICRL analyzed using confusion matrices and the CS curves which showed the effectiveness of our system. ICRL achieved a best estimation performance, with an AUC of 13.83 on MORPH II, 13.08 on AFAD, and 11.44 on UTKFace. In future, we tend to apply ICRL system to more complicated datasets with different facial expressions and poses. In addition, ICRL can be extended by combing multiple models.

Competing interests

No competing financial interests or personal relationships exist that can influence this research paper.

References

Gao

B.B.

, Zhou

H.Y.

, Wu

and Geng

, Age estimation using expectation of label distribution learning, In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 2018, 712–718. doi: 10.24963/ijcai.2018/99.

Zhang

, Liu

, Xu

and Zhu

, C3ae: Exploring the limits of compact model for age estimation, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, 12587–12596. DOI: 10.1109/CVPR.2019.01287.

Agustsson

, Timofte

, Escalera

, Baro

, Guyon

and Rothe

, Apparent and Real Age Estimation in Still Images with Deep Residual Regressors on Appa-Real Database, 12th IEEE International Conference on Automatic Face & Gesture Recognition, 2017, 87–94. doi: 10.1109/FG.2017.20.

Liu

, Lu

, Feng

and Zhou

, Label-sensitive deep metric learning for facial age estimation, IEEE Transactions on Information Forensics and Security 13(2) (2017), 292–305. DOI: 10.1109/TIFS.2017.2746062.

Pan

, Han

, Shan

and Chen

, Mean-Variance Loss for Deep Age Estimation from a Face, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, 5285–5294. DOI: 10.1109/CVPR.2018.00554.

Yang

H.-F.

, Lin

B.-Y.

, Chang

K.-Y.

and Chen

C.-S.

, Joint estimation of age and expression by combining scatteringand convolutional networks, ACM Transactions on Multimedia Computing, Communications, and Applications(TOMM) 14(1) (2018), 1–18. https://doi.org/10.1145/3152118

and Lu

B.-L.

, An adaptive image Euclidean distance, Pattern Recognition 42 (2009), 349–357. https://doi.org/10.1016/j.patcog.2008.07.017

Xie

J.-C.

and Pun

C.-M.

, Deep and ordinal ensemble learning for human age estimation from facial images, IEEE Transactions on Information Forensics and Security 15 (2374), 2361–2374. https://doi.org/10.1109/TIFS.2020.2965298

Ricanek

and Tesafaye

, Morph: A longitudinal image database of normal adult age-progression, In Proceedings of IEEE 7th International Conference on Automatic Face Gesture Recognition, Southampton, UK, 2006, 341–345. DOI: 10.1109/FGR.2006.78.

10.

Liu

K.-H.

and Liu

T.-J.

, A structure-based human facial age estimation framework under a constrainedcondition, , IEEE Transactions on Image Processing 28(10) (2019), 5187–5200. DOI: 10.1109/TIP.2019.2916768.

11.

Chang

K.-Y.

, Chen

C.-S.

and Hung

Y.-P.

, Ordinal hyperplanes ranker with cost sensitivities for age estimation, Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, USA, 2011, 585–592. doi: 10.1109/CVPR.2011.5995437.

12.

Jäntschi

, A test detecting the outliers for continuous distributions based on the cumulative distributionfunction of the data being tested, Symmetry (Basel) 11(6) MDPI (2019), p. 835. https://doi.org/10.3390/sym11060835.

13.

Xia

, Zhang

, Weng

, Xu

and Liu

, Multi-Stage Feature Constraints Learning for Age Estimation, in IEEE Transactions on Information Forensics and Security 15 (2020), 2417–2428. doi: 10.1109/TIFS.2020.2969552.

14.

Duan

, Li

, Yang

and Li

, A hybrid deep learning CNN–ELM for age and gender classification, Neurocomputing 275 (2018), 448–461. https://doi.org/10.1016/j.neucom.2017.08.062

15.

Duan

, Li

and Li

, An ensemble CNN2ELM for age estimation, IEEE Transactions on Information Forensicsand Security 13(3) (2017), 758–772. doi: 10.1109/TIFS.2017.2766583.

16.

Liu

, Zhang

and Duan

, Facial Age Estimation Using a Multi-Task Network Combining Classification and Regression, , IEEE Access 8 (2020), 92441–92451. doi: 10.1109/ACCESS.2020.2994322.

17.

Osman

O.F.

and Yap

M.H.

, Computational intelligence in automatic face age estimation: A survey, IEEE Transactions on Emerging Topics in Computational Intelligence 3(3) (2018), 271–285. DOI: 10.1109/TETCI.2016.2646278.

18.

, Hu

, He

and Sun

, A coupled evolutionary network for age estimation, arXiv preprint arXiv:1809.07447, 2018, 1809–07447.

19.

Rothe

, Timofte

and Van Gool

, Deep expectation of real and apparent age from a single image withoutfacial landmarks, International Journal of Computer Vision 126(2) (2018), 144–157. https://doi.org/10.1007/s11263-016-0940-3

20.

Escalera

, Fabian

, Pardo

, Baro

, Gonzalez

, Escalante

H.J.

, Misevic

, Steiner

, Guyon

, et al., Chalearn looking at people 2015: Apparent age and cultural event recognition datasets and results, In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 2015, 1–9. doi: 10.1109/ICCVW.2015.40.

21.

Escalera

, Baró

, Escalante

H.J.

and Guyon

, Chalearn looking at people: A review of events and resources, International Joint Conference on Neural Networks, Anchorage, AK, USA, 2017, 1594–1601. DOI: 10.1109/IJCNN.2017.7966041.

22.

Taheri

and Toygar

Ö.

, On the use of DAG-CNN architecture for age estimation with multi-stage featuresfusion, Neurocomputing 329 (2019), 300–310. https://doi.org/10.1016/j.neucom.2018.10.071

23.

Chen

, Zhang

, Dong

, Le

and Rao

, Using ranking-cnn for age estimation, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, 5183–5192. DOI: 10.1109/CVPR.2017.86.

24.

Yang

and Ramanan

, Multi-scale recognition with DAG-CNNs, Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 2015, 1215–1223, doi: 10.1109/ICCV.2015.144.

25.

Yang

T.-Y.

, Huang

Y.-H.

, Lin

Y.-Y.

, Hsiu

P.-C.

and Chuang

Y.-Y.

, SSR-Net: A Compact Soft Stagewise RegressionNetwork for Age Estimation, In Proceedings of the Twenty-Seventh International Joint Conference onArtificial Intelligence 5(6), Stockholm, Sweden, 2018, 1078–1084. https://doi.org/10.24963/ijcai.2018/150

26.

, Lu

, Feng

, Xu

, Zhou

and Tian

, Bridgenet: A continuity-aware probabilistic network for age estimation, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, 1145–1154. DOI: 10.1109/CVPR.2019.00124.

27.

Cao

, Mirjalili

and Raschka

, Rank consistent ordinal regression for neural networks with application toage estimation, Pattern Recognition Letters 140 (2020), 325–331. https://doi.org/10.1016/j.patrec.2020.11.008.

28.

Wang

, Guo

and Kambhamettu

, Deeply-learned feature for age estimation, In Proceedings of IEEE Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA, 2015, 534–541. DOI: 10.1109/WACV.2015.77.

29.

Geng

, Yin

and Zhou

Z.-H.

, Facial age estimation by learning from label distributions, IEEETransactions on Pattern Analysis and Machine Intelligence 35(10) (2013), 2401–2412. DOI: 10.1109/TPAMI.2013.51.

30.

Liu

, Li

, Kan

, Zhang

, Wu

, Liu

, Han

, Shan

and Chen

, Agenet: Deeply learned regressor and classifier for robust apparent age estimation, In Proceedings of the IEEE International Conference on Computer Vision Workshop, Santiago, Chile, 2015, 16–24. DOI: 10.1109/ICCVW.2015.42.

31.

Chen

X.-W.

and Jeong

J.C.

, Enhanced recursive feature elimination, In Sixth International Conference on Machine Learning and Applications, Cincinnati, OH, USA, 2007, 429–435. DOI: 10.1109/ICMLA.2007.35.

32.

Zeng

, Huang

and Ding

, Soft-Ranking Label Encoding for Robust Facial Age Estimation,, IEEE Access 8 (2020), 134209–134218. doi: 10.1109/ACCESS.2020.3010815.

33.

Yang

, Gao

B.-B.

, Xing

, Huo

Z.-W.

, Wei

X.-S.

, Zhou

, Wu

and Geng

, Deep label distribution learning for apparent age estimation, In Proceedings of the IEEE International Conference on Computer Vision Workshop, Santiago, Chile, 2015, 102–108. DOI: 10.1109/ICCVW.2015.53.

34.

Guo

, Zhang

, Hu

, He

and Gao

, Ms-celeb-1m: A dataset and benchmark for large-scale face recognition, In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 2016, 87–102. DOI: 10.1007/978-3-319-46487-9_6.

35.

Niu

, Zhou

, Wang

, Gao

and Hua

, Ordinal regression with multiple output CNN for age estimation, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, 4920–4928. DOI: 10.1109/CVPR.2016.532.

36.

Zhang

, Song

and Qi

, Age progression/regression by conditional adversarial autoencoder, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, 5810–5818. DOI: 10.1109/CVPR.2017.463.

37.

Tan

, Wan

, Lei

, Zhi

, Guo

and Li

S.Z.

, Efficient group-n encoding and decoding for facial ageestimation, IEEE Transactions on Pattern Analysis and Machine Intelligence 40(11) (2017), 2610–2623. DOI: 10.1109/TPAMI.2017.2779808.

38.

, Shen

and Hengel

A.D.

, A.: Wider or Deeper: Revisiting theResNet Model for Visual Recognition, Pattern Recognit 90 (2019), 119–133. https://doi.org/https://doi.org/10.1016/j.patcog.2019.01.006.

ICRL: Using landmark ratios with cascade model for an accurate age estimation system using deep neural networks

Abstract

Keywords

1 Introduction

2 Related work

4.1 Datasets

4.2 Evaluation metrics

4.3.1 Experimental setup

Table 3 Complexity analysis of the ICRL system on CLAP2015 The Algorithm O() Consumed Memory (MB) Elapsed time (minute) Face alignment O(n) 5 42.7 Facial context information O(n) 5 42.13 Training (minutes) O(n2) 341.1284 129.38 Testing O(n) 341.1298 0.71

Competing interests

References

Table 3
Complexity analysis of the ICRL system on CLAP2015

The Algorithm O() Consumed Memory (MB) Elapsed time (minute)

Face alignment O(n) 5 42.7

Facial context information O(n) 5 42.13

Training (minutes) O(n²) 341.1284 129.38

Testing O(n) 341.1298 0.71