Analytical learning classifier based on predefined evenly-distributed class centroids

Abstract

For the pattern recognition, most classification models are solved iteratively, except for Linear LDA, KLDA and ELM etc. In this paper, a nonlinear classification network model based on predefined evenly-distributed class centroids (PEDCC) is proposed. Its analytical solution can be obtained and has good interpretability. Using the characteristics of maximizing the inter-class distance of PEDCC and derivative weighted minimum mean square error loss function to minimize the intra-class distance, we can not only realize the effective nonlinearity of the network, but also obtain the analytical solution of the network weight. Then, the sample is classified based on GDA. In order to further improve the performance of classification, PCA is used to reduces the dimensionality of the original sample, meanwhile, the CReLU activation function are adopted to enhances the expression ability of the features. The network transforms the samples into the higher dimensional feature space through the weighted minimum mean square error, so as to find a better separation hyperplane. In experiments, the feasibility of the network structure is verified from pure linear $\bm{W}$ , $\bm{W}+$ Tanh, and PCA $+\bm{W}+$ Tanh respectively on many small data sets and large data sets, and compared with SVM and ELM in terms of training speed and recognition rate. The results show that, in general, this model has advantages on small data set both in recognition accuracy and training speed, while it has advantages in training speed on large data sets. Finally, by introducing a multi-stage network structure based on the latent feature norm, the classifier network can further significantly improve the classification performance, the recognition rate of small data sets is effectively improved and much higher than that of existing methods, while the recognition rate of large data sets is similar to that of SVM.

Keywords

Pattern recognition image classification machine learning GDA

1. Introduction

The fundamental task of pattern recognition is to extract the features of the pattern samples, and then minimize the intra-class distance and maximize the inter-class distance in the feature space to achieve the best recognition performance. In order to achieve the above goals, experts and scholars in the field of pattern recognition and machine learning have conducted long-term research and proposed a large number of feature extraction and classification methods, from the Bayesian classifier based on the statistical pattern recognition theory to the linear classifier, support vector machines, radial basis function networks, multilayer feedforward networks, convolutional neural networks, and recurrent neural networks, etc., the recognition performance is greatly improved.

The basis of statistical pattern recognition [1] is Bayesian decision theory, that is, according to the Bayesian formula, the posterior probability of the sample is obtained through prior probability and class conditional probability density to classify the pattern. However, due to the difficulty in obtaining the class conditional probability density function, most classifier models belong to the Discriminative Model, which skips the solution of the class conditional probability density and directly calculates the mapping relationship between the input sample $\bm{x}$ and its posterior probability $p(w_{i}/\bm{x})$ , which can be expressed by $p(w_{i}/\bm{x})=f(\bm{x},\theta)$ , where $\theta$ is the parameters of the classification model $f$ . The parameters of the model are obtained by minimizing the experience risk during training. However, except for linear LDA, KLDA and ELM etc, most nonlinear classifiers usually can only obtain numerical solutions through iterative operations, which is not always to get the global optimal solution.

Based on PEDCC [2], this paper studies the classification problem from the latent feature space $\bm{h}$ of the input samples. The supervision signal of the sample is regarded as a point in the feature space, and each class corresponds to a class centroid is evenly distributed and solidified in the feature space, achieving the maximization of the class inter-class distance. In this way, the classification problem is transformed into the class conditional probability density estimation problem in the latent feature space, which is usually solved by the criterion function of minimizing the intra-class distance. Knowing the class conditional probability density, the posterior probability can be easily obtained. Because the training objectives in the feature space are solidified to reduce variable parameters in model training, it is possible for us to obtain analytical solutions under certain criteria even if the model is non-linear. Since this method obtains the class conditional probability density function, it belongs to the generation model.

Since PEDCC was proposed in the classification supervised autoencoder [2], it has been applied to the PEDCC-Loss loss function for the CNN classifier [3], clustering [4], semi-supervised learning [5], incremental learning [6] and active learning [7] and OOD Detection [8, 9] etc. Meanwhile, the analytical generation method and frame characteristic analysis of PEDCC have also been solved mathematically [10].

In this article, based on the characteristics of PEDCC, we propose a new type of simple analytical learning classifier, as shown in Fig. 1a, where PCA+Whitening+CReLU is used as a feature preprocessing method to improve classification performance. It extracts the principal feature components of the sample and reduces the dimensionality of high-dimensional data. The CReLU [11] and Tanh activation functions are used to enhance the nonlinear ability of the network. In the entire network, PCA can be solved analytically, meanwhile, the fully connected weight parameter $\bm{W}$ can also be solved analytically by the derivative weighted minimum mean square error criterion function, so it has a full analytical solution. If we do not consider the PCA+CReLU feature preprocessing unit, our network can be actually considered as an ordinary three-layer feedforward network using Tanh activation function, but the linear operation between the second and third layers and the corresponding linear discriminant criterion are replaced by GDA (Gaussian Discriminant Analysis). Experimental results show that the analytical learning classifier is feasible.

Figure 1.

(a) is a diagram of structure of analytical learning classifier, where PCA $+$ Whitening is used to reduce the dimensionality of the input data and select principal feature components to reduce the correlation between the features, and each dimension is normalized to reduce data redundancy. Meanwhile, the CReLU activation function can double the output dimension, which has the effect of non-linearity and feature enhancement. And the Tanh function can increase its non-linearity. (b) is a distribution diagram of various training samples and corresponding PEDCC in the three-dimensional space when the Iris data set is selected in 3 principal features and 3 latent dimensions. Left, middle, and right are the distribution of each type of samples, and star represents the corresponding PEDCC points.

Figure 1b is the training results of the Iris data set. The Iris data set includes 150 samples, divided into 3 categories, each with 50 samples and each sample includes 4 length attributes. The result in the figure is a distribution diagram of the sample feature vector $\bm{h}=(h_{1},h_{2},h_{3})^{T}$ after the analytical learning classifier, when the principal component is 3 and the feature dimension is 3. After extracting the latent feature $\bm{h}$ , since the calculation of the weight $\bm{W}$ is based on the minimum mean square error criterion, the class conditional probability density of each class in the feature space can be regarded as a Gaussian distribution, and the distribution of all samples is a Gaussian mixture model. With GDA, if the prior probability is assumed to be the same, the final class recognition result is: $J=\mathop{\textit{argmax}}\limits_{i}p_{i}$ , where $p_{i}$ is class conditional probability density function satisfying normal distribution.

This paper uses PEDCC and weighted linear regression function to construct a new and effective analytical learning classifier. The main contributions of the article are as follows:

Based on the derivative weighted minimum mean square error criterion function, we theoretically obtain the analytical expression of training weight parameter, avoiding the complexity of iterative solution, improving the training speed, interpretability of the network and the number of effective latent features.

PCA $+$ Whitening $+$ CReLU for feature preprocessing improves the nonlinearity of the network and reduces the amount of intermediate feature parameters. The network is upgraded to a higher-dimensional feature space by CReLU, which is conducive to finding better separation hyperplane to effectively improve the recognition performance.

The feasibility of the network design of the analytical learning classifier is proved experimentally. Compared with SVM and ELM, in general, this method has advantages on small dataset both in recognition accuracy and training speed, while it also has advantages in training speed on large data sets.

By analyzing the relationship between the norm of latent features and the recognition rate, a multi-stage network structure based on the latent feature norm is introduced, which can further significantly improve the classification performance.

This paper is mainly divided into 5 parts. The first part introduces the background of pattern classification and the contribution of the article, as well as the structure diagram of the analytical learning classifier based on PEDCC; the second part introduces the background and basic concepts of SVM, ELM, VDA, VFC and PEDCC; The third part is the network structure and theoretical derivation; the fourth part verifies the feasibility of the network under different data sets, showing the performance of analytical network compared with ELM and SVM,and verified the effective improvement of multi-layer analytical learning classifier; the fifth part is the generalization and summary of the analytical learning classifier.

2. Related work

The theoretical basis of traditional statistical pattern recognition is Bayesian decision theory. Typical models include Linear Classifiers [12, 13], Support Vector Machines [14, 15], Radial Basis Function Networks [16, 17], and Multilayer Feedforward Networks [18, 19], Extreme Learning Machine [20, 21], etc. Deep learning is also a statical learning method.

2.1 SVM and ELM in the form of a three-layer feedforward network

SVM classifier is the most important pattern classification method before the re-emergence of deep learning. Although SVM is short of recognition rate compared to deep learning methods, it is still widely used in many occasions because of its small sample learning ability and good interpretability. The traditional SVM is a two-class classification method. According to the characteristics of the data set, the hard interval maximization is used for the linearly separable data set, and the soft interval maximization is used for the linearly inseparable data set. The common point is to train a hyperplane to classify the data as much as possible. It mainly uses decision functions:

$\displaystyle y=\textit{sign}(\bm{w}^{T}\bm{x}+b)$ (1)

$\textit{sign}(\,)$ is a symbolic function, $\bm{w}$ is a $k$ -dimensional weight vector, $b$ is a scalar, $\bm{x}\in{R}^{n}$ is $k$ -dimensional sample vector. In contrast, for linearly inseparable data sets, the kernel function is used to map sample in the low-dimensional linear space to the high-dimensional nonlinear space to find the optimal classification interface. Their common goal is to find an optimal separating hyperplane to reduce the error rate of data classification as much as possible. Intuitively speaking, it is to keep the concentrated samples on the data set as far away as possible from the classification hyperplane.

From network perspective, SVM is a typical single-latent-layer three-layer feedforward network. The optimization algorithm actually obtains the weight between the latent layer and the output layer, and the parameters of the latent layer kernel function (support vectors can be considered as the parameters between the input layer and the latent layer). Similarly, the Extreme Learning Machine (ELM) [20, 21] studies the learning method of the three-layer feedforward network from another unique perspective, namely by randomizing the weight between the input layer and the latent layer to reduce the parameters that need to be trained, so the parameters between the latent layer and the output layer can be solved by the least square error method, and the analytical solution of the three-layer network is achieved. This method has fast network learning speed and good generalization performance, but usually requires a large number of latent layer nodes.

Compared with the extreme learning machine solidifying the weight between the input and the latent layer, our algorithm solidifies the weight between the latent layer and the output layer, and only the weights between the input layer and the latent layer are needed to determined. Since this solidification is only a training objective, so the parameters do not appear in the classifier. The training results are embodied in the mean and covariance of various class conditional probability densities, which can better classify and reduce the number of latent layer nodes.

2.2 FF-MLP multilayer feedforward network with analytical solution

In order to improve the interpretability and training efficiency of neural networks, Kuo et al. [22] proposed a mathematical model called RECOS (Rectified-Correlations on a Sphere), which answered the necessity of a nonlinear activation function and why two-layer cascade is better than the single layer. Furthermore, they [23] interpreted the multi-layer perceptron (MLP) as a generalization of a two-class LDA classifier, and proposed a 4-layer feedforward MLP (FF-MLP) model. The model does not need gradient back propagation, and the network can be trained through a single feedforward calculation, and obtaining good results on small data sets. However, the network parameters of this method require prior knowledge or a lot of attempts, and cannot be applied to complex data sets such as images, which limits its application.

2.3 VDA and VFC based on regular simplex vertices

VDA (Vertex Discriminant Analysis) [24, 25] is a discriminative analysis method based on the regular simplex vertices, which are equidistant points in space, and the number of vertices used as class indicators is $k+1$ . The loss function is defined as:

$\displaystyle R(a,b)=\frac{1}{n}\sum_{i=1}^{n}\|\bm{y}_{i}-\bm{Ax_{i}}-\bm{b}% \|_{\varepsilon}+\lambda\sum_{j=1}^{k}\|\bm{a}_{j}\|^{2}$ (2)

where $n$ is sample number, $k$ is output feature dimension; $\bm{y}_{i}$ is the class indicator (i.e. In fact, it is a regular simplex vertex. There are $k+1$ vertices corresponding to $k+1$ classes) corresponding to $i$ -th sample $\bm{x}_{i}$ with dimension $p$ ; $\bm{a}_{j}^{T}$ is the $j$ -th row of the $k*p$ matrix $\bm{A}$ , and $\bm{b}$ is the $k$ -dimensional intercept column vector. Thus, we can judge that the loss function is strictly convex, and the only minimum point can be found. Once the optimal $\bm{A}$ and $\bm{b}$ are determined, the sample can be assigned to the nearest class.

In order to minimize the loss function, the algorithm of VDA is to initialize $\bm{A}^{0}=0,\bm{b}^{0}=0$ , and generate class indicators $\bm{y}_{j}$ with construction method of simplex, and then by

$\displaystyle\bm{\gamma}_{i}^{(m)}=\bm{y}_{i}-\bm{A}^{(m)}\bm{x}_{i}-\bm{b}^{(% m)}$ (3)

where $m$ is the number of iterations to adjust the regular loss function. Meanwhile, $\bm{A}^{(m)}$ , $\bm{b}^{(m)}$ is determined by solving $k$ linear equations and iterated continuously. If $\|\bm{A}^{(m+1)}-\bm{A}^{(m)}\|<\gamma$ , $\|R(\bm{A}^{(m+1)}$ , $\bm{b}^{(m+1)})-R(\bm{A}^{(m)}$ , $\bm{b}^{(m)})\|<\gamma$ , $\gamma=10^{-4}$ is satisfied, the optimal $\bm{A}$ and $\bm{b}$ are determined.

VFC (Vertex Feature Classification) [26], which is also a classification method related to regular simplex, connects the vertices of the simplex with category to explore the geometric distribution in the high-dimensional feature space (simplex space) and uses multi-point positioning technology to classify unknown samples. Its basic idea is to project all its features into the simplex space, and then estimate the center of gravity, which is used as the coordinates of the sample features in the simplex space. Therefore, it can classify the samples into the category closest to the center of gravity based on the metric matrix.

The above two methods have certain recognition performance on small data sets, but both face the problems of high computational complexity of iterative solution, and the limitation of the output feature dimension.

2.4 Definition and characteristics of PEDCC

PEDCC was originally proposed in CSAE [4], which can simultaneously realize the functions of classification and autoencoders under a unified network architecture. PEDCC are some points evenly distributed on the hypersphere in the high-dimensional space. Figure 2 is a schematic diagram of the distribution of 2–4 PEDCC points in a three-dimensional space. Initially, PEDCC relied on the physical model of electric charges distributed on the hypersphere to generate iteratively. For pattern classification, we hope that the inter-class distance is as large as possible, which can be achieved by using PEDCC.

Figure 2.

The distribution diagram of evenly-distributed points for $k=$ 2, 3, 4 in a 3-dimensional space.

The difference between PEDCC and regular simplex is that in $n$ -dimensional space, simplex can only form $n+1$ equidistant distributed vertices, while PEDCC can generate any number of evenly distributed points, which provides more flexibility for pattern recognition. Recently, we use the regular polyhedron in the high-dimensional space and construct the evenly distributed points on the $n$ -dimensional hypersphere to mathematically generate PEDCC [10], and discuss the characteristics of the framework formed by PEDCC, which theoretically better solves some fundamental problems of PEDCC.

For the CNN classifier based on PEDCC-Loss [5], the convolutional layer plays the role of feature extraction, and the first fully connected layer maps these features to latent features, and all sample features are closely distributed in C (Number of categories) PEDCC points (class centroids) evenly distributed on the unit hypersphere. Since the linear layer can achieve arbitrary spatial rotation, these PEDCC points can be randomly generated, and as long as the distribution is evenly, the position does not affect the recognition result, but it must remain unchanged during training and testing. In addition, PEDCC is not only for classification, but also for feature extraction and uncertainty analysis.

By the solidification characteristics of PEDCC, this paper hopes to construct a new shallow neural network, which can be solved analytically, and has good interpretability and classification performance. After large comparison of trial, this paper proposes a new simple analytical learning classifier, as shown in Fig. 1a. Here PCA $+$ Whitening as a means of improving the classification performance is used to extract the principal feature components of the sample and reduce dimensionality of data. The CReLU and Tanh functions not only enhance the non-linear characteristics of the network to improve the recognition rate, but also helpful for the samples to be distributed in the high-dimensional latent feature space, rather than in the C (number of classes)-1 dimensional subspace expanded by PEDCC. The similarity between the network and the VDA is that they use equidistant points in the geometric space as the class centroids, but the difference is that the VDA can only uses C-1 dimensional latent feature. The network can improve the non-linearity and number of latent features through derivative weighted loss function and activation functions to improve recognition accuracy.

3. Theoretical analysis of the classification network model

3.1 Analytical solution of weight matrix W

As shown in Fig. 1a, the main problem of network training is how to obtain the weight matrix $\bm{W}$ . Assuming that the intermediate feature $\bm{y}$ is known (the original feature of the sample can be used directly, or it can be obtained through feature preprocessing, see section 3.2), In this way, we focus on the network solution of the last two layers, that is, the solution of $\bm{W}$ .

The nonlinear network with activation function is usually trained by means of BP algorithm which is backpropagating the gradient to the previous layer network to update the network weights. Here, since the training objectives of output $\bm{h}$ are PEDCC which is known, if there is no Tanh function layer, it is usually a problem of solving an overdetermined linear equation system, which can obtain the pseudo-inverse solution based on the minimum mean square error criterion, and the error obtained in this way theoretically conforms to the normal distribution, that is, the class conditional probability density of each class in the feature space $\bm{h}$ is Gaussian distribution.

Here, due to the non-linearity of the activation function, in order to achieve the training objectives at the output $\bm{h}$ , the objectives need to be back mapped to the intermediate feature $\bm{z}$ for training. If the activation function $\bm{h}=g(\bm{z})$ is reversible and derivable, we have an inverse function $\bm{z}=s(\bm{h})$ . For the training objectives $\bm{h}_{i}$ of sample $i$ (that is, the PEDCC vector corresponding to the sample label), we inversely map it back to obtain $\bm{z}_{i}$ in the $\bm{z}$ -space, and then perform training based on the minimum mean square error criterion function here.

However, because of the inverse mapping of the training objectives, our goal is to output the minimum mean square error at $\bm{h}$ , it is needed weight the mean square error criterion function in the $\bm{z}$ space. Because $d(\bm{h})=g^{\prime}(\bm{z})d\bm{z}$ , the weight value can be approximately the derivative of the activation function $g(\bm{z})$ at $\bm{z}_{i}$ . As shown in Fig. 1a, the training process of the weight parameter $\bm{W}$ is:

For the training objectives in the $\bm{h}$ space, we map the PEDCC points, which is the label category of each sample, back to the $\bm{z}$ space. This change is approximately inversely proportional to the derivative of the activation function when the PEDCC value is mapped back to $\bm{z}$ space, that is, the larger the derivative, the smaller the error of $\bm{z}$ that can be tolerated. Therefore, we obtain the following derivative-weighted minimum mean square error loss function:

$\displaystyle L=tr{[(\bm{WY}-\bm{Z})*\bm{G}][(\bm{WY}-\bm{Z})*\bm{G}]^{T}}=% \sum_{j}[(\bm{w}_{j}\bm{Y}-\bm{z}_{j})*\bm{g}_{j}][(\bm{w}_{j}\bm{Y}-\bm{z}_{j% })*\bm{g}_{j}]^{T}$ (4)

where * represents the dot product, $\bm{W}=[\bm{w}_{1},\bm{w}_{2},\ldots,\bm{w}_{D}]_{D\times M}^{T}$ , besides, $\bm{w}_{j}=[\bm{w}_{j1},\bm{w}_{j2},\ldots,\bm{w}_{jM}]$ is the $j$ -th row of $\bm{W}$ , $\bm{Y}=[\bm{y}_{1},\bm{y}_{2},\ldots,\bm{y}_{M}]_{M\times N}$ , $\bm{y}_{j}$ is the $j$ -th row vector; $D$ is the latent feature number, $N$ is the number of samples, and $M$ is the dimension of $\bm{y}_{j}$ which contains a dimension of all 1 to calculate the bias in term of the $\bm{W}$ weight. In fact, $\bm{y}_{j}$ dimension is $M-1$ . $\bm{Z}=[\bm{z}_{1},\bm{z}_{2},\ldots,\bm{z}_{D}]_{D\times N}^{T}$ , $\bm{z}_{j}=[\bm{z}_{j1},\bm{z}_{j2},\ldots,\bm{z}_{jN}]$ is the $j$ -th row of $\bm{Z}$ , that is, the value of the $i$ -th latent feature of the PEDCC vector (training objective) corresponding to the category labels of all samples is back mapped to the $\bm{z}$ -space. $\bm{G}=[\bm{g}_{1},\bm{g}_{2},\ldots,\bm{g}_{D}]_{D\times N}^{T}$ , $\bm{g}_{j}=[\bm{g}_{j1},\bm{g}_{j2},\ldots,\bm{g}_{jN}]$ is the $j$ -th row of the $\bm{G}$ matrix, and the derivative of the activation function corresponding to the $j$ -th latent feature when all samples arrive in the $\bm{z}$ -space. After partially differentiating:

$\displaystyle\frac{\partial L}{\partial\bm{w}_{j}}=[(\bm{w}_{j}\bm{Y}-\bm{z}_{% j})*\bm{g}_{j}]*\bm{g}_{j}\bm{Y}^{T}=\bm{w}_{j}\bm{B}_{j}-(\bm{z}_{j}*\bm{g}_{% j}*\bm{g}_{j})\bm{Y}^{T}=0$ (5)

where,

$\displaystyle\bm{B}_{j}=\left[\begin{array}[]{ccc}\bm{g}_{j}*\bm{y}_{1}(\bm{g}% _{j}*\bm{y}_{1})^{T}&\dots&\bm{g}_{j}*\bm{y}_{1}(\bm{g}_{j}*\bm{y}_{M})^{T}\\ \dots&\dots&\dots\\ \bm{g}_{j}*\bm{y}_{M}(\bm{g}_{j}*\bm{y}_{1})^{T}&\dots&\bm{g}_{j}*\bm{y}_{M}(% \bm{g}_{j}*\bm{y}_{M})^{T}\end{array}\right]_{M\times M}$ (6)

is a symmetric matrix. Then, we have

$\displaystyle\bm{w}_{j}=(\bm{z}_{j}*\bm{g}_{j}*\bm{g}_{j})\bm{Y}^{T}\bm{B}_{j}% ^{-1}$ (7)

The weight vector can be regarded as a pseudo-inverse solution weighted by the derivative. When calculating, $\bm{B}_{j}$ must be nonsingular, that is, full rank. In practical problems, sometimes it is singular, so it must add a small value, which is equivalent to L2 regularization constraints on the weights.

In the pure linear network without activation function Tanh, we know that the sample is projected to the $C-1$ dimensional space after the fully connected layer, and this is also the result obtained by VDA and VFC. So latent feature dimension and the rank of the weight matrix can only be $C-1$ . In this method, the dimension of latent feature $\bm{h}$ is determined by the dimension of PEDCC, which can be greater than the $C-1$ . Then, after taking the derivative of the activation function as the step, will it affect the rank of the fully connected layer $\bm{W}$ ?

In fact, if $\bm{B}_{j},(1\leqslant j\leqslant D)$ are the same, because $\bm{z}_{j}$ and $\bm{g}_{j}$ are one-to-one corresponding, $R(\bm{W})=C-1$ (consistent with the weight $\bm{W}$ of pure linear network), but the linear independence of $\bm{w}_{j},(1\leqslant j\leqslant D)$ are enhanced due to various $\bm{B}_{j}$ , that is, different $\bm{g}_{j}$ , so $R(\bm{W})>C-1$ .

Since we adopt the criterion of minimum mean square error, the probability density function distribution of various latent features $\bm{h}$ can be considered to be Gaussian normal distribution. The latent features $\bm{h}_{i}$ of samples for each class can obtain the corresponding mean and covariance matrix through training, so that the probability density function of each class can be expressed as $p_{i}=N(\bm{\mu}_{i},\bm{\sum}_{i})$ . By Gaussian Discrimination Analysis, if the prior probability is assumed to be the same, the final recognition result category is:

$\displaystyle J=\mathop{\textit{argmax}}\limits_{i}p_{i}$ (8)

The above method requires the inverse and the derivative function of the activation function, so the activation function must be reversible and derivable. ReLU fucntion, which has a zero derivative on the negative semi-axis, is not suitable for the method of this article because of the inability to reversely map $\bm{h}$ back to $\bm{z}$ , while LeakyRelu can. We use Tanh activation function in the experiment. If the activation function does not exist, that is, $\bm{z}=\bm{h}$ , then the above method degenerates into a linear classifier, that is, by using PEDCC, we get a method to find the analytical solution of a multi-class linear classifier. In addition, after the training, since the parameter of the class conditional probability density is determined by the training samples and has nothing to do with PEDCC, so PEDCC which only appears as training objective during training does not appear in the final classification.

Here, only PEDCC is feasible for the analytical learning classifier. Because of the particularity of the basic PEDCC structure (any basic PEDCC point has a element is 1), so the inverse of the activation function cannot be obtained. So $\bm{W}$ can not be trained. However, will the randomness of PEDCC affect the recognition result? In PEDCC-Loss [5], we have proved that it has no obvious effect on the recognition result for CNN classifier, which is also basically true here. Since the randomness of PEDCC corresponds to a rotation matrix, which can be absorbed into weight matrix $\bm{W}$ in training process.

3.2 Multi-stage analytical learning classifier

Figure 3.

The diagram of Multi-stage analytical learning classifier, where AL is PCA $+$ Crelu $+$ W $+$ Tanh. After AL, the features of the samples are judged, and the qualified samples with $f<1$ are switched to he next stage of classifier, otherwise, the result of current classifier is the final classification result.

Figure 4.

(a) the samples distribution in the two-dimensional space; (b) the samples distribution under the new axis after PCA; (c) the samples distribution after whitening; (d) the samples distribution after the CReLU function, the blue is separation hyperplane extended to the three-dimensional space.

Through the analysis of experimental data, we found that the recognition rate of samples with small latent feature norm was significantly low, which means that samples with different latent feature norm have different classification difficulties, and inspired us to build a multi-stage analytic learning network as shown in Fig. 3. For each stage of classifier, the samples with low latent feature norm are further input to the next stage of analytic learning classifier for better recognition performance, otherwise ending the recognition process.

For any input sample $x\in\omega_{i}$ , $\mu_{j}$ is the mean of sample features of class $\bm{j}$ , and $\bm{h}$ is the latent feature of sample $\bm{x}$ , we define a measure $f(\mathbf{h})$ as Eq. (9). In test phase of our multistage classification network, if for each mean there is $f(\mathbf{h})<1$ , the sample will be input to the next stage for further recognition; if there exists a mean that makes $f(\mathbf{h})\geqslant 1$ , or it’s the final stage, the recognition process ends, and its recognition result is final recognition result.

$\displaystyle f(\mathbf{h})=<\bm{\mu}_{\mathbf{j}},\mathbf{h}>\|\bm{\mu}_{% \mathbf{j}}\|^{-1}$ (9)

In the training phase, samples satisfying $f(\mathbf{h})<1$ will form a new subset $\bm{X_{s}}$ , whose number is about half of the previous stage dataset (the sample number of each class is also approximately half to ensure the balance of training samples). Subset $\bm{X_{s}}$ will be used to train the next stage of classifier. The whole network will be constructed to a predetermined number of stages, or the recognition rate cannot be improved.

3.3 Feature preprocessing based on PCA

PCA $+$ Whitening and CReLU function are added to the analytical learning classifier. PCA is mainly used to reduce the dimensionality of the input data and extract the principal components, so the correlation between the feature components after dimensionality reduction is reduced. The new principal components fall under the new coordinate system, which is beneficial to reduce the redundancy of data and the complexity of calculation. Figure 4a and b show that binary classification data fall on the new coordinate axis after PCA. The whitening can normalize each dimension of the data after dimensionality reduction shown in Fig. 4c, so that all features have the same variance. In addition, the CReLU function can increase the non-linear characteristics of the network and improve the overall recognition rate. The Fig. 4d describes binary data distribution in three-dimensional space after CReLU function. The feature space rises to three-dimension, so as to a hyperplane separating the binary samples can be found.

4. Experimental analysis of ALC

In order to verify the feasibility and rationality of the proposed analytic learning classifier, the following three cases of pure linear W, W $+$ Tanh, PCA $+$ W $+$ Tanh are compared and verified, at the same time, we compare it with SVM and ELM to test the performance of the analytical learning classifier.

4.1 Verification of the analytical learning classifier structure

4.1.1 Results of the experiment

In order to verify the rationality and effectiveness of the analytic learning classifier, Mnist, Fashion Mnist, Digits, Banknote Authentication, Pima, Diadates Disease, Iris, Breast Cancer Disease, Waveform, Wine and other data sets are used to carried out the comparative experiments in terms of recognition accuracy and training speed. The selection of the data set is based on two principles: typicality and low recognition rate of conventional recognition methods.

MNIST data set consists of 55,000 training sets and 10,000 test sets, with a total of 10 categories. Each picture is a 28*28 gray scale handwritten picture.

Fashion MNIST data set are made of 60,000 training samples and 10,000 test samples, with a total of 10 categories. Each picture is a 28*28 picture.

Digits data set has 1797 samples, which are digital labels of [0, 9]. There are 10 classes in total, and each picture has a pixel value of 8*8.

Breast Cancer Diagnostic data set has a total of 569 samples, a total of 2 categories, and each sample has a total of 30 attributes.

Banknote Authentication data set has a total of 1372 samples with 2 classes, and each sample has a total of 4 attributes.

PIMA Indian Diabetes data set has 768 samples, a total of 2 categories, each sample has a total of 8 attributes.

Waveform data set has 5000 samples, each sample has 21 attributes, and there are 3 categories in total.

Wine datasets has 178 samples, each sample has 13 features, and 3 classes in total.

Iris data set has 150 samples, 4 features, and 3 categories.

Here, our experiment is implemented using Pytorch on an Intel(R) Core(TM) i7-8700CPU, 8.0GB RAM, and Nvidia GeForce GT 730.

4.1.2 Analysis of experimental results

The experimental results are obtained by ten-fold cross-validation. Generally, the judgment of network performance is mainly based on training time and recognition rate. Table 1 mainly shows the recognition rate and training speed of each data set under linear, non-linear and analytical learning classifier. Compared non-linearity (training $\bm{W}$ through the weighted minimum mean square error) with the pure linear network (the effective feature can be only $C-1$ dimension), in addition to the Breast Cancer Diagnostic and Waveform data sets, other data sets have better recognition rates under non-linear networks, and the Breast and Waveform data sets are more suitable for linear classification methods. At the same time, due to the large number of parameters for the latent variables and weighted derivatives to solve the weight $\bm{W}$ on large data sets and Digits data sets, the training speed is slower under the $\bm{W}+$ Tanh network; the training time of other data sets is basically consistent. Therefore, training the network through the weighted minimum mean error can not only improve the nonlinearity of the network, but also has a better overall recognition performance.

Table 1
The recognition rate (Acc.) and training time (Tr.Ti.) of each data set under linear, non-linear and analytical learning classifier

	PCA $+\bm{W}+$ Tanh		$\bm{W}$ (class number-1)	$\bm{W}+$ Tanh
	PCA/D	Tr.Ti(s)/Acc. (%)	Tr.Ti.(s)/Accu. (%)	D	Tr. Ti(s)/Accu. (%)
Mnist	120/90	22.57/94.13	1.56/89.51	100	70.19/92.44
Fashion mnist	150/50	26.11/84.43	1.84/81.99	20	15.75/82.31
Digits	30/60	0.98/98.21	0.003/96.55	50	0.003/97.51
Banknote	4/3	0.015/99.64	0.001/98.25	2	0.002/98.8
Pima	8/2	0.016/76.55	0.001/75.63	2	0.003/75.84
Breast	20/2	0.024/96.25	0.001/97.36	3	0.002/97.01
Waveform	18/3	0.09/85.88	0.005/86.02	8	0.009/85.53
Wine	13/8	0.015/98.87	0.001/98.33	4	0.001/98.85
Iris	3/8	0.013/98.66	0.001/97.33	8	0.001/97.77

Furthermore, compared analytical learning classifier with nonlinear network, in addition to the Breast Cancer Diagnostic data set, the recognition rate of the network structure with PCA is better than the network without PCA, especially on large data (Mnist, FashionMnist). It shows that feature preprocessing can not only increase the nonlinearity of the network, but also promote it to find a better separation hyperplane in the higher-dimensional feature space to improve the recognition rate. Meanwhile, feature processing for multiclassification data sets can significantly reduce the number of parameters to reduce training time; on the contrary, feature pre-processing of small data sets will increase the number of parameters and slow down the training speed.

4.2 Some ablation experiments

4.2.1 Selection of activation function after PCA

Table 1 shows the feature preprocessing has an impact on network performance. But why CReLU is chosen as activation function in feature preprocessing? We also used ReLU and Tanh functions to compare the result on some data sets as shown in Table 2.

Table 2
The recognition performance of different activation functions in feature preprocessing

Datasets	PCA/D	Accuracy (%)
		Tanh	Relu	CReLU
Iris	3/6	97.33	90.66	98.66
Wine	13/8	96.07	92.03	98.87
Digits	30/60	97.04	93.54	98.21
Fashion mnist	150/50	83.34	79.36	84.43
Mnist	120/90	92.74	87.49	94.13

From Table 2, compared to ReLU and Tanh functions, the recognition with CReLU is the best. Based on the experiment, it is verified that CReLU can increase the nonlinearity of the network structure, so that the sample can find the separation hyperplane in the higher-dimensional feature space.

4.2.2 The influence of random PEDCC on performance

Will the randomness of PEDCC affect the performance of the network? Below we use Digits and Iris datasets to prove it.

Table 3
The influence of random PEDCC on performance under Digits and Iris data sets

Digits	Accuracy (%)	Iris	Accuracy (%)
Random PEDCC	97.88	Random PEDCC	98.66
(PCA/D $=$ 30/60)	98.21	(PCA/D $=$ 3/8)	98.00
	97.99		96.66
	97.99		98.66

Table 4

Network performance of different principal feature and latent features on the Digits and Waveform datasets

Digits		Waveform
PCA/D	Accuracy (%)	PCA/D	Accuracy (%)
30/70	96.91	21/3	85.10
30/60	98.21	18/3	85.88
30/40	97.99	15/3	85.34
40/60	97.83	21/5	85.13
40/80	98.05	18/5	84.96
50/80	97.77	15/5	85.20

Table 5

The influence of latent feature $D$ on the rank of $\bm{W}$ based on Mnist, Digits and Iris data sets

Datasets	Mnist			Digits			Iris
PCA	120	120	150	30	30	60	3	3	4
D	70	90	70	40	60	80	3	4	3
Rank ( $\bm{W}$ )	70	90	70	40	60	80	3	4	3

Table 6

The recognition rate (Acc.) and training time (Tr.Ti.) of PCA $+\bm{W}+$ Tanh, ELM, SVM and PCA $+$ SVM on nine data sets

	PCA $+\bm{W}+$ Tanh		ELM		SVM		PCA $+$ SVM
	PCA/D	Tr.Ti(s)/ Acc. (%)	D	Tr. Ti(s)/ Acc. (%)	Linear Tr.Ti(s)/ Acc. (%)	Rbf Tr.Ti(s)/ Acc. (%)	Linear Tr.Ti(s)/ Acc. (%)	Rbf Tr.Ti(s)/ Acc. (%)
Mnist	120/90	22.57/94.13	3500	223.311/96.29	177.012/93.93	171.362/97.86	181.674/94.53	175.375/98.15
Fashion mnist	150/50	26.11/84.43	3500	188.89/85.80	NA/NA	284.520/88.28	NA/NA	287.379/88.97
Digits	30/60	0.98/98.21	300	0.204/95.6	0.023/97.99	0.039/98.77	0.031/98.25	0.043/98.75
Banknote	4/3	0.015/99.61	30	0.011/99.92	0.006/98.83	0.005/99.63	0.006/99.13	0.005/99.63
Pima	8/2	0.016/76.55	300	0.136/66.22	2.821/76.53	0.011/75.88	2.931/77.01	0.011/76.53
Breast	20/2	0.024/96.25	300	0.111/91.21	0.551/95.43	0.003/91.74	0.124/92.23	0.003/92.33
Waveform	18/3	0.09/85.88	300	0.373/85.20	0.352/87.17	0.211/86.70	0.382/87.85	0.231/87.16
Wine	13/8	0.015/98.87	300	0.051/75.88	0.428/95.49	0.001/69.11	0.429/96.16	0.001/72.13
Iris	3/8	0.013/98.66	30	0.001/97.33	0.001/98.66	0.001/97.33	0.001/98.65	0.001/97.63

Although random PEDCC does not affect the recognition performance in a linear network, Table 3 shows the accuracy of the Digits and Iris data sets are slightly affected by random PEDCC on the analytical learning classifier. So once the nonlinearity of network is improved, the random PEDCC has an influence on the performance of the network. In fact, we need choose the best result from different random PEDCC.

4.2.3 The influence of the number of latent features

Besides the influence of random PEDCC on network performance, what is more important is the influence of the principal features and latent features. We take Digits and Waveform as examples to illustrate the influence of principal components and latent features on network performance.

It can be concluded from Table 4 that whether in terms of principal feature components or latent features, the overall trend of the network performance of Digits and Waveform is to become larger first and smaller later. The same trend exists for other datasets, so we can find the optimal principal features and latent features to achieve a better network performance.

4.2.4 The influence of the number of latent features on the rank of W

Previously, we objectively explained that taking the derivative of the activation function as the step into the weighted minimum mean square error can increase the rank of $\bm{W}$ . Next, we also verify it experimentally based on Mnist, Iris and Digits data sets.

In Table 5, based on Mnist, Digits and Iris data sets, we can find that different latent features $D$ and $R(\bm{W})$ are consistent and full rank, which enhances the effectiveness of the theory by the experiment.

4.3 Comparison of classification performance among analytic learning classifier (ALC), SVM and ELM

In order to verify the effectiveness of the analytical learning classifier, we compare it with SVM and ELM in Table 6. Because our algorithm adopts the PCA method, in order to ensure fairness and reliability of comparison, we also use the PCA method for SVM, which is SVM $+$ PCA. The experiments show that although the recognition rate of large sample data and banknote data set under ALC is slightly lower than ELM, the recognition rate of other small samples is higher than ELM, and except for Iris data set, the training speed of other data sets under the analytical learning classifier is more than three times faster. Compared with SVM, although the recognition rate of large data is slightly lower, its training speed is more than 6 times faster. For small data sets, except for Iris data, other data sets are much faster. Meanwhile, in addition to Waveform and Digits data sets, the recognition rates of other data sets are slightly higher.

From the Table 6, it can be seen that the training speed of the analytical learning classifier is much higher than that of SVM. The reason is that the fully connected weight parameter W (Fig. 1a) used by the analytical learning classifier is obtained by matrix multiplication, which avoids the complex iterative process in traditional methods and effectively improves the training speed, as shown in Eq. (7). Its Time complexity involves matrix multiplication and matrix inversion. The Time complexity of basic matrix multiplication is $O(n^{4})$ for a matrix with size n*n, but there are some optimization algorithms of Matrix multiplication that can divide the matrix phase by different inverses, reaching $O(n^{3})$ and O $(n^{\log_{2}{7}})$ .

4.4 Multistage analytical learning classifier

4.4.1 Results of Multistage analytical learning classifier

From the experimental data, we find that the recognition rate of samples with $f\geqslant 1$ is significantly higher than that of samples with $f<1$ . For example, in Waveform dataset, the recognition rate of samples with $f\geqslant 1$ is 91.52%, while the recognition rate of samples with $f<1$ is only 79.47%. Under this premise, we tried multistage analytic learning classifier. The results are shown in Table 7.

Table 7
Recognition rate results of 9 data sets on multi-stage ALC network

Datasets	Acc.
Mnist	94.13 $\rightarrow$ 95.14 $\rightarrow$ 95.54 $\rightarrow$ 95.98
Fashion mnist	84.43 $\rightarrow$ 86.03 $\rightarrow$ 87.25 $\rightarrow$ 88.37
Digits	98.21
Banknote	99.61 $\rightarrow$ 99.73
Pima	77.35 $\rightarrow$ 78.94 $\rightarrow$ 81.25
Breast	96.25
Waveform	85.88 $\rightarrow$ 88.05 $\rightarrow$ 89.31 $\rightarrow$ 90.21
Wine	98.87
Iris	98.66

Table 8

Performance of Multi-stage analytical learning classifier recognition rate before and after adding noise with 10 dB SNR

Datasets	No noise	Noise
Mnist	95.98	95.91
Fashion mnist	88.37	88.63
Banknote	99.73	99.67
Pima	81.25	81.53
Waveform	90.21	90.47

Figure 5.

The impact of SNR on the recognition rate of Waveform dataset.

It can be seen from the table that the multi-stage analytical learning classifier can effectively improve the recognition rate, especially for large datasets such as Mnist and Fashion. The number of data in each row of the table represents the number of stages of the network, among which Digits, Breast and Iris cannot be applied due to the small number of data samples. For the large datasets MNIST and fashion MNIST, the improvement of recognition rate is very obvious. Compared with ELM and SVM results in Table 6, the recognition rate is similar. For some small datasets with low recognition rates, such as Pima and Waveform, there is also a significant improvement, which is a substantial improvement over SVM and ELM. In general, multistage analytical learning classifier is efficient and feasible.

4.4.2 Effect of noise

In image classification, noise may have an impact on the experimental results. Based on this consideration, we conducted a noise experiment and added Gaussian noise (noise whose probability density function conforms to the Gaussian distribution) to the multi-stage analytical learning classifier. During the experiment, we added Gaussian noise to compare the recognition rate under different SNR. As shown in Fig. 5, the curve of recognition rate of Waveform dataset changes with SNR. From the figure, it can be seen that a low SNR has a significant impact on the recognition rate. When the SNR is greater than 10 dB, the impact on the recognition rate is relatively small, and in some cases, it may even improve the recognition rate. This is because our method is based on MSE and inherently has a good ability to resist noise influence. The experimental result under 10 dB SNR are shown in the Table 8.

From the experimental results, it can be seen that the influence of noise greater than 10 dB on the multi-stage analytical learning classifier is relatively small, and it can maintain a relatively stable recognition effect when adding noise, further verifying the effectiveness and stability of the classifier.

5. Conclusion

In this paper, a new analytical learning classifier with simple structure is proposed, and the analytical expression of full connection layer $\bm{W}$ is obtained. This new network structure avoids the complexity of repeated iteration, can effectively classify different samples, and has good interpretability. Meanwhile, through experiments, by large data sets represented by MNIST and small data sets such as Iris and Pima, it can be found that the analytical learning classifier can not only realize the $C-1$ dimensional features under the pure linear network, but also extend the features to the higher-dimensional feature space through feature preprocessing and activation function to find the better separation hyperplane, so as to correctly classify the samples. The network is compared with SVM and ELM in terms of training speed and recognition rate. The results show that, in general, this method has advantages on small dataset both in recognition accuracy and training speed, while it has advantages in training speed on large datasets. Finally, by introducing a multi-stage network structure based on the latent feature norm, the classifier network can further significantly improve the classification performance,the recognition rate of small data sets is effectively improved and much higher than that of existing methods, while the recognition rate of large data sets is similar to that of SVM.

In the future, we will further improve the network, including the optimization method of network parameter about principal component and latent feature number, the distribution characteristics of class conditional probability density, the comparative application of other feature preprocessing methods, and the method of using PEDCC for pattern recognition, etc.

References

Jain

A.K.

Duin

R.P.W.

and Mao

, Statistical pattern recognition: A review, IEEE Transactions on Pattern Analysis and Machine Intelligence 22(1) (2000), 4–37. doi: 10.1109/34.824819.

Zhu

and Zhang

, A Classification Supervised Auto-Encoder Based on Predefined Evenly-Distributed Class Centroids, CoRR, abs/1902.00220, 2019. http://arxiv.org/abs/1902.00220.

Zhu

Zhang

Wang

and Ye

, A new loss function for CNN classifier based on predefined evenly-distributed class centroids, IEEE Access 8 (2020), 10888–10895. doi: 10.1109/ACCESS.2019.2960065.

Zhu

and Wang

, An image clustering auto-encoder based on predefined evenly-distributed class centroids and mmd distance, Neural Processing Letters 51(2) (2020), 1973–1988. doi: 10.1007/s11063-020-10194-y.

Zhu

Q.-y.

and Li

T.-t.

, Semi-supervised learning method based on predefined evenly-distributed class centroids, Applied Intelligence 50(9) (2020), 2770–2778. doi: 10.1007/s10489-020-01689-1.

Zhu

and Ye

, Incremental classifier learning based on PEDCC-loss and cosine distance, Multimedia Tools and Applications 80(25) (2021), 33827–33841. doi: 10.1007/s11042-021-11163-w.

Zhu

Luan

and Zu

, Active Learning of Pattern Classification Based on PEDCC-Loss, IEEE Access 9 (2021), 147626–147633. doi: 10.1109/ACCESS.2021.3123095.

Zhu

Zheng

and Yan

, Effective Out-of-Distribution Detection in Classifier Based on PEDCC-Loss, CoRR, abs/2204.04665, 2022. doi: 10.48550/arXiv.2204.04665.

Zhu

Zheng

Shen

and Wang

, Out-of-Distribution Detection Based on Feature Fusion in Neural Network Classifier Pre-Trained by PEDCC-Loss, IEEE Access 10 (2022), 66190–66197. doi: 10.1109/ACCESS.2022.3184694.

10.

Yan

Zhu

and Zheng

, Generation and frame characteristics of predefined evenly-distributed class centroids for pattern classification, IEEE Access 9 (2021), 113683–113691. doi: 10.1109/ACCESS.2021.3083764.

11.

Shang

Sohn

Almeida

and Lee

, Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units, in: Proceedings of The 33rd International Conference on Machine Learning Balcan

M.F.

and Weinberger

K.Q.

, eds, Proceedings of Machine Learning Research, Vol. 48, PMLR, New York, New York, USA, 2016, pp. 2217–2225. https://proceedings.mlr.press/v48/shang16.html.

12.

Zhang

and Iyengar

V.S.

, Recommender systems using linear classifiers, The Journal of Machine Learning Research 2 (2002), 313–334.

13.

Skurichina

and Duin

R.P.

, Bagging, boosting and the random subspace method for linear classifiers, Pattern Analysis & Applications 5(2) (2002), 121–135.

14.

Noble

W.S.

, What is a support vector machine? Nature Biotechnology 24(12) (2006), 1565–1567.

15.

Byvatov

and Schneider

, Support vector machine applications in bioinformatics, Applied Bioinformatics 2(2) (2003), 67–77.

16.

Majdisova

and Skala

, Radial basis function approximations: Comparison and applications, Applied Mathematical Modelling 51 (2017), 728–743.

17.

Xie

Paszczyñski

and Wilamowski

B.M.

, Advantages of radial basis function networks for dynamic system design, IEEE Transactions on Industrial Electronics 58(12) (2011), 5438–5450.

18.

Cantarella

G.E.

and de Luca

, Multilayer feedforward networks for transportation mode choice analysis: An analysis and a comparison with random utility models, Transportation Research Part C: Emerging Technologies 13(2) (2005), 121–155.

19.

Daqi

and Genxing

, Influences of variable scales and activation functions on the performances of multilayer feedforward neural networks, Pattern Recognition 36(4) (2003), 869–878.

20.

Huang

G.-B.

Zhu

Q.-Y.

and Siew

C.-K.

, Extreme learning machine: a new learning scheme of feedforward neural networks, in: 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541), Vol. 2, 2004, pp. 985–990. doi: 10.1109/IJCNN.2004.1380068.

21.

Huang

G.-B.

Zhou

Ding

and Zhang

, Extreme learning machine for regression and multiclass classification, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42(2) (2011), 513–529.

22.

Kuo

C.-C.J.

, Understanding convolutional neural networks with a mathematical model, Journal of Visual Communication and Image Representation 41 (2016), 406–413.

23.

Lin

Zhou

You

Rao

and Kuo

C.-C.J.

, From two-class linear discriminant analysis to interpretable multilayer perceptron design, arXiv preprint arXiv:2009.04442, 2020.

24.

Lange

and Tong Wu

, An MM algorithm for multicategory vertex discriminant analysis, Journal of Computational and Graphical Statistics 17(3) (2008), 527–544.

25.

T.T.

and Lange

, Multicategory vertex discriminant analysis for high-dimensional data, The Annals of Applied Statistics 4(4) (2010), 1698–1721. doi: 10.1214/10-AOAS345.

26.

Dentamaro

Impedovo

Pirlo

and Massaro

, Vertex feature classification (VFC), in: 2020 IEEE Conference on Evolving and Adaptive Intelligent Systems (EAIS), IEEE, 2020, pp. 1–8.