A new probabilistic neural network model based on backpropagation algorithm

Abstract

Since the traditional probabilistic neural network (PNN) cannot systematically solve the difficulty of estimating probability function and the high space complexity, this paper introduces backpropagation (BP) algorithm into the classical PNN. By designing appropriate error function and BP algorithm based on the steepest descent, an improved BP-PNN is presented, with its algorithm and effectiveness deduced. Three synthetic datasets and ten benchmark problems have been tested, compared with Probabilistic Neural Networks (PNN), Multi-Layered Perceptron (MLP) and Support Vector Machine (SVM). The results prove that (1) the accuracy of classification of BP-PNN is much higher than PNN, and it has a significant advantage compared with MLP and SVM; (2) BP-PNN has strong capacity to identify the importance of input indicators; (3) BP-PNN is a new pattern classification method to estimate the probabilistic function, reduce the space complexity and identify the importance of the indicators.

Keywords

Probabilistic neural network backpropagation algorithm classification decision analysis

1 Introduction

Probabilistic Neural Network (PNN) is a common neural network model on Bayesian classifier and the probabilistic density function [1 –4]. PNN has a wide range of applications in model identification, time series prediction, credit evaluation, discrete pattern recognition, as well as fault diagnosis and other fields [5 –11]. Specht [3] pointed out that PNN learned quickly from examples in one pass and asymptotically achieve the Bayes-optimal decision boundaries. The classical PNN is characterized by simple process and fast convergence, and it can converge at Bayes optimal solution with high stability [12]; however, this method depends on the probability density function of Parzen window estimated by nonparametric method [13], and needs all the samples to calculate [14], thus two important problems should be further studied. One problem is the estimation of probability density function. A method based on Parzen window needs a large amount of samples, and determination of the window width has constantly been a hot topics discussion. There have been a great many studies on the improvement of PNN which relate to the method to estimate the probability density function. For example, Michael et al. [15] employed hidden node method to support the estimation of probability density function; Yu [16] and Mishra [17] introduced functional transformation to solve the estimation of probability density function; Ramakrishnan [18] designed different forms of probability density function for comparative analysis. Another problem is the high space complexity of PNN pattern layer. This problem is mainly caused by using all the samples to construct network. For this purpose, there have been a great many studies on the method to reduce this complexity, for example, Specht [4] cut down the number of samples by clustering learning; Xing [19] and Huang [20] determined the hidden nodes by polynomial least square method to reduce this space complexity. These improved methods above have succeeded in specific scopes of application, however, they cannot systematically solve the problem of estimating the function and the problem of reducing the complexity at the same time; moreover, no appropriate mechanism was designed to judge the indicator importance, which is typically important for more accurate classification and providing better reference for making decisions.

In fact, BP mechanism is a classic statistical learning method. For the shortcomings of BP, such as long computation time, instable network learning and memory, many researchers have combined it with other neutral network algorithms. Lee [21] put forward the integration of Newton’s method and BP; Christian, Summy and Jose [22 –24] improved the BP algorithm using evolutionary strategy; Kirkpatrick and Chowdhury [25, 26] combined the simulated annealing algorithm and the conventional BP algorithm; Yam [27] applied Cauchy inequality and linear algebra method to optimize the initial weight value of the network, so as to improve the convergence rate of BP. Ashena et al. [28 –31] proposed the integration of genetic algorithm (GA) and BP neural network. In these fusion algorithms the effectiveness and the stability of BP has been improved to some extent, but still does not apply to complex classification problems with large training sample sets. In 2015, José Miguel Hernández-Lobato and Ryan P. Adams [32] presented probabilistic backpropagation (PBP), a new algorithm for scalable Bayesian learning of neural networks (NN). PBP uses a product of Gaussians to approximate the posterior over weights. The parameters of these Gaussians are updated in a two stage process similar to the one used by the backpropagation algorithm. Their experiments show that PBP makes accurate predictions and can be feasible to compute for large networks. But PBP is based on NN with hidden layers. Because the number of hidden layer and the number of hidden neurons in each hidden layer are uncertain, the authors need to try in the experiments. And in PBP more prior factors and likelihood factors are incorporated in a factored distribution q, which lead to inference and calculation more complicated.

Based on the problems and ideas above, it is noticed that the original PNN has no BP mechanism, resulting in the difficulty of estimating the function and the excessively high complexity of the algorithm, as well as inability to identify the importance of the indicators. However, the introduction of BP mechanism allows the function to have a clear learning direction, with no need to involve all the samples into the training. So it is helpful to reduce the complexity, more importantly, to identify the important indicators through the learning and training of weight.

This paper focuses on backpropagation (BP) algorithm of multi-layer perceptron (MLP) network, and combines the relevant principles of BP control in control theory, to perform sample learning effectively. When it is able to introduce this more effective learning mechanism into the existing PNN, is it helpful to estimate probabilistic density function, reduce the complexity and identify the importance of the indicators? Especially, PNN has only one pattern layer, then the introduction of BP is easier than NN? This paper proposes to introduce BP mechanism to overcome the difficulty of estimating the probability density function and the high space complexity of the network, as well as to identify the importance of the variables effectively. This new approach is called backpropagation probabilistic neural networks (BP-PNN).Thus, the contributions and innovations of this study are as follows:

The mechanism of BP algorithm is introduced into the classical PNN, which will enable BP-PNN to have a strong learning ability, so as to determine the probability density function and the importance of the input variables.

The samples are divided into learning samples and training samples to reduce the space complexity (space overhead) of the network.

Numerical experiments and application experiments are designed to verify the effectiveness of the model and its good properties.

The rest of this paper is organized as follows: the structure and the algorithm of BP-PNN are presented in Section 2. Section 3 includes the relative steps of the learning and training. The applications are provided to illustrate efficiency of the proposed model respectively with 3 artificial and 10 actual classification problems in Section 4 and Section 5. Finally, concluding remarks are given in Section 6.

2 Structure and algorithm of BP-PNN

2.1 Classical PNN

PNN is a structured and parallel processing pattern identification method, which is composed of threelayers, namely the input layer, the pattern layer and the class layer. In PNN, the input layer represents the several input variables of training sample, with the number equal to that of sample attributes. The pattern layer is the probability functions obtained from the samples, with the number equal to that of samples. And the class layer means the classes of the samples in the pattern layer, with the number equal to that of sample classes, as shown in Fig. 1.

Assuming that a classification problem has k classes as C₁, C₂, …, C_k, and the classification rules are made by m-dimensional feature vectors, X = (X₁, X₂, …, X_m). x_ij represents the i-th input variable of the j-th sample, wherein, i = 1, 2, ⋯ , d, j = 1, 2, ⋯ , n. f_j represents the probability density function of the j-th sample in the pattern layer. c_k is the score in the k-th class, k = 1, 2, ⋯ , m. Unless specially stated, the symbols below have the same meanings defined here, so f_j is defined as below: $f_{j} (x_{j, g}^{input}) = exp [- \frac{(x_{j, g}^{input} - x_{j}^{f})^{2}}{2 h_{j}^{2}}]$ (1)

Where $x_{j, g}^{input}$ is the independent variable of function f_j, namely the input of the j-th density function in the pattern layer when the input test sample is g (g = 1, 2, ⋯ , n). $x_{j}^{f}$ is the mean parameter of function f_j, which can be calculated from the input indicator of the j-th sample refer to Equation (4). h_j represents the window width. This parameter must be determined first in the functions above, while its estimation is exactly one of the challenges in PNN.

With the definitions of the symbols above, the training algorithm of PNN is as below. Let w_ij represent the weight of x_i in the input layer to Sample j in the pattern layer, and its value is: $w_{ij} = x_{ij} / (\sum_{s = 1}^{d} x_{sj}^{2})^{1 / 2}$ (2)

With the calculation method of the weight w_ij as Equation (2), it is able to get the calculation methods of $x_{j, g}^{input}$ and $x_{j}^{f}$ given above, which are expressed as Equations (3) and (4) respectively: $x_{j, g}^{input} = \sum_{i = 1}^{d} x_{ig} \cdot w_{ij}$ (3) $x_{j}^{f} = \sum_{i = 1}^{d} x_{ij} \cdot w_{ij}$ (4)

Each of the connection lines from the pattern layer to the class layer represents the class of each sample, for example, Sample 1 is of Class m, and then there is a connection line between. This relation between the pattern layer and the class layer is denoted as t_jk, which is defined as: $t_{jk} = {\begin{matrix} 1 & Sample j \in Class k \\ 0 & Sample j \notin Class k \end{matrix}$ (5)

The training algorithm of PNN is very simple, for which only the values of the input variables are used. This method features low time complexity, but high space complexity as the number of the units in the pattern layer is equal to that of samples. At the same time, it can be found that the weight of each indicator is consistent during the training.

2.2 Introduction backpropagation mechanism

Because PNN only includes a forward propagation of probabilities, it is only a model training process and not a model learning process. So backpropagation algorithm is introduced the classical PNN to build a new model, which is called backpropagation probabilistic neural networks (BP-PNN). BP-PNN does not have the disadvantages of classical PNN and the structure is as shown in Fig. 2.

BP-PNN has a two stage process, namely Model Learning and Model Training. And the training samples in the classical PNN are split into the learning samples and training samples in the new model.

In the first phase, the input data of learning samples is propagated forward through the network to obtain the initialized value of the variable weight. The number of nodes in the pattern layer is equal to that of learning samples, so that BP-PNN greatly reduces the space complexity relative to PNN.

In the second phase, the error function is propagated back as the same as in classical BP. Based on the error function this model uses the training samples to determine the values of the parameters of the function, and the weight matrix in the network.

In BP-PNN, the error function is the core of the introduction of BP algorithm. Following the arrow of error function, two variable parameters play a role.

H represents the window width to estimate probability density, which will overcome the difficulty of the existing Parzen window method.

W is the weight matrix from the input layer to the pattern layer. This parameter means the importance of input variable, in order to improve the capabilities of accurate classification and rational decision-making.

2.3 Learning rules of network parameters

2.3.1 Design of error function

For any sample q in a training set (let the number of samples in this training set be n_q), the vector composed of its input variables is denoted as X_q, with d elements, X_q = [x_1q, x_2q, ⋯ , x_dq]. The weight matrix from the input layer to the pattern layer is denoted as W, and it is d × n_p matrix. n_p is the number of learning samples, equal to the number of nodes in the pattern layer, and thus W = [w_ip] _{d×n_p}. F is the vector composed of n_p probability density functions in the pattern layer, F = [f₁, f₂, ⋯ , f_{n
_p}], and let F_q = F (X_q · W). T means the attribute matrix from the pattern layer to the class pattern, T = [t_pk] _{n_p×m}. C_q is the score vector at each class when a training sample q passes the neural network. $C_{q} = F_{q} \cdot T = F (X_{q} \cdot W) \cdot T$ (6)

It is easy to see that C_q is an m-dimensional row vector. Through the normalization of its elements ◊C_q is obtained, which is expressed as: $◊ C_{q} = C_{q} / (C_{q} \cdot E^{T})$ (7)

Where E is a row vector composed of m’s 1. Then, the real class of training sample q is Class k, and the vector of its real class is expressed as $C_{q}^{real}$ , where, the k-th element is 1, representing the class of the sample, with other elements as 0. Accordingly, it is obtained that the error function of the q-th training sample is E_q, which is defined as: $E_{q} = E (◊ C_{q}, C_{q}^{real}) = 1 - ◊ C_{q} \cdot (C_{q}^{real})^{T}$ (8)

Equation (8) is exactly the definition of error function proposed in this paper, which is characterized in that:

The value is zero when $◊ C_{q} = C_{q}^{real}$ .

The value of E_q is between 0 and 1.

For Sample q of Class k, when the k-th element of ◊C_q is the greater, the error function defined above is the smaller. The result of model classification is closer to the real situation.

It is easy to get the three characters of the model above. However, it is only needed to notice that the probability density function constituting F is not negative, so that each element in ◊C_q is not negative. And the detailed proof of these characters will not be repeated here. In fact, the three characters also indicate the rationality of the error function defined by Equation (8).

2.3.2 Learning rules of W and H

Through the consociation of Equations (6–8), it is possible to get an error function specifically expressed with the input parameters and the weight factors in the network, which is expressed as: $E_{q} = 1 - \frac{F (X_{q} \cdot W) \cdot T \cdot (C_{q}^{real})^{T}}{F (X_{q} \cdot W) \cdot T \cdot E^{T}}$ (9)

Thus, the learning rules of W and H are obtained as below, where, H is a row vector composed of p window width parameters of the probability density function in the pattern layer, which is expressed as H = [h₁, h₂, ⋯ , h_p].

$Δ W = - η \cdot \frac{\partial E_{q}}{\partial W}$ (10) $Δ H = - η \cdot \frac{\partial E_{q}}{\partial H}$ (11)

Where η represents the learning rate to control the speed of learning process.

Specifically, taking Equation (10) as an example, the learning rule for an element w_ip in W can be expressed as: $Δ w_{ip} = - η \cdot \frac{\frac{\partial F}{\partial w_{ip}} \cdot T \cdot (C_{q}^{real})^{T} \cdot F \cdot T \cdot E^{T} - F \cdot T \cdot (C_{q}^{real})^{T} \cdot \frac{\partial F}{\partial w_{ip}} \cdot T \cdot E^{T}}{{(F \cdot T \cdot E^{T})}^{2}}$ (12)

Where F represents F (X_q · W); in fact, for the p-th element in F (p = 1, 2, ⋯ , n_p), according to the definitions of Equations (1), (3) and (4), there is specifically,

$\begin{matrix} \frac{\partial f_{p} (X_{q} \cdot W)}{\partial w_{ip}} \\ = - \frac{w_{ip} \cdot (x_{iq} - x_{ip})^{2}}{h_{p}^{2}} \\ \cdot exp [\frac{- \sum_{s = 1}^{d} w_{sp}^{2} \cdot (x_{sq} - x_{sp})^{2}}{2 h_{p}^{2}}] \end{matrix}$ (13)

At the same time, it is noticed as a fact that:

$\begin{matrix} \frac{\partial F}{\partial w_{ip}} \\ = [\frac{\partial f_{1} (X_{q} \cdot W)}{\partial w_{ip}}, \frac{\partial f_{2} (X_{q} \cdot W)}{\partial w_{ip}}, \dots, \frac{\partial f_{n_{p}} (X_{q} \cdot W)}{\partial w_{ip}}] \end{matrix}$ (14)

When Equations (13) and (14) are substituted into Equation (12), the definition of t_jk by Equation (5) shall be noticed, whose learning rule is clear, and can be quantified specifically by substituting the known parameters.

Similarly, an element h_p in H can be expressed as: $Δ h_{p} = - η \cdot \frac{\frac{\partial F}{\partial h_{p}} \cdot T \cdot (C_{q}^{real})^{T} \cdot F \cdot T \cdot E^{T} - F \cdot T \cdot (C_{q}^{real})^{T} \cdot \frac{\partial F}{\partial h_{p}} \cdot T \cdot E^{T}}{(F \cdot T \cdot E^{T})^{2}}$ (15)

Specifically, $\frac{\partial f_{p} (X_{q} \cdot W)}{\partial h_{p}} = \frac{\sum_{s = 1}^{d} w_{sp}^{2} \cdot (x_{sq} - x_{sp})^{2}}{h_{p}^{3}} \cdot exp [\frac{- \sum_{s = 1}^{d} w_{sp}^{2} \cdot (x_{sq} - x_{sp})^{2}}{2 h_{p}^{2}}]$ (16)

Thus the design of learning algorithm is completed. Theoretically, there is no essential difference between this algorithm and backpropagation learning algorithm in MLP network. This is only a specific problem, and the deduction is performed with respect to a specialized error function and a special function form.

2.4 Theoretical deduction of algorithm effectiveness

This paper is intended to prove the three improvements of BP-PNN, namely small space overhead, high accuracy of classification and high ability to identify the importance of the variables. Small space overhead will be proved in Section 3, and high accuracy of classification will be explained through simulation experiment in Section 4 and Section 5, therefore, this section focuses on theoretical deduction of the existence of the ability to identify the importance of the variables and its quantitation. Before this deduction, it can be found that the core of the improved algorithm of this paper is the introduction of BP mechanism and the application of the classical optimization method based on the steepest descent. Therefore, the effectiveness proving of the algorithm with respect to “convergence” and “convergence rate” is a classical and settled problem. These can be found in many teaching materials about optimization and neural network, for example, the works of Xue [33], Fredrick and Ivica [34] and so on, and the details will not be repeated here.

When taking the partial derivative of the p-th output variable f_p (X_q · W), with respect to the i-th input variable x_ip, it can be obtained that:

$\begin{matrix} \frac{\partial f_{p} (X_{q} \cdot W)}{\partial x_{ip}} = \frac{w_{ip}^{2} \cdot (x_{iq} - x_{ip})}{h_{p}^{2}} \\ \cdot exp [\frac{- \sum_{s = 1}^{d} w_{sp}^{2} \cdot (x_{sq} - x_{sp})^{2}}{2 h_{p}^{2}}] \end{matrix}$ (17)

The equation above shows that: when x_iq ≠ x_ip, namely when the q-th training sample is consistent with the i-th variable of the p-th learning sample, the variation of the input variable has an influence on the output variable, and is restricted by the weight matrix and the function parameters of the pattern layer. Through the learning of the training samples, it is able to quantify all the influences to determine the index of the importance of the input variables ${Index}_{i}^{p}$ . It means the importance of the i-th input variable to the p-th output variable, and is defined as below: $I n d e x_{i}^{j^{'}} = \sqrt{\frac{\sum_{j'' = 1}^{n^{″}} {(\frac{\partial_{j^{'}} (X_{j^{″}} \cdot W)}{\partial x_{i j^{'}}})}^{2}}{n ″}}$ (18)

The equation above includes first derivatives only; however, it takes the sum of the squares of the first derivatives. Therefore, no matter the input variable is in positive, negative and/or any other non-liner relation with the output variable, when the importance of the independent variable is high, the value of the Equation (18) is great. When the importance of the independent variable is low, the value is close to zero. It is noticed that the equation above takes the values of all training samples on average; therefore, it is a comprehensive index to measure the importance of the variables.

The analysis above indicates the ability of the model to identify the importance of the variables, and quantifies the importance of the variables. In fact, there is no training process in traditional PNN, and all the samples are involved in learning. This is equivalent to the condition that all the first reciprocals are zero when x_iq = x_ip in Equation (17), namely the training samples are consistent with the learning samples. Therefore, the classical PNN is not able to identify the importance of the input indicators. Admittedly, the analysis above proves the existence of the ability to identify the importance of the input indicators, and quantifies the importance of the indicators. However, it is necessary to make numerical experiment for a comparison between different methods, to determine the ability of identification. This issue is to be analyzed in Section 4 Numerical Experiment.

2.5 Further discussion of application of model

The BP-PNN model integrates both characteristics of PNN and BP mechanism of MLP network, and is divided into the learning process and the training process in application. For the learning process, this model relies on the existing PNN for the learning of some of the samples, so as to determine the number of nodes in the pattern layer and the initial values of the weights in the network, which inspires the further BP learning. For the training process, this model uses the remaining samples for BP training based on the error function and BP algorithm given above, to determine the values of the parameters of the function, and the weight matrix in the network.

As for the learning process and the training process above, it can be found that the BP-PNN model integrates both characteristics of PNN and BP method. The introduction of BP mechanism is helpful to overcome the disadvantages of PNN, such as high space complexity and difficulty of identifying the importance of the input variables. Meanwhile, the architecture and leaning process of PNN is helpful to determine the number of nodes in the intermediate layer and the initial values of the weights in the network, which are two knotty problems of the existing BP algorithm.

It can be found that there are three differences between BP-PNN and PBP [32].

The network structure is different. BP-PNN is based on PNN. The number of nodes in the pattern layer is equal to that of learning samples, so it is certain. PBP is based on NN with hidden layers. In each hidden layer the number of hidden layer and the number of hidden neurons are uncertain, so the authors need to try in the experiments.

The learning algorithm is different. In BP-PNN there is no essential difference between this algorithm and the backpropagation learning algorithm in MLP network. This is only a specific problem, and the deduction is performed with respect to a specialized error function and a special function form. PBP uses a product of Gaussians to approximate the posterior over weights. PBP does one full expectation propagation update of these approximate factors for the prior after each ADF pass through the data.

The sample classification method is different. The learning samples and training samples in BP-PNN are split according to the proportion of 3:7. They are corresponding to model learning and model training. PBP can Active learning. It split each data set into training and test sets, and pool sets with all the remaining data. PBP is fitted using the training data, and then its performance is evaluated on the test data. After this, one data point is collected from the pool set and then moved into the training set.

Although BP-PNN and PBP can be applied to massive data, BP-PNN algorithm is relatively simple and has a smaller space and higher recognition ability of variable importance.

3 Steps of model learning and model training

There are two important rules in model learning and training: one is used to describe the method to classify the samples into learning samples and training samples; and the other is used to determine the criterion of the end of training. This section discusses the two rules at first, and then gives the flowchart of the model and training.

Rule I: The samples are classified at first, and then 30% samples of each class are selected randomly for model learning, with the remaining samples for mod-el training.

Rule II: The end of model training is set at the time when the variation of each parameter to be solved is less than one ten thousandth, or the number of training steps comes to one thousand.

Explanation of the design intention of these rules and the proof of its rationality is as follows:

In Rule I, selecting the samples by classes is for balancing. If selecting is made completely randomly, there might be any class in which only a small amount of samples are selected or even no sample is selected. Selecting samples according to the ratio of the existing samples in the classes is helpful to maintain the basic characteristics of the original samples, and eliminates the situation that there is any class in which no sample is selected. Selecting the learning samples and the training samples according to the proportion of 3:7, it is in accordance with the basic characteristics of BP training, as well as the existing rule of a large amount of studies to select two kinds of samples. Thus, it is able to get that the number of learning samples is n_p, the number of training samples is n_q, which meets the condition n_p + n_q = n.

In Rule II, the criterion of the end of training is generally applicable for BP network, which is able to control the time of training effectively, and to ensure the accuracy of training to a certain extent.

Based on the rules above, the flowchart of model learning and the flowchart of model training in BP-PNN are shown in Figs. 3 and 4, which are the foundation of model programming.

The analysis of space complexity above indicates that less nodes in the pattern layer means less space overhead. Because the number of nodes in the pattern layer is equal to that of learning samples, and the introduction of BP mechanism does not involve all the samples for learning. Therefore, when only 30% samples are involved for learning, it is safe to come to the conclusion that 70% space overhead is saved; when only n^0.5 samples are involved, the space complexity can be reduced exponentially.

4 Numerical experiments

The numerical experiment of this paper is intended to verify the two advantages of BP-PNN: one is its ability to identify the importance of the input indicators, and another is the accuracy of model classification. In this paper, three different types of classification function are used to test and investigate the performance and characteristics of BP-PNN.

In these functions, there are ten input variables (X₁ ∼ X₁₀) and a binary output variable which denotes two classes. The input variables of each function are in the range of 0∼1, producing 1000 data. According to the steps shown in flowchart, 85% samples above are selected randomly for learning and training (the proportion is 3:7), and the remaining 15% samples are to be tested. The three data sets will be used to compare the accuracy of compared them with Probabilistic Neural Networks (PNN), Multi-Layered Perceptron (MLP) and Support Vector Machine (SVM). The three classification functions are as shown below.

Experiment A is linear and quadratic classification function. Assuming the classification function: $\begin{matrix} Y & = & 4 \cdot [0 \cdot (X_{1} - 0.5)^{2} - 1 \cdot (X_{2} - 0.5)^{2} \\ + 2 \cdot (X_{3} - 0.5)^{2} - 4 \cdot (X_{4} - 0.5)^{2} \\ + 8 \cdot (X_{5} - 0.5)^{2}] + 0 \cdot X_{6} + 1 \cdot X_{7} - 2 \cdot X_{8} \\ + 4 \cdot X_{9} - 8 \cdot X_{10} \end{matrix}$ if Y > 0, it will belong to the first category; otherwise, the second category. It is clear that the relationship between X₁ ∼ X₅ and output variable is quadratic and increases gradually, and that the relationship between X₆ ∼ X₁₀ and output variable is linear and increases gradually.

Experiment B is mixed classification function. Assuming the following classification function: $\begin{matrix} Y & = & X_{1} - X_{2} + (X_{5} - 0.25)^{2} - (X_{6} - 0.75)^{2} \\ + (X_{9} - 0.25) \cdot (X_{10} - 0.75) \end{matrix}$ if Y > 0, it will belong to the first category; otherwise, the second category. X₁ and X₂ are the linear effect variables; X₅ and X₆ are the curvature effect variables, X₉ and X₁₀ are a group of variables with interaction, and the others are irrelevant variables.

Experiment C is interactive and quadratic classification function. Assuming the following classification function: $\begin{matrix} Y & = & 1.5 (X_{1} - 0.5) (X_{3} - 0.5) \\ + 2 (X_{5} - 0.5) (X_{7} - 0.5) + (X_{9} - 0.5)^{2} \end{matrix}$ if Y > 0, it will belong to the first category; otherwise, the second category. {X₁, X₃} and {X₅, X₇} are the two groups of interactive variables; X₉ is the curvature effect variable; and the rest five variables are irrelevant variables.

In order to measure the importance of variables, v₁ represents the sum of weight of the i-th input variable, which is defined as: $v_{i} = \sum_{p = 1}^{n_{p}} w_{ip}$ (19)

This value is used to represent the weight of the input indicator in BP-PNN. It is noticed that the input data of three experiments have the same order of magnitude and the same variation scope; therefore, the indicators above are valid. The activating function used by the hidden layer in MLP network is the “tansig function”, the output layer uses the “logsig function”, and the number of nodes in the intermediate layer of the network is 2.

4.1 Evaluation of variable importance

In Experiment A, the importance indexes of X₁ ∼ X₅ and X₆ ∼ X₁₀ on the output variable increase gradually as shown in Fig. 5. The variable weights of X₅ and X₁₀ are relatively high, and (v₄ + v₅ + v₉ + v₁₀)/Σv_i is 0.8568 according to Equation (19), indicating that these variables are very important. This is consistent with the classification function above, and confirms the importance indexes can measure the importance of input variables of the classification function hybrid of linear and quadratic variables.

In Experiment B, X₁, X₂, X₅, X₆, X₉ and X₁₀ have much larger indexes, indicating that they are important variables; the others have much smaller indexes, indicating that they are irrelevant variables. In Fig. 6, (v₁ + v₂ + v₅ + v₆ + v₉ + v₁₀)/Σv_i is 0.9413, which confirms BP-PNN has a strong ability to measure the importance of input variables of the classification function hybrid of linear, quadratic, and interactive variables.

In Experiment C, Fig. 7 shows that X₁, X₃, X₅, X₇, and X₉ have much larger indexes, so they are important variables. (v₁ + v₃ + v₅ + v₇ + v₉)/Σv_i is 0.9189, so BP-PNN can measure the importance of input variables of the classification function hybrid of interactive and quadratic variables.

Figure 8 is the comparisons of the importance of variables in the three numerical experiments. Because PNN, MLP and SVM cannot present the importance of input variables, therefore these models are the black box model. However, the variable weight of BP-PNN may show the importance of input variables, such a characteristic is one of advantages of BP-PNN superior to PNN, MLP and SVM.

4.2 Evaluation of classification accuracy

Figure 9 shows the comparison of the classification accuracy in the three numerical experiments. It is clear that the classification accuracy of BP-PNN is close to that of MLP and SVM, but much higher than that of PNN. This is probably because in the learning process, unlike PNN that uses the fixed variable weights, BP-PNN can adjust the variable weight closer to the nature of the classification problem to reduce of the error of the model. And BP-PNN saves 70% space overhead compared with PNN, so it will be a good solution for larger data sets. In fact, the numbers of samples in the three experiments are not large. When there are more samples, the existing PNN is not applicable any longer. Because the memory capacity of computer is limited, too much result will overflow.

5 Application experiments

In this section, there are 10 benchmark problems in the UCI Machine Learning Repository [35] (http://archive.ics.uci.edu/ml) listed in Table 1. It is noticed that the real data sets from UCI are widely used for the study of data mining, so that it is of creditability and certain authority. The numbers of samples in the ten experiments are large in order to prove the applicability of the new model in these cases. Then it is compared with MLP, PNN, and SVM to verify the accuracy of classification of BP-PNN in more data sets.

In order to test the effectiveness of learning, 85% samples are selected randomly for learning and the remaining 15% samples are to be tested. And in the same way of Section 4, the proportion of the learning samples and the training samples is still 7:3 in BP-PNN. The performance of all the test data is used to assess the accuracy of each network model.

Table 2 shows the results of BP-PNN, MLP, PNN, and SVM. On each data set, the results of the best method are shown in bold. Overall, BP-PNN performs best, with BP-PNN obtaining the best results in 6 out of 10 data sets. MLP is slightly better than SVM, and PNN is the worst. Firstly, the accuracy of classification of BP-PNN is much higher than that of PNN because of the introduction of backpropagation mechanism. Unlike PNN, BP-PNN can greatly reduce the numbers of nodes, identify the importance of input variables, and calculate the window width. So it is effective in large scale settings. Secondly, BP-PNN has advantage compared with MLP and SVM. It can solve the uncertain of node number in MLP and has higher recognition ability in the variable importance than SVM. As a result, it has better classification accuracy than other two.

The convergence history can show the stability of a neural network paradigm. Figures 10–12 shows the convergence histories of BP-PNN on three typical cases, Insurance Company, Forest Cover Type, and Spambase data sets, which indicate that BP-PNN is a neural network paradigm with stability. When BP-PNN model is applied to a large-scale classification problem, it has higher accuracy of classification than others, and it would not result to the overflow of the result due to save the memory capacity of the computer. At the same time, the convergence history also shows that BP-PNN has good stability. Therefore, there is no need to make further design and adjustment to the structure of this model, and it is very easy to use, which is exactly the practical meaning of the method of this paper.

6 Conclusions and future work

This paper has established BP-PNN as a new pattern classification model, which can be used not only for the problem of pattern classification, but also for the practical problem of making management decisions. According to the establishment of method and the deduction of algorithm, it can be found the new model uses the basic structure of PNN and combines the BP algorithm of MLP. The new model is superior to PNN and MLP in terms of the capability of identifying the importance of the indicators and the accuracy of classification for either artificial data or actual data, so as to realize the complementation between PNN and BP, for the original purpose of the design.

Compared with the existing classical classification methods, this new model is characterized by three points.

Compared with PNN, BP-PNN has much less nodes in the pattern layer, which is helpful to reduce the space complexity of the model, so that it can be used to analyze classification question with larger scale. At the same time, this model is designed with a mechanism to identify the importance of the input indicators of the sample, so that the estimating the functions in the pattern layer becomes smoother and more direct, and there is no need to determine any unknown window width which requires a lot of computation resources. Accordingly, BP-PNN is at an advantage theoretically.

Compared with MLP, BP-PNN is designed with a clear structure, while there is great uncertainty when the existing MLP is used to determine the number of hidden nodes. At the same time, MLP is a “black box” model, while the pattern layer of BP-PNN has practical meaning, which comes from the samples. This model inherits the principle of PNN that the work scope of the samples in the pattern layer is estimated (embodied as the window width and the location), and the spatial relation between any sample to be determined and the samples in the pattern layer is investigated to determine its class. So BP-PNN is closer to the question itself than MLP from the perspective of the principle of model construction, and it is more rational.

Compared with SVM, BP-PNN shows good ability of classification and identification the importance of the indicators. Moreover, its application is much easier that there is no need to design or adjust the parameters as SVM does, which is convenient for the application of the new model of this paper to a certain extent.

However, this model has the shortcoming that it requires a long time for the computation, and a further study is necessary to solve this problem. Although it saves 70% space overhead compared with PNN, the number of parameters to be trained is much more than that of MLP, resulting to more computation time. We are going to make more explorations based on the present effect we have obtained so far, for example, to select less samples for learning try, or to accelerate the learning speed during BP process, or to improve PNN before the combination with BP, in hope of a better effect.

Footnotes

Acknowledgments

This paper was funded by the National Natural Science Foundation of China (71271070).

References

Specht

D.F.

, Probabilistic neural networks for classification, or associative memory, in: IEEE International Conference on Neural Networks, San Diego, 1988, pp. 525–535.

Specht

D.F.

, Probabilistic neural networks, Neural Networks 3(1) (1990), 109–118.

Specht

D.F.

and Shapiro

P.D.

, Generalization accuracy of probabilistic neural networks compared with back-propagation networks, in: International Joint Conference on Neural Networks, Seattle, Vol. 1, 1991, pp. 887–892.

Specht

D.F.

, Enhancements to probabilistic neural networks, in: International Joint Conference on Neural Networks, Baltimore, Vol. 1, 1992, pp. 761–768.

Adeli

and Panakkat

, A probabilistic neural network for earthquake magnitude prediction, Neural Networks 22(7) (2009), 1018–1024.

Song

, Jamshidi

M.M.

, Lee

R.R.

and Huang

, A modified probabilistic neural network for partial volume segmentation in brain MR image, in: IEEE Transactions on Neural Networks 18(5) (2007), 1424–1432.

Tang

M.Z.

, Yang

C.H.

and Gui

W.H.

, Cost-sensitive probabilistic neural network with its application in fault diagnosis, Control and Decision 25(7) (2010), 1074–1079.

Mehdi

and Mehdi

, Hybridization of the probabilistic neural networks with feed-forward neural networks for forecasting, Engineering Applications of Artificial Intelligence (25) (2012), 1277–1288.

Pang

S.L.

, Study on credit scoring model and forecasting based on probabilistic neural network, Systems Engineering-Theory & Practice 25(5) (2005), 43–48.

10.

Jin

Y.P.

and Li

B.L.

, Lithology identification based on genetic optimized radial basis probabilistic neural network, Journal of Computer Applications 33(2) (2013), 353–356.

11.

Savchenko

A.V.

, Probabilistic neural network with homogeneity testing in recognition of discrete patterns set, Neural Networks (46) (2013), 227–241.

12.

James

O.B.

, Statistical decision theory and bayesian analysis, Spring-Verlag, New York, 1985.

13.

Thierry

, A K-nearest neighbor classification rule based on Dempster-Shafer theory, Studies in Fuzziness and Soft Computing 21(9) (2008), 737–760.

14.

Yang

Z.R.

, Marjorie

B.P.

and Harlan

D.P.

, Probabilistic neural networks in bankruptcy prediction, Journal of Business Research 44(2) (1999), 67–74.

15.

Michael

R.B.

and Jay

, Constructive training of probabilistic neural networks, Neurocomputing 19(1-3) (1998), 167–183.

16.

S.N.

and Chen

Y.H.

, Electrocardiogram beat classification based on wavelet transformation and probabilistic neural network, Pattern Recognition Letters 28(10) (2007), 1142–1150.

17.

Mishra

, Bhende

C.N.

and Panigrahi

B.K.

, Detection and classification of power quality disturbances using S-transform and probabilistic neural network, in: IEEE Transactions on Power Delivery 23(1) (2008), 280–287.

18.

Ramakrishnan

and Ibrahiem

M.M.

, Comparative study between traditional and modified probabilistic neural networks, Telecommunication Systems 40(1-2) (2010), 67–74.

19.

Xing

and Xiao

D.Y.

, PCA-based probability neural network structure optimization, Journal of Tsinghua University (Science and Technology) 48(1) (2008), 141–144.

20.

Huang

D.S.

and Zhao

W.B.

, Determining the centers of radial basis probabilistic neural networks by recursive orthogonal least square algorithms, Applied Mathematics and Computation 162(1) (2005), 461–473.

21.

Lee

D.C.

, Kim

B.J.

and Yun

J.M.

, A combined method for improving backpropagation algorithm, in: IEEE International Conference on Systems, Man, and Cybernetics, Charlottesville, Vol. 1-3 1991, pp. 1581–1586.

22.

Igel

and Husken

, Empirical evaluation of the improved Rprop learning algorithms, Neurocomputing 50(1) (2003), 105–123.

23.

Sammy

, Yang

S.S.

and Lee

C.M.

, Improving the back-propagation algorithm Using Evolutionary Strategy, IEEE Transactions on Circuits and Systems II-Express Briefs 52(2) (2007), 171–175in.

24.

Jose

, Leonardo

and Patricia

, Hybrid backpropagation training with evolutionary strategies, Soft Computing 18(8) (2014), 1603–1614.

25.

Kirkpatrick

, Gelatt

C.D.

and Vecchi

M.P.

, Optimization by simulated annealing, Science 220(5) (1983), 671–680.

26.

Chowdhury

P.R.

, Singh

Y.P.

and Chansarkar

R.A.

, Dynamic tunneling technique for efficient training of multilayer perceptions, IEEE Transaction on Neural Networks 10(1) (1999), 48–55in.

27.

Yam

J.Y.F.

and Chow

T.W.S.

, A weight initialization method for improving training speed in feedforward neural network, Neurocomputing 30(1) (2000), 219–232.

28.

Rahman

and Jamshid

, Bottom hole pressure estimation using evolved neural networks by real coded ant colony optimization and genetic algorithm, Journal of Petroleum Science and Engineering 77(3) (2011), 375–385.

29.

Amin

A.E.

, A novel classification model for cotton yarn quality based on trained neural network using genetic algorithm, Knowledge-Based Systems 39(2) (2013), 124–132.

30.

Sepehr

and Hesam

, Thermal modeling of gas engine driven air to water heat pump systems in heating mode using genetic algorithm and Artificial Neural Network methods, International Journal of Refrigeration 36(8) (2013), 2262–2277.

31.

Kangji

and Hongye

, Forecasting building energy consumption with hybrid genetic algorithm-hierarchical adaptive network-based fuzzy inference system, Energy and Buildiings 42(11) (2010), 2070–2076.

32.

Hernández-Lobato

J.M.

and Ryan

P.A.

, Probabilistic backpropagation for scalable learning of bayesian neural networks, in: 32nd International Conference on Machine Learning, Lille, France, JMLR: W&CP, Vol. 37, 2015.

33.

Xue

J.Q.

, Optimization Theory and Method, Metallurgy Industry Press, Beijing, 2003.

34.

Fredric

M.H.

and Ivica

, Principles of Neurocomputing for Science & Engineering, McGraw-Hill Companies, New York, 2003.

35.

Frank

and Asuncion

, UCI machine learning repository [http://archive.ics.uci.edu/ml], School of Information and Computer Science, University of California, Irvine CA, 2010.