One-class classifiers ensemble based anomaly detection scheme for process control systems

Abstract

This paper focuses on the issue of anomaly detection for data in process control systems (PCSs). Considering data features in PCSs, this paper proposes to utilize the notion of one-class classification (OCC). In order to provide a general solution for more types of systems, ensemble learning is combined with OCC models. Two different ensembles of OCC models are proposed based on different scenarios in the process of detection. Performance of the proposed detection scheme is validated via several UCI datasets and two practical PCSs.

Keywords

Anomaly detection process control system one-class classification ensemble learning

Introduction

Process control systems (PCSs) are always playing vital roles in a plethora of industrial segments such as metallurgic, chemical, automotive, astronautic and instrumental (Shinskey, 1979). During past decades, several advanced control techniques, such as adaptive control, generalized predictive control and their corresponding evolving ones, have always being in an extraordinary effervescence. Roughly speaking, the success of these control methods mostly lies in the employment of data-driven models, which could make up the defect of mechanism models. Nevertheless, the quality of real-time process data collected from PCSs has rarely been taken into account when applying these data-driven techniques in realistic systems. Unfortunately, it turns out that anomalies mixing in process data that severely hinders direct implementations of these data-driven techniques.

In practice, there are two major sources resulting in anomalies in PCSs: (i) system disturbances– owing to uncertain factors and intricate operating conditions, values of system states are prone to deviating from normal pattern that can be described by state values under normal conditions; (ii) faulty measurements by sensors– sensors have been extensively utilized in PCSs owing to their microminiaturization and systematization during past decades. However, every sword has two sides; some measurements by sensors may also inevitably be contaminated owing to their own restrictions or harsh deployed environments. These anomalies could lead to significant system performance degradation or even catastrophic failure (Chen et al., 2014). Therefore, effective detection of anomalies is of high importance for reliability and safety of the overall PCS.

Motivation

In this paper, we focus on anomaly detection devised particularly for PCSs. Our objective is to develop a scheme possessing the following properties.

Real-time detection: Since data-driven models employed in PCSs need to be constructed online and even a single abnormal critical measurement may lead to a catastrophic cascade of failures throughout the whole system. Therefore, anomalies should be detected as early as possible to minimize the possibility of potential damage. To this end, an online scheme providing real-time anomaly detection is necessary. This scheme should be capable to identify the anomaly condition of each measurement, as soon as it is collected.

Dedicated to PCSs: Process data in PCSs has its particular features, which can be summarized as follows: (i) absence of data labels; (ii) atypical and imbalanced class distribution; and (iii) containing strong noise. As a result, anomaly detection methods implemented in PCSs should be designed elaborately in order to cope with these special issues.

General solution: For different PCSs, the system behaviors and dynamic could be very distinct. Hence, a general solution covering various PCSs is highly desirable. Those methods with unrealistic assumptions or models related to specific scenarios are not required.

Our approach

We propose a general anomaly detection method dedicated to PCSs based on one-class classifier ensemble. This method could achieve all the aforementioned properties. The contributions of this paper are summarized as follows:

This is the first time that one-class classifier ensemble is employed to detect anomalies in PCSs. We scientifically and systematically analyze the scenario for anomaly detection in PCSs, and propose to facilitate this problem by the means of one-class classifiers. In addition, we propose to construct ensemble models in order to develop a general solution for varying PCSs. According to ‘no free lunch theorem’, there is not a single classifier that is suitable for all tasks. Integrating different classifiers into one ensemble could make our method more competent.

Based on the particular characteristics of anomaly detection in PCSs. We propose an integrated scheme, in which three phases are included. At the initial training phase, we propose to construct a heterogonous ensemble model that performs well in the sense of small training set. Then a homogenous ensemble method, where a novel one-class classifier based on SVDD is proposed to enhance the robustness to noise, is constructed with more samples (including negative ones) are collected. Finally, we propose a simple adaptive strategy to make the detection model capture the new trend in the data.

When aggregating outputs of base leaners in the ensemble model, we employ a more sophisticated algorithm instead of voting method to further improve the performance of classification.

We carry out extensive experiments on several real-life datasets to investigate the performance of He-EOCC and Ho-EOCC in case of specific scenarios. Moreover, we apply the whole detection scheme in two real-life PCSs, namely electric arc furnace (EAF) and wind tunnel (WT). Comparisons are also conducted with several other methods.

Related work

In practice, anomaly/outlier detection has always been in an extraordinary effervescence in industrial area. Owing to continuous usage and normal wear and tear, industrial units are likely to suffer from various damages, which need to be detected as early as possible to prevent further escalation and losses (Chandola et al., 2009). Combining a novel fast and high quality segmentation algorithm (YASA) with a one-class support vector machine, (Martí et al., 2015) proposed an efficient anomaly detection method in turbomachines. Kiss et al., (2014) proposed a clustering based approach, in which various clustering techniques were explored to choose the most suitable for clustering the time-series data features, for detecting cyber-attacks that cause anomalies in networked critical infrastructures. Two types of anomalies are classified by (Zhang et al., 2014) in the energy system of steel industry. Accordingly, a dynamic time warping (DTW) based method combining with adaptive fuzzy C means (AFCM) is proposed for the trend anomaly and a k-nearest neighbor AFCM (KNN-AFCM) algorithm is designed for deviant anomalies. With the consideration of physics and information, (Zhou et al., 2015) comprehensively analyzed the multi-domain knowledge of field control layers in industrial process automation in order to construct multiple models. Then they proposed a novel multi-model based anomaly intrusion detection system with embedded intelligence and resilient coordination for the field control system in industrial process automation. Furthermore, an intelligent hidden Markov model is designed to differentiate the actual attacks from faults. (Zhang et al., 2016) proposed an angle-based subspace anomaly detection approach to high-dimensional data, where a meaningful feature subspace could be selected and anomaly detection could be conducted in this subspace projection by a normalized Mahalanobis distance. By taking flow information, application data and the packet order into account (Schuster et al., 2013) proposed a self-learning anomaly detection approach for industrial control systems (ICSs). Without proper customization for the special requirements of ICS network environment many existing anomaly detection stems will perform sub-optimally. Cateni et al., (2013) presented an application of fuzzy logic to the problem of outlier detection. Several traditional methods for outlier detection are combined with the aim to point out anomalous data due to different causes. This effectiveness of this automatic and self-adaptive method has been verified via a dataset coming from a real industrial context. Considering the characteristics of time series in PCSs, (Liu et al., 2012) proposed an improved RBF network to construct the model of controlled object and an Auto-Regression Hidden Markov Model was used to detect outliers according to corresponding fitting residuals.

Organization

The remainder of this paper is organized as follows. Section 2 introduces some fundamental knowledge concerning PCS, OCC and ensembles of one-class classifiers.

The design of our proposed method is presented in Section 3. The evaluation of our method is represented in Section 4, and we finally conclude this paper in Section 5.

Preliminaries

Here, we present the problem statement of our approach and the background information with regard to PCSs, OCC and ensembles of one-class classifiers.

PCSs

Our work focuses on PCSs that contain many variables, such as input and output variables of the whole system, input and output variables of controlled objects. Figure 1 presents the simplified block diagram of PCS. From this figure, we could discover both of two aforementioned sources of anomalies. One is system disturbances that usually occur at the controlled plant and the other is faulty measurements by sensors that usually occur at the feedback loop. In addition, Figure 2 and Figure 3 presents the general structure of self-tuning control and dynamic matrix control, respectively. As depicted in Figure 2, the online identification of process model needs the data that stems from real-time output of controller and output of the system. Similarly, the construction of forecasting model also needs real-time data that stems from system. As previously mentioned, such methods that contain data-driven models all assume that real-time data is totally ‘clean’. Unfortunately, such assume is impossible in practical applications. Therefore, detecting anomalies mixing in process data of PCSs is of much significance.

Figure 1.

General structure of PCS.

Figure 2.

General structure of Self-tuning control.

Figure 3.

General structure of dynamic matrix control.

OCC

In OCC, it is assumed that only information about one class, namely target class, is available. Therefore, the task of OCC is to define a boundary around the target class, such that it accepts as much of the target objects as possible, while it minimizes the opportunity of accepting outlier objects (Tax and Duin, 1999). Owing to this particular mechanism, OCC has become a solution to many real-life problems where data from a single class is abundant but counter-examples is hard even impossible to obtain, such as intrusion detection and machine fault diagnosis (Giacinto et al., 2008; Mahadevan and Shah, 2009).

Several traditional methods dedicated to solving OCC problem have been proposed and can be categorized into three classes according to their mechanisms.

Density-based methods: These methods are the most straightforward ones, which obtain a one-class classifier via estimating the density of the training data. For this type of method, in general, a distribution should be assumed beforehand, such as Gaussian distribution, and a test that is referred to as discordancy test, is then available to test new objects. Among the most popular density-based methods for OCC issue the Gaussian model, the mixture of Gaussian model and the Parzen density (PD) are especially prominent.

Reconstruction-based methods: This type of methods was originally introduced as a tool for data modeling. Via utilizing prior knowledge about the data and making assumptions about the generating process, a model is chosen and fitted to the data. And the use of this type of methods for OCC issue is based on the idea that outliers deviate from this model. Among reconstruction-based methods, K-means, self-organizing map (SOM), auto-encoder and PCA are the most popular ones.

Boundary-based methods: The main aim of this type of methods is to find the optimal size of volume enclosing given training points. Since these methods heavily rely on the distances between objects, they tend to be sensitive to scaling of the features. In addition, compared with the other two types of methods, since there is no need to estimate density or structure of the data, the required number of training examples is much smaller. Nearest neighbor (NN-d), one-class support vector machine (OCSVM) and support vector data description (SVDD) are the most popular ones among boundary-based methods.

In addition to these classical one-class classifiers, several evolving ones were also proposed for the sake of higher performance of classification. Mygdalis et al., (2016) proposed two extensions of the OCSVM and SVDD, where the generic graph structure that expresses the geometric data relationships of interest in optimization process is incorporated. Wu et al., (2016) also proposed an improved SVDD referring to as self-adaptive SVDD in which affinity clustering was combined with SVDD, whose parameters were adapted by an improved PSO algorithm. Juszczak et al., (2009) proposed a novel one-class classifier called minimum spanning tree class descriptor (MST_CD) that is built on the structure of the minimum spanning tree constructed on the target training set only. Empirical results have shown that MST_CD performs especially well in case of small sample size problems and in high-dimensional spaces.

Ensemble of one-class classifiers

Along with the ongoing advancement of ensemble learning, ensemble of one-class classifiers is becoming a significant research field. An aggregation of many simple but different classifiers can reduce the complexity and achieve better performance, which cannot be achieved by a single classifier. Classifier ensemble design aims to include mutually complementary individual classifiers that are characterized by high diversity and accuracy.

When designing an ensemble of one-class classifiers, two key issues should be taken into account meticulously. One is diversity enhancement and the other is decisions fusion.

For the first issue, there usually exist two categories, namely homogenous ensemble and heterogeneous ensemble, which can be distinguished according to their employed individual classifiers. In a homogeneous ensemble, all sub-classifiers are with the same base learner. Enhancement of ensemble diversity could be achieved via the manipulations of input data and this single base learner. (i) Partition the training dataset into several sample subsets. Bagging and Random Forest are two important representatives and they can also be developed for one-class classifier ensemble (Breiman, 1996, 2001). In addition, cluster algorithms are also employed to analyze the structure of training data and base learners are trained by these clusters. TOCC (Wang et al., 2006) and OCClustE (Krawczyk et al., 2013) are two representatives. (ii) Partition the feature space of training data into several feature subsets. Random Subspace (Ho, 1998) and Genetic Ensemble Feature Selection (Oliveira et al., 2006) are two representatives that can also be incorporated by ensembles of OCC models. (iii) Diversify parameters of the single base classifier. By diversifying parameters of the single base learner, several ‘different learners’ can be generated. But such a strategy has not been applied for ensembles of OCC models. While in heterogeneous ensemble, different base learners are used to train with the whole training set. The most popular heterogeneous ensemble is Stacking (Wolpert, 1992).

Referring to the second issue, the most popular solution should be majority voting scheme, which assigns the label of the class predicted by the highest number of classifiers from the pool to a new object. However, several works focused on the analytical expected error of majority voting have proven that its classification performance hold only when strong conditions are met so that it is not useful from practical point of view. Therefore, continuous outputs like support functions were proposed for each individual base learner. Support functions provide a score for the decision embodied by an individual classifier. Simple fusions of support functions, such as only supremum or the mean value, do not involve learning, hence improvements of their performance are limited. When fuser weight selection began to be treated as a specific learning process, several algorithms, such as linear function, mixture of Gaussian, perceptron and evolutionary algorithms, have been developed as models of the fuser. Recently, several heuristic search algorithms, like genetic algorithm (GA) (Krawczyk and Filipczuk, 2014) and firefly algorithm (FA) (Krawczyk, 2015), have been employed to select the classifier ensemble according to the fitness function defined by users. Their performances are promising regardless of their high computation complexity.

The proposed scheme

In this section, we introduce our proposed scheme dedicated to detecting anomalies in PCSs in detail. In contrast to other detection methods, our scheme incorporates two different methods, which would be used at two different phases of the whole detection process, based on the size of training set at the corresponding phase. Meanwhile, in order to achieve its adaptability with dynamic changes of stream data, we propose to update our detection method with a simple rule.

Outline of the whole scheme

As can be seen in Figure 4, the whole detection scheme consists of three main components.

Initial detection model: Throughout the whole detection process, all training sets are constituted by measurements collected on-line and thus detection would not be activated until enough examples have been obtained. Since we have no prior information about the data distribution, a heterogeneous ensemble of OCC model, which will be mentioned henceforth as He-EOCC, that integrates several one-class classifiers together is proposed to train the initial detection model. The reason why we use a heterogeneous ensemble herein is that all training examples will participate in the train of base learners so that the problem of under fitting can be mitigated in the case of small training set.

Core detection model: With ongoing collection of measurements, we develop the core detection model of our scheme with a homogenous ensemble of one-class classifiers, which will be mentioned henceforth as Ho-EOCC. Since more examples (including some labeled anomalies stored in Buffer 2) are used to train the detection model, the performance has achieved much improvement. Meanwhile, as the training set can be partitioned into several sample subsets by Ho-EOCC to training base learners, the issue of over fitting can be alleviated in case of large training set.

Model updating: Referring to the issue of model updating, usually two strategies, batch update and incremental update, can be utilized. Owing to the specificity of PCS applications, we chose batch update for our scheme. When a batch of new measurements has been stored in Buffer 1 (storing normal measurements), the oldest batch of examples should be removed from Buffer 1, and then the algorithm would use this new training set stored in Buffer 1 and Buffer 2 to retrain the model. Finally, this new model will replace the old one directly instead of comparing with previous model, which must be achieved through a validation set. This idea is based on an assumption that ‘everything is related to everything else, but closer things are more related than distant things’.

Figure 4.

General structure of the whole detection scheme.

He-EOCC

He-EOCC stands for a he terogeneous e nsemble of o ne- c lass c lassifiers, whose schematic diagram can be shown by Figure 5. As previously mentioned, two vital problems namely diversity enhancement and decisions fusion, should be carefully considered when designing an ensemble model. Based on the requirements of the scheme, He-EOCC resolves these two problems, as follows:

To enhance the diversity of the ensemble, we use four one-class classifiers, among which Parzen, SVDD and PCA are typical representatives of three types of OCC models mentioned beforehand. The last one is a graph-based method called MST-CD, which is suitable for small datasets. The reason why we choose these four classifiers is that they usually outperform other competitors of their own categories. This will enhance the final performance of the ensemble model. Since no more improvements have been done by us for these four methods, details of these methods are not provided in this paper.

In order to apply a more sophisticated fusion method, we use the method proposed in Tax and Duin (2001) to transform the output of distance-based one-class classifiers into a probability estimate. Then, we use an operator named EIOWA (Parhizkar and Abadi, 2015) to fuse the outputs of these four classifiers. Assume $x_{k}$ is a test sample, then EIOWA is defined as

E (β_{1} (x_{k}), . . ., β_{L} (x_{k})) = \sum_{j = 1}^{L} ω_{j} γ_{j} (x_{k})

(1)

where $β_{j} (x_{k}) = p_{j} (ω_{T} | x_{k})$ is the posterior probability of $x_{k}$ for the base classifier $C_{j} \in C$ and L is the number of based one-class classifiers in the ensemble. $γ_{1} (x_{k}), . . ., γ_{L} (x_{k})$ are simply $β_{1} (x_{k}), . . ., β_{L} (x_{k})$ reordered in descending order.

Figure 5.

The schematic diagram of He-EOCC.

Then the weights are calculated by the following exponential function

ω_{j} = \frac{g (j / L)}{\sum_{k = 1}^{L} g (k / L)}

(2)

where

g (z) = {\begin{matrix} (1 - z) e^{- λ z}, 0 \leq z \leq 1 \\ 0 otherwise \end{matrix}

(3)

Finally, the decision for $x_{k}$ can be written as

y (x_{k}) = {\begin{matrix} ω_{T}, E (β_{1} (x_{k}), . . ., β_{L} (x_{k})) \geq E (ξ_{1}, . . ., ξ_{L}) \\ ω_{O} otherwise \end{matrix}

(4)

where $ξ_{i}$ is the decision threshold for classifier $C_{j}$ .

Ho-EOCC

Ho-EOCC stands for a ho mogenous e nsemble of o ne- c lass c lassifiers, whose schematic diagram can be shown by Figure 6. In contrast to the scenario of initial training, training samples are extremely sufficient during the subsequent detection. Therefore, we use a homogenous ensemble to describe the training data. Similarly, we also have to solve two vital problems when designing Ho-EOCC.

Figure 6.

General structure of Ho-EOCC.

To enhance the diversity of the ensemble, we firstly use K-meansto partition the training set into several subsets (clusters), for each of which an OCC model is constructed. In order to enforce the robustness to noise in training set, we proposed a new method, which will be mentioned henceforth as K-SVDD, as the base learner in Ho-EOCC. In contrast to SVDD, each example in the training set will be assigned a weight, which stands for its membership to its cluster generated by K-means. For samples that far from the center of its cluster, their memberships should be smaller, and thus they will have less influence on the final boundary. Furthermore, anomalies that have been identified by former detectors (stored in Buffer 2) will participate in the training of subsequent model. Therefore, to construct a more closed and tighter boundary around the normal data, we maximize the margin between the hyper-sphere and the abnormal data apart from minimizing the volume of the hyper-sphere. The restructured optimization problem is expressed by formula (5).

\begin{matrix} min_{R, o, ξ_{i}, ξ_{j}} F = R^{2} - C ρ^{2} + C_{1} \sum_{i} ω^{N} (Φ (x_{i})) ξ_{i} \\ + C_{2} \sum_{j} ω^{O} (Φ (x_{j})) ξ_{j} \\ s . t . ‖ Φ (x_{i}) - o ‖^{2} \leq R^{2} + ξ_{i}, x_{i} \in S_{N} \\ ‖ Φ (x_{j}) - o ‖^{2} \geq R^{2} + ρ^{2} - ξ_{i}, x_{i} \in S_{O} \\ ξ_{i} \geq 0, ξ_{j} \geq 0, \forall i, j \end{matrix}

(5)

In the above formula, $β^{2}$ is the margin between the hyper-sphere and the abnormal data; $Φ$ is an implicit mapping that transforms training data into high-dimension feature space. $ω^{N}$ and $ω^{O}$ are memberships of normal examples and anomalies towards their corresponding clusters, respectively; $ξ_{i}$ and $ξ_{j}$ are slack variables for normal examples and anomalies, respectively; $C, C_{1}, C_{2}$ are three positive constants. $S_{N}$ and $S_{O}$ are sample sets of normal and outlier training examples, respectively.

By introducing Lagrange multipliers, we can obtain its dual problem

\begin{matrix} max_{α_{i}, α_{j}} \sum_{i = 1}^{n} α_{i} K (x_{i} \cdot x_{i}) - \sum_{i = 1}^{n} \sum_{j = 1}^{n} α_{i} α_{j} K (x_{i} \cdot x_{j}) \\ s . t . 0 \leq α_{i} \leq C_{1} ω^{N} (x_{i}), 1 \leq i \leq m_{1} \\ 0 \leq α_{j} \leq C_{2} ω^{O} (x_{j}), 1 \leq j \leq m_{2} \\ \sum_{i = 1}^{n} α_{i} = 1 \\ \sum_{j = 1}^{m_{2}} α_{i} = C \end{matrix}

(6)

Above, $m_{1}$ and $m_{2}$ are numbers of normal and abnormal training samples, respectively. And let $n = m_{1} + m_{2}$ is the size of the whole training samples.

After solving the dual problem, we obtain a set of Lagrange multipliers, from which we can calculate the center and radius of the hyper-sphere

o = \sum_{i = 1}^{n} α_{i} Φ (x_{i})

(7)

R^{2} = ‖ Φ (x_{k}) - o ‖^{2} = \sum_{x_{i} \in S_{N}} K (x_{i}, x_{i}) - 2 \sum_{i = 1}^{n} α_{k} K (x_{i}, x_{k}) + (o \cdot o)

(8)

where $x_{k}$ can be any support vector.

Fusion method used by Ho-EOCC is almost the same with that of He-EOCC. The only difference is the number of based classifiers in ensemble. In He-EOCC, this number is 4; while it will change with the number of clusters embodied by the training set in Ho-EOCC.

Model updating

In contrast to offline detection, we rarely have any information about the coming data. The only thing that we can do seems to capture the dynamics of data as soon as possible. Therefore, real-time updating for detection model is extremely necessary.

First time updating: As demonstrated by Figure 4, the first time updating can be activated when the number of normal samples stored in Buffer 1 reaches M (user define), which is sufficient for training Ho-EOCC.

Subsequent updating: When enough new measurements have been collected in Buffer 1, the oldest N samples should be removed from it. Then Ho-EOCC will be retrained with all the samples in Buffer 1 and Buffer 2. It is noteworthy that N is much smaller than M.

Note that when updating the detection model, we only retrain it and replace the previous one. The reason why comparison is not carried out like some methods is twofold: (i) the selection of suitable validation sets for online updating is not an easy task; (ii) the consideration of computation complexity.

Experimental analysis

In this section, we carry out extensive experiments to investigate the performance of our method. The objective can be summarized as three parts:

Investigate the significance of He-EOCC for initial detection model where small size of training set is available.

Investigate the significance of the proposed K-SVDD for the core detection model, which should have strong robustness to noise in training set.

Investigate the significance of whole proposed detection scheme for anomaly detection in PCSs, whose properties have been mentioned before.

Based on these objectives, we execute the following three experiments. In all experiments, the class that has the largest number of samples is regarded as target class, the rest samples from other classes are regarded as anomalies. Fifty percent of samples in target class are used as the train set, then all the rest of samples from both target class and outlier class are used as the test set. The mean values are recorded and compared.

Experiment A

UCI datasets (http://archive.ics.uci.edu/ml/) are used in this experiment and information about these dataset is listed in Table 1. Through these datasets, we compare He-EOCC with its four base one-class classifiers and 4 ‘Ho-EOCC’ whose base learner is not K-SVDD but Parzen, PCA, SVDD and MST-CD, respectively.

Table 1.

Details of datasets used in Experiment A. (‘#’ represents the number of the corresponding item).

No.	Dataset	# of instances	# of attributes
1	Biomed	194	5
2	Breast	699	9
3	Blood transfusion service (BTS)	748	5
4	Colic	368	22
5	Congressional voting records (CVR)	435	16

The metric used in this experiment is Matthews Correlation Coefficient (MCC), which was also employed by Liu et al., (2012) to alleviate the influence of imbalance ratio between the numbers of normal and abnormal samples on the performance of classifiers. Based on the Confusion matrix depicted in Table 2, MCC is expressed as the following formula

MCC = \frac{TP \cdot TN - FP \cdot FN}{\sqrt{(TP + FP) \cdot (TP + FN) \cdot (TN + FP) \cdot (TN + FN)}}

(9)

Table 2.

Confusion matrix.

		Actual label
		Target class	Negative class
Predicted label	Target class	True positive (TP)	False negative (FN)
Predicted label	Negative class	False positive (FP)	True negative (TN)

The value range of MCC is $[- 1, 1]$ and the higher MCC value indicates that the corresponding model performs better. Since few benchmark data sets are provided for one-class classifiers, some manipulations for these data sets are necessary. Class that has the largest number of samples is regarded as target class, the rest samples from other classes are regarded as anomalies.

Results are reported in Table 3. Furthermore, results of the Shaffer post-hoc test between the proposed He-EOCC and other methods are showed in Table 4. From Table 3, we can see that He-EOCC outperforms other competitors in the majority of cases. The average values indicate that He-EOCC is extremely suitable for datasets with small size. Furthermore, for single one-class classifiers, MST-CD outperforms all the others in majority of cases. It is interesting that performance of several homogenous ensembles is worse than its base classifier. This is mainly because that homogenous ensemble may be not suitable for training set with small size. Then, from results in Table 4 we can see that the difference between He-EOCC and other methods is statistically significant.

Table 3.

Comparison of He-EOCC and several competitive methods on datasets with small size (Pseudo Ho-EOCC represents the homogenous ensembles where the base learners are listed below).

Dataset	He-EOCC	Parzen	PCA	SVDD	MST-CD	Pseudo Ho-EOCC
						Parzen	PCA	SVDD	MST-CDD
Biomed	0.349	0.318	0.119	0.019	0.298	0.209	0.009	0.102	0.301
Breast	0.909	0.577	0.672	0.473	0.894	0.589	0.701	0.522	0.896
BTS	0.884	0.792	0.722	0.811	0.809	0.799	0.738	0.833	0.814
Colic	0.478	0.012	0.230	0.411	0.449	-0.009	0.249	0.422	0.461
CVR	0.590	0.141	0.502	0.422	0.521	-0.001	0.511	0.492	0.533

Table 4.

Shaffer test for comparison between He-EOCC and reference methods. ‘+’ represents that method on the left is superior.

Hypothesis	p-value
He-EOCC vs Parzen	+(0.0098)
He-EOCC vs PCA	+(0.0177)
He-EOCC vs SVDD	+(0.0127)
He-EOCC vs MST-CD	+(0.0253)
He-EOCC vs PHo-Parzen	+(0.0088)
He-EOCC vs PHo-PCA	+(0.0168)
He-EOCC vs PHo-SVDD	+(0.0133)
He-EOCC vs PHo-MST-CD	+(0.0259)

Experiment B

In this experiment, the size of training set is larger than that of Experiment A. Details of datasets used in this experiment are depicted in Table 5.

Table 5.

Details of datasets used in Experiment B. (‘#’ represents the number of the corresponding item).

No.	Dataset	# of instances	# of attributes
1	Banknote authentication (BA)	1372	5
2	Waveform	2000	6
3	Artificial characters (AC)	6000	7
4	Bach choral harmony (BCH)	5665	17
5	Combined cycle power plant (CCPP)	9568	4

To investigate the significance of Ho-EOCC for the core detection model and the significance of the proposed K-SVDD for Ho-EOCC, we compare our Ho-EOCC with He-EOCC and four pseudo Ho-EOCCs whose base learner are Parzen, PCA, SVDD and MST-CD, respectively.

Results with respect to MCC and Shaffer post-hoc test are depicted in Table 6 and Table 7, respectively. From results in Table 6, we can see that Ho-EOCC outperforms other competitors in majority of cases. The reason why the performance of He-EOCC is worse than that of Ho-EOCC is that base learners in He-EOCC may be over fitting, which leads to high false positive rate. When comparing results between Ho-EOCC with PHo-SVDD, we can discover the significance of the proposed K-SVDD for homogenous ensemble. Compare results of PHo-Parzen with that in Experiment A and we can also find that performance of Parzen has improved much. This is just the property of density-based method for OCC. Similarly, results from Table 7 also prove the significance of Ho-EOCC.

Table 6.

Comparison of Ho-EOCC and several competitive methods on datasets with large size (Pseudo Ho-EOCC represents the homogenous ensembles where the base learners are listed below).

Dataset	Ho-EOCC	He-EOCC	Pseudo Ho-EOCC
			Parzen	PCA	SVDD	MST-CDD
BA	0.882	0.871	0.812	0.822	0.848	0.841
Waveform	0.909	0.897	0.889	0.901	0.822	0.896
AC	0.884	0.792	0.799	0.738	0.833	0.814
BCH	0.928	0.932	0.901	0.892	0.900	0.899
CCPP	0.950	0.941	0.921	0.911	0.892	0.933

Table 7.

Shaffer test for comparison between Ho-EOCC and reference methods. (‘+’ represents that method on the left is superior).

Hypothesis	p-value
Ho-EOCC vs PHo-Parzen	+ (0.0198)
Ho-EOCC vs PHo - PCA	+ (0.0177)
Ho-EOCC vs PHo-SVDD	+ (0.0127)
Ho-EOCC vs PHo-MST-CD	+ (0.0133)
Ho-EOCC vs He-EOCC	+ (0.0259)

Experiment C

In this experiment, we apply the whole detection scheme (Hy-EOCC) in two real-life PCSs: electric arc furnaces (EAF) and wind tunnel (WT).

EAF: Since the temperature around the electric arc can reach 4000 ã, under which most alloys would melt, EAF has become the primary equipment in the production of advanced steel alloys and special steel alloys. Furthermore, as the main crude material of EAF steelmaking is from the discarding steel, which is also the source of pollution, which can be remitted greatly owing to the implementation of EAF in the steelmaking domain. Owing to the limitation of present techniques, however, the direct measurement of electric arc is impossible. Therefore, arc current and arc voltage are very significant variables in the analysis and control of EAF system.

WT: A wind tunnel system is used for testing scale models, mostly of airplanes, in the speed region of 0 to Mach 1.3. It is necessary to keep the Mach number constant at a predefined set point because most measured variables are a function of the Mach number, for given test conditions of stagnation pressure and temperature. In addition, the predictive controller for the Mach number has significantly better performance than the PID controller and the key issue in predictive control is the prediction of Mach number. Correspondingly, detecting the contaminative data in Mach number time series is a significant work, which may be ignored by most researchers. Distinct from EAF system, the accuracy requirement for Mach number is higher, which may be a challenge for the detection model. As stated by Wang et al., (2016) there are totally nine variables that have strong connection with the control of WT. Therefore, this nine-dimension vector is the detection objects of our scheme.

Table 8 summarizes the description of two data sets employed in this experiment. In this experiment, both data sets from two PCSs contains 10,000 instances. The first one hundred examples are used to train the initial detection model. When the number of collection reaches one thousand, the Ho-EOCC model starts to work. Also, the number N in the update phase is set as 1000.

Table 8.

Details of datasets used in Experiment C. (‘#’ represents the number of the corresponding item).

No.	Dataset	# of instances	# of attributes
1	EAF	10000	3
2	WT	10000	9

Limited by the related literature, we only compare our proposed detection scheme with that proposed by Liu et al., (2012) Detection rate (DT) and false alarm rate (FAR) are two metrics that evaluate these two competitive methods.

DR = \frac{TP}{TP + FN}

(10)

FAR = \frac{FP}{FP + TN}

(11)

Then, results with respect to these two metrics are shown in Table 9, from which we can see that detection rate and false alarm rate are compared between our scheme and the method proposed by Liu et al., (2012) (AR-HMM). Results indicate that our proposed scheme outperforms AR-HMM with respect to both metrics. The reasons are twofold: (i) two ensemble models proposed in our scheme are elaborately designed for different phases of detection, while such design is not available for AR-HMM; (ii) robustness to noise has been considered in our scheme and algorithm dedicated to solving it has been embedded in the scheme. Since strong noise is a prominent property of data in PCSs, such design is significant; (iii) the model updating rule proposed in our scheme makes the detector capture data dynamics earlier.

Table 9.

Comparison between the proposed scheme and the method proposed by Liu et al. (DR/FAR).

Dataset	Hy-EOCC	AR-HMM
EAF	97.96/1.96	86.02/3.92
WT	100/1.96	88.75/11.39

Conclusion and future work

This paper proposed an integrate scheme for anomaly detection in PCSs. Two one-class classifier ensembles are elaborately design according to the characteristics of data at different phases. The employment of the idea of OCC could solve the problem of counter-examples absence when training an anomaly detection method. The combination with ensemble learning could make the detecting scheme suitable for more types of PCSs. Each design has been evaluated through dedicated experiments with corresponding datasets. Performance of whole scheme was also evaluated via two real-life PCSs.

The results are promising and encourage further research on these topics. We anticipate improvements in our proposed method by enhancing diversity of ensemble models and using more smart fusion methods, for example, using a lightweight searching algorithm to prune the ensemble, using a trainable algorithm to learn the fuser. It also will be interesting to apply our scheme to more real-life PCSs to ascertain whether the idea of ensemble could provide a general solution.

Footnotes

Declaration of conflicting interest

The authors declare that there is no conflict of interest.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

References

Breiman

(1996) Bagging predictors. Machine Learning 24(1): 123–140.

Breiman

(2001) Random forests. Machine Learning 45(1): 5–32.

Cateni

Colla

Nastasi

(2013) A multivariate fuzzy system applied for outliers detection. Journal of Intelligent & Fuzzy Systems 24(4): 889–903.

Chandola

Banerjee

Kumar

(2009) Anomaly detection: A survey. Acm Computing Surveys 41(3): 1–58.

Chen

Yang

McCann

(2014) Distributed real-time anomaly detection in networked industrial sensing systems. IEEE Transactions on Industrial Electronics 62(6): 3832–3842.

Giacinto

Perdisci

Del Rio

Roli

(2008) Intrusion detection in computer networks by a modular ensemble of one-class classifiers. Information Fusion 9(1): 69–82.

(1998) The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis & Machine Intelligence 20(8): 832–844.

Juszczak

Tax

DMJ

Pekalska

Duin

RPW

(2009) Minimum spanning tree based one-class classifier. Neurocomputing 72(7): 1859–1869.

Kiss

Genge

Haller

Sebestyen

(2014) Data clustering-based anomaly detection in industrial control systems. In: IEEE International Conference on Intelligent Computer Communication and Processing, pp. 275–281.

10.

Krawczyk

(2015) One-class classifier ensemble pruning and weighting with firefly algorithm. Neurocomputing 150(150): 490–500.

11.

Krawczyk

Filipczuk

(2014) Cytological image analysis with firefly nuclei detection and hybrid one-class classification decomposition. Engineering Applications of Artificial Intelligence 31(1): 126–135.

12.

Krawczyk

Woźniak

Cyganek

(2013) Clustering-based ensembles for one-class classification. Information Sciences 264(6): 182–195.

13.

Liu

Mao

(2012) Outlier detection for process control data based on a non-linear Auto-Regression Hidden Markov Model method. Transactions of the Institute of Measurement & Control 34(5): 527–538.

14.

Mahadevan

Shah

(2009) Fault detection and diagnosis in process data using one-class support vector machines. Journal of Process Control 19(9): 1627–1639.

15.

Martí

Sanchezpi

Molina

Garcia

(2015) Anomaly detection based on sensor data in petroleum industry applications. Sensors 15(2): 2774–2797.

16.

Mygdalis

Iosifidis

Tefas

Pitas

(2016) Graph embedded one-class classifiers for media data classification. Pattern Recognition 60(C): 585–595.

17.

Oliveira

Morita

Sabourin

(2006) Feature selection for ensembles applied to handwriting recognition. International Journal on Document Analysis and Recognition (IJDAR), 8(4): 262–279.

18.

Parhizkar

Abadi

(2015) BeeOWA: A novel approach based on ABC algorithm and induced OWA operators for constructing one-class classifier ensembles. Neurocomputing 166: 367–381.

19.

Schuster

Paul

König

(2013) Towards Learning Normality for Anomaly Detection in Industrial Control Networks. Berlin Heidelberg: Springer.

20.

Shinskey

FGG

(1979) Process Control Systems: Application, Design and Tuning. New York: McGraw Hill.

21.

Tax

DMJ

Duin

RPW

(1999) Support vector domain description. Pattern Recognition Letters 20(11–13): 1191–1199.

22.

Tax

DMJ

Duin

RPW

(2001) Combining one-class classifiers. In: Proceedings of the Second International Workshop on Multiple Classifier Systems. London: Springer Verlag, pp. 299–308.

23.

Wang

Yeung

Tsang

(2006) Structured one-class classification. IEEE Transactions on Systems Man & Cybernetics Part B Cybernetics A Publication of the IEEE Systems Man & Cybernetics Society 36(6): 1283–1295.

24.

Wolpert

(1992) Original contribution: Stacked generalization. Neural Networks 5(2): 241–259.

25.

Liang

Ramiro

et al . (2016) Self-adaptive SVDD integrated with AP clustering for one-class classification. Pattern Recognition Letters 84: 232–238.

26.

Zhang

Lin

Karim

(2016) An angle-based subspace anomaly detection approach to high-dimensional data: With an application to industrial fault detection. Reliability Engineering & System Safety 142: 482–497.

27.

Zhao

Liu

Wang

Liu

(2014) Adaptive fuzzy clustering based anomaly data detection in energy system of steel industry. Information Sciences 259: 335–345.

28.

Zhou

Huang

Xiong

Yang

(2015) Design and analysis of multimodel-based anomaly intrusion detection systems in industrial process automation. IEEE Transactions on Systems Man & Cybernetics Systems 45(10): 1345–1360.