An active learning ensemble method for regression tasks

Abstract

Active learning is a typical approach for learning from both labeled and unlabeled examples aiming to build efficient and accurate predictive models at minimum expense under an expert’s guidance. Since there is a lack of labeled data in many scientific fields whilst, at the same time, the labeling cost of unlabeled data is typically high in terms of time and expenditure, active learning has grown rapidly over recent years with great success. This is reflected in various studies providing insights and analyzing several active learning methods, especially in the case of classification tasks, whereas, there is only a limited number of studies concerning the implementation of active learning methods for regression ones. Within this context, the present paper sets out to put forward a pool-based active learning regression algorithm employing the query by committee strategy to evaluate the informativeness of unlabeled examples. The experimental results on a plethora of benchmark datasets demonstrate the efficiency of the proposed method, since it prevails over the baseline active learning approach applying the random sampling strategy, as well as familiar supervised methods.

Keywords

Active learning regression pool-based sampling query by committee random forest support vector machines model trees

1. Introduction

Nowadays, a vast range of data is stored in databases and data warehouses on a daily basis. Finding interesting patterns within digitally stored data is the main concern of Data Mining (DM) by means of Machine Learning (ML) algorithms to mine the hidden knowledge [1]. Applying one or more ML algorithms to a training example set results into the generation of a learning model with the ultimate purpose of predicting the labels or the target values of new examples. Several ML settings have been used for that purpose in order to solve a variety of problems depending on the existence of labeled and unlabeled examples.

Learning from labeled examples is the main objective of Supervised Learning aiming to predict the output for future data by training a function $f$ [2]. Classification and regression are the two tasks which can be carried out using supervised methods. The former refers to the problem of learning a discrete function $f$ , while the latter to a real-valued one [3]. On the other hand, Unsupervised or Descriptive Learning is related to the case of learning only from unlabeled examples without there being any knowledge regarding the output classes [4]. In this case, the unlabeled input examples are grouped together according to interesting similarities or dissimilarities. In addition, two approaches, both dealing concurrently with learning from labeled and unlabeled data, can be detected: Semi-Supervised Learning (SSL) and Active Learning [5]. The lack of labeled data in many real-world applications and the simultaneous high labeling cost of unlabeled data in terms of time and expense, has given rise to the development of these approaches. The general concept is to train a model using a small training set of labeled examples, refine it with appropriately selected unlabeled ones depending upon the method used and retrain it with the view to optimizing its predictive performance [6]. The main difference of the two methods is the generation of a machine learning model, with (Active Learning) or without (SSL) human intervention [7].

Regarding the active learning approach, it is considered a fast growing and promising subfield of ML with emphasis on building efficient and accurate predictive models at minimum expense under an expert’s guidance [8]. Two main sub-branches of active learning are discernible, depending on the type of the predicted variable: active learning for classification problems and active learning for regression ones. While active learning has been broadly applied to both binary and multi-class classification problems in several scientific fields with quite satisfactory results, there are only a limited number of studies concerning the implementation of active learning methods for regression. This is reflected more clearly in various studies providing insights and analyzing several active learning classification methods, whereas, there are only a limited number of studies concerning the implementation of active learning methods for regression tasks.

In that connection, we make an effort towards developing a new active learning regression algorithm. The proposed algorithm depends on the pool-based sampling scenario for asking queries, employing at the same time the query by committee strategy to evaluate the informativeness of unlabeled examples, of which the most informative are labeled by a domain expert. The experimental results on a plethora of benchmark and artificially generated datasets demonstrate the efficiency of the proposed method, since it prevails over the baseline active learning approach employing the random sampling strategy, as well as familiar supervised methods.

The rest of this paper is structured as follows: The active learning framework is clearly delineated in Section 2 while defining the problem of active learning regression, while Section 3 reviews recent studies relating to several active learning methodologies for regression tasks. The proposed active learning regression algorithm is introduced and described in detail in Section 4, whilst the experiments carried out and their consequent results are presented and analyzed in Section 5. Finally, in Section 6 we comment on the results considering some thoughts for future work.

2. Active learning for regression problems

Active learning or Query learning is a typical example of learning from both labeled and unlabeled examples aiming to build highly effective predictive models by using the minimum number of labeled examples, while asking from a domain expert the labels from the most informative and uncertain to label unlabeled examples [8]. The focus of active learning is all about minimizing the querying process, regarding its total effort, along with the generalization error of the generated regression model [9]. The novelty of the active learning framework lies in the fact that the learning algorithm actively participates in knowledge acquisition having discretion in the matter of selecting the unlabeled data thereby reducing the labeling effort and cost [10]. To this end, several methods have been proposed for asking queries and a lot more for selecting the most informative examples for querying their labels from the expert (human or system).

Concerning the main scenarios in which active learners are able to pose queries, pool-based, stream-based and membership query synthesis are the most characteristic and predominantly used in classification tasks. The diversification of these scenarios lies in the different ways in which the unlabeled data are obtained. According to pool-based sampling, the most “valuable” unlabeled examples for label querying are selected from the entire pool $U$ of unlabeled examples, while in the case of a very large $U$ the unlabeled examples are drawn from a smaller sized subset [11]. Stream-based or Sequential sampling refers to the case in which an unlabeled example is selected from data streams and the active learner decides whether to request its label or throw it out according to a predefined informativeness measure [8]. Last but not least, according to the third scenario, the active learner may obtain the label of any unlabeled example of $U$ [12] including queries generated from scratch [8].

Regarding the query strategies applied for selecting the most informative instances from $U$ and subsequently querying their labels (these strategies are often referred to as query algorithms [13]), the most important ones are being outlined as follows:

Figure 1.

The active learning framework.

•

Query by Committee (QBC) is a frequently used query strategy based on a finite set of classifiers, hereinafter referred to as committee learners, each of whom is trained on a small pool $L$ of labeled examples. Then, an input unlabeled example is queried for labeling, selected on the basis of a specified disagreement criterion among committee learners, usually in accordance with the principle of maximal disagreement [13]. The training set $L$ is augmented by the selected input example which was labeled by the expert, the model is re-trained, and the process is repeated until some stopping criterion is satisfied.

•

Uncertainty sampling is another widely used but also simple scenario, according to which the active learner queries the labels for the most uncertain to label unlabeled examples. Several measures have been employed under this strategy, but the majority of them are not directly applicable to regression tasks, since there is no notion of the posterior probabilities of output classes for real-valued variables [14]. In this case, the posterior variance is used instead, querying the label of the unlabeled example with the largest posterior variance [15].

•

Random sampling is evidently considered as the baseline strategy for comparison to other active learning methods. Unlike the other query strategies, random sampling queries the label of randomly selected examples from the $U$ set. It appears that the randomness of selection of unlabeled examples for label querying in conjunction with the simplicity of the method produce very positive results in most cases [16], which apparently makes it one of the most noteworthy active learning strategies.

•

Expected Model Change (EMC) concerns a theoretic active learning strategy aiming to select the examples which entail the maximum change to the models’ parameters when new queried examples are added, where the model change is defined as the difference between the current model parameters and the updated ones [8].

•

Expected Error Reduction (EER) is another theoretic active learning strategy measuring how much the generalization error is likely to be decreased for each unlabeled example. In practical terms, the example implying the minimum expected error is queried for labeling, which, however, results to high computational cost, since the expected error must be calculated for each specific unlabeled example [8].

•

Variance Reduction is a frequently used query strategy for regression problems aiming to reduce the generalization error of a regression model given in Eq. (1) by minimizing the output models’ variance which corresponds to the third term of the Eq. (2):

$\displaystyle\textit{error}=\int_{x}E_{t}[{(\hat{y}-y)}^{2}|x]P(x)dx$ (1) $\displaystyle E_{t}[{(\hat{y}-y)}^{2}|x]=E[{(y-E[y|x])}^{2}]+{(E_{L}[\hat{y}]-% E[y|x])}^{2}+E_{L}[{(\hat{y}-E[\hat{y}])}^{2}]$ (2)

where $P(x)$ is the marginal distribution of $x$ , $E[.]$ and $E_{L}[.]$ denote the expectation over the conditional density $P(y|x)$ and the labeled example set respectively, $E_{t}[.]$ denotes the expectation over both, while the first two terms correspond to models’ noise and squared bias [17].

•

Bagging and Boosting, constitute two very powerful and effective ensemble methods for constructing a committee of classifiers [18].

Regression analysis is a statistical method for building a mathematical model that best fits a specified set of data. This model is generally referred to as “regression model” and is utilized for the prediction of the value of a real-valued output variable $y$ given the values of a finite set of independent variables $\{x_{i},i=1,2,\ldots,k\}$ . Regression is classified into simple regression and multivariate regression depending upon the number of independent variables.

On this basis, the active learning regression problem can be defined as follows. Given a small set of $l$ labeled examples $L=\{(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{l},y_{l})\}$ and a large set of $u$ unlabeled examples $U=\{x_{l+1},x_{l+2},\ldots,x_{l+u}\}$ with $x_{i}\in R^{N},i=1,2,\ldots,l+u$ and $y_{i}\in R$ , we want to build an initial regression model trained on $L$ , employ a predetermined query strategy for selecting the most informative instances from $U$ for labeling by a domain expert and retrain until some stopping criterion is met with a view to predicting the value of $y$ for any new example $x$ (Fig. 1).

3. Related work

A plethora of significant studies deals with the implementation of active learning methods for classification problems, contrary to regression which is clearly less studied as easily identified in the pertinent literature. Since there is no study reviewing the main active learning methodologies that have been developed for regression tasks, we intend to provide a brief but concise review for some of the most preeminent studies concerning the active learning methods for regression tasks and which are outlined and discussed below.

Early works on active learning for regression were introduced by Krogh and Vedelsby [19] and RayChaudhuri and Hamey [9], thus serving as a reference point for almost all subsequent studies. Krogh and Vedelsby were among the first to extend the QBC approach for classification problems to regression ones by proposing an ensemble of five Neural Networks (NNs) for real-valued functions. Initially, the networks were trained individually on the same set of one single labeled example, while adding the example with the largest ambiguity at a time. Ambiguity was used to measure the disagreement among committee members on an input example and is defined as the variance of the weighted ensemble output averaged over unlabeled data. Moreover, they proposed an efficient approach for finding the optimal ensemble weights from unlabeled examples, thus leading to better generalization error. RayChaudhuri and Hamey [9] proposed a similar active learning scheme using ten different NNs trained on random subsamples of a randomly sampled dataset consisting of ten labeled examples each one. Subsequently, the variance of 100 randomly selected unlabeled examples is calculated, the one example with the higher variance is added to the previous labeled subset and the process is repeated until the maximum model variance is less than a predefined threshold.

On the basis of the previous studies, Burbidge et al. [10] considered a committee-based active learning method for regression tasks. The variance minimization criterion was used for querying the labels of unlabeled examples with a view to minimizing the expected generalization error, which includes a term regarding learners’ variance. Concerning the learners, a committee of five linear regressors was applied with a second order polynomial kernel. Moreover, each learner was trained on different subsets of the labeled data, thereby setting out the disagreement among committee members and subsequently the samples with the maximum disagreement among themselves were collected. The proposed approach proved to be very powerful and effective enough, especially in the case of a small learner’s bias.

At the same time, Sugiyama and Rubens [20] proposed Ensemble Active Learning (EAL), an active learning framework for choosing the best model to minimize the generalization error. According to EAL, all the models are applied to all training input examples and the corresponding outputs are collected. The term model refers to the number and type of linearly independent functions and the optimization parameter of a linear regression model, while the generalization error is defined by the expected squared error for the test input examples. Finally, the generalization error is computed for each model and the final model is the one which minimizes it. The main advantage of this method is the over-fitting risk reduction of a single and usually randomly selected model.

More recently, a number of very rewarding studies have been carried out in the active learning field by applying Support Vector Machines (SVM) for regression. Ceperic et al. [21] presented sparse Multi-Kernel Active Learning Support Vector Regression (MK-ALSVR) consisting of two layers: a non-linear preprocessing layer and a linear one. The algorithm starts with one data example in the training set, while a Radial Basis Function (RBF) kernel is applied to the model that best fits the data by optimizing the model parameters. The difference between the model output and the entire training set is calculated, the next data example is selected, and the training dataset is gradually augmented with simultaneous incremental construction of the model using the same kernel, but with different parameters. Using multiple kernels (in this case it is achieved through different parameters), whilst constructing the model in an incremental process, guarantees its high performance (the RMSE measure is used) as confirmed by the experimental results.

In the same context, Demir and Bruzzone [22] presented a novel method based on $\varepsilon$ -insensitive Support Vector Regression (SVR) termed Multiple Criteria Active Learning (MCAL). MCAL is considered to successfully handle small sized and high-dimensional training sets; it requires less computational time and presents good generalization performance. Moreover, the method selects the most informative and representative unlabeled examples for labelling on the basis of three criteria in two successive steps based on a k-means clustering algorithm. In the first step, relevant examples are selected from the unlabeled pool (relevancy), while the most diverse examples (diversity) between them are selected in the second step from high density areas of the feature space (density).

Cai et al. [14] put forward an active learning framework for regression termed EMC Maximization (EMCM). According to this approach, the examples which result into the largest “changes” to the models’ parameters are selected (a linear regression model is used). In particular, for each unlabeled example the models’ parameters are updated using the gradient of the error with respect to each single training example. In this way, the unlabeled example with the largest expected change is chosen each time and the training set is gradually augmented. The current framework was applied for both linear and non-linear regression, using the non-parametric Gradient Boosting Decision Tree (GBDT) learner in the second case. In a more recent study [23] the EMCM method was extended to batch mode active learning, under which, the k most informative examples are selected in each query instead of a single one.

A stream-based active learning algorithm for linear regression was proposed by Sabato and Munos [24]. Initially, the dataset is partitioned by stratification into a finite set of subsets while the Ordinary Least Squares (OLS) approach is applied to each of them. It is important to note that the algorithm is regarded as a black box, meaning that theoretically any other algorithm could be used instead. Ultimately, a generalized median procedure is used for choosing one of the resulting predictors and the algorithm converges to the globally optimal oracle risk, under a feasibly undefined model.

Very recently, Riquelme et al. [25] studied the case of online active learning for optimal prediction of high-dimensional linear models based on a threshold sampling rule. From a total of $n$ unlabeled examples, the $k$ most informative examples which present a weighted norm above a predefined threshold are selected. Moreover, the high-dimensionality problem seems to be effectively tackled using Lasso estimators. They proved that using threshold instead of random sampling, the estimation performance is remarkably improved, since the expected mean and the variance of the squared errors is further reduced, as confirmed by several experiments on real world data.

In response to the two previous studies, Lughofer and Pratama [26] proposed an online stream-based active learning approach incorporating the uncertainty sampling strategy while using three efficient sample selection criteria regarding the uncertainty in model outputs and parameters, as well as the extrapolation degree in conjunction with the models’ non-linearity degree. In this way, the boundless online data streams are loaded in the form of sequential sub-samples which are automatically rejected or accepted for updating the model. The aforementioned criteria were applied together with “generalized evolving Takagi-Sugeno” fuzzy systems for handling non-linear problems. Compared to random sampling, the method achieved lower error rates using only a small percentage of the total number of data streams sub-samples.

It follows from the abovementioned studies that pool-based and stream-based sampling is predomi- nantly used for regression tasks. These scenarios have proven to be more flexible than membership query synthesis for both classification and regression tasks, since the former is usually problematic in the case of a human expert [8]. In addition, stream-based sampling can handle both offline and online data streams [27], while pool-based assumes that the pool of unlabeled examples is invariable. Moreover, stream-based sampling obtains better results if the distribution of input examples is uniform [8], however, is extensively affected by data distribution changes over time [27].

4. Proposed method

The proposed regression algorithm (ALERegression) is based upon the pool-based scenario for asking queries, since it is independent from the distribution of input examples [28] and it has been successfully applied to a wide range of active learning problems for regression tasks. The adopted sampling strategy is the QBC, where a distance criterion is introduced for discerning the examples that should be extracted from the unlabeled pool and provided afterwards for labeling by a domain expert.

Let $L^{0}$ denote the initial labeled example set, $U^{0}$ the initial unlabeled example set with $|L^{0}|\ll|U^{0}|$ and $C=\{\textit{reg}_{1},\textit{reg}_{2},\textit{reg}_{3}\}$ a set of three supervised regressors that constitute the committee under the selected sampling strategy, henceforth referred to as “active regressors”. The careful selection of the committee members has particularly significant impact on the effectiveness of the active learning method [29], since choosing less informative unlabeled instances for being labeled by the expert would induce slower learning rates. Moreover, using different regressors has a positive impact on reducing the generalization error of the regression model, while simultaneously increasing its predictive performance [19]. Therefore, we chose three representative and widely used regression methods, in particular Random Forest (RF), Support Vector Regression (SVR) and Model Trees (M5P). Instead of adopting the same combination of regressors for the final stage, where the building of regression models take place, we employ only the M5P algorithm which has been assigned with this task, leading to fast enough predictions even for multi-dimensional datasets, without this being at the expense of accuracy. On top of this, the M5P algorithm is in a position to obtain higher accuracy scores compared to other types of regression trees [30]. Some brief comments relating to the selected active regressors are mentioned below.

More specifically, the first active regressor is an M5P algorithm that stems from model trees. This kind of regression algorithm is based on the initial demonstration of Quinlan [30] modified in [31] and its main function is to construct tree-based piecewise linear models. Its modifications were made with the purpose of tackling appropriately at the same time both categorical and numerical variables and missing values while reducing the tree size, as well as avoiding continuing the recursive split when only few instances have remained in a node. Especially, the last property namely the reducing of tree size offers better comprehensibility and leads to predictive behaviors that seem compatible enough with several kinds of regression problems [32].

The second regressor is an SVR algorithm based on the well-known Support Vector Machines, which has been widely used for regression problems improving the generalization performance [21]. The initial purpose of SVR was to describe all the training instances through a suitable function that does not diverge from the corresponding real value more than a pre-defined quantity [33]. Several variants of SVR have been formulated regarding the exploited kernels in case of non-linear problems, the cost functions and the optimization process. In our case, Sequential Minimal Optimization (SMO) is preferred, being updated by corrections proposed in [34] regarding some inefficiencies that occurred during the operation of the original algorithm. As it concerns its pre-process function, any nominal feature is transformed suitably into the appropriate number of binary features, while a normalization process is applied before the main core of this algorithm takes place.

Finally, the third component of the constructed committee is a RF algorithm able to operate into regression mode [35]. Its main function is based on growing ensemble tree regressors built over random splits of the initial feature vector space. The selected instances for each induced tree are filtered through a bootstrap process, enabling their replacement back to the original pool of training instances for being exploited by any other tree regressor, after having been selected randomly. No pruning stage is applied and the outputs of all the created tree regressors are averaged so as to finalize the prediction of the whole algorithm. The great applicability of RF over several applications is reported in the literature, while one of the most powerful assets of RF is the unwrapping of useful relationships that may exist into provided features.

Having defined the active regressors of the committee and the regressor that is responsible for the final estimation of the ensemble method, a general description of the proposed process follows next. Each one of the active regressors is trained on the same initial labeled set ( $L^{0}$ ), which is a subset of the training set ( $D$ ) and its size is analogous to the labeled ratio factor ( $R$ ). The iterative procedure in accordance with which the augmentation of the collected labeled set’s cardinality takes places, is described below.

In each iteration, all active regressors are applied on the current unlabeled set $U^{\textit{iter}}$ (denoted initially as $U^{0}$ ) and a matrix ( $X_{pr}$ ) is produced consisting of the predictions for each unlabeled instance accordingly. We symbolize the index of the current iteration with the iter parameter. The dimensions of $X_{pr}$ should be $\textit{size}(U^{\textit{iter}})\times 3$ , where the function size returns the number of rows of the input matrix. In order to select the lowest confident predictions among $X_{pr}$ a simple, yet effective query measure ( $q_{m}$ ) is implemented. This measure attempts to rank the predictions of the three active regressors for each unlabeled instance by measuring the range between their predictions. Then, $X_{pr}$ is sorted in descending order using as key the computed range for each unlabeled instance, thus naming the measure Range of Predicted Outputs (RPO), whose formula is provided here:

$\displaystyle\textit{RPO}^{i}=\max(X_{pr}^{i})-\min(X_{pr}^{i}),i=1,2,3\ldots$ (3)

Subsequently, the most confusing unlabeled examples from current unlabeled set $U^{\textit{iter}}$ are queried for being assigned by the expert and are integrated into the initial training set $L^{0}$ . This is implemented by holding only the $T$ -top instances of $X_{pr}$ , where the $T$ (%) is a pre-selected parameter, defined as a percentage of the size of $L^{0}$ . The procedure is repeated until some stopping criterion is satisfied. In our case a maximum number of 10 iterations were defined along with a $T$ value equal to 10%. The final regression model is then built over the exported $L^{\textit{LastIter}}$ . The pseudo-code of the ALERregression algorithm is described below as Algorithm 1.

Algorithm 1. ALERegression
Input
C – Set of base Regressors (M5P, SMOreg, RF)
D – Initial Training Set
R – Ratio of labeled instances along D
$L^{0}$ – Initial labeled instances, $L^{0}\subseteq D$
$U^{0}$ – Initial unlabeled instances, $U^{0}\subseteq D$
T – Percentage of accepted instances per iteration
$q_{mc}$ – Prediction confidence evaluation measure
FReg – Final Regressor (M5P)
1. Initialization
a. Train C as base models on $L^{0}$ and set iter $=$ 0
b. Declare a procedure PRC implementing the computations:
I. Loop an input matrix and for each instance of the matrix compute $q_{m}\equiv$ RPO [Eq. (3)]
II. Sort the input matrix descending according to the computed rankings
2. Loop for a number of iterations (1 $\leqslant$ iter $\leqslant$ MaxIter) until stopping criteria are satisfied (iter of last iteration $\equiv$ LastIter)
a. Apply C members to $U^{\textit{iter}}$ and store them to $X_{pr}$ matrix
b. Utilize PRC to rank and sort $X_{pr}$
c. Select T*size ( $L^{0}$ ) most confusing unlabeled examples (located in the top of $X_{pr}$ ), for the current iteration, and store them in a matrix ( $X_{\textit{MCP}}$ )
d. Obtain the true estimations for $X_{\textit{MCP}}$ using human expertise
e. Remove $X_{\textit{MCP}}$ from $U^{\textit{iter}}$ and add them to $L^{\textit{iter}}$
f. Re-train C as base model on new enlarged $L^{\textit{iter}}$
g. Set iter $=$ iter $+$ 1
3. Output
Use FReg trained on $L^{\textit{LastIter}}$ to predict the labels of the unknown test cases.

5. Experimental procedure and results

In this section, a brief description of the conducted experiments is given, concerning the adopted settings that will clarify the discrimination of the presented comparisons. The experiments were conducted in two phases – one phase with experiments on benchmark datasets and the second one on artificial created ones – which are described below.

5.1 Experiments on benchmark datasets

At first, 25 datasets were mined from the publicly available UCI repository, describing problems stemming from various scientific fields. Hence, the assumption of a proposed algorithm (ALERegression) with powerful generalization ability is clearly more effective, rather than restricting our evaluated datasets to specific domains. Moreover, in order to obtain a clear enough insight into the predictive ability of the proposed algorithm, two different labeled ratio ( $R$ ) values are examined (20% and 30%). Therefore, each corresponding comparison includes five algorithms: the proposed one, its baseline rival – which is based on the random sampling strategy and employs the same regressor for predicting the target values with the proposed one – and the three corresponding supervised approaches that exploit the same initial instances with the previous two methods, but they do not augment their cardinality by any iterative procedure. It is obvious that any other regressor could participate in these comparisons, but following the principle of the previous section, our comparisons are towards verifying the suitability of our distance measure for extracting informative instances from the unlabeled pool based on decisions of the three specific active regressors that are already available during the construction of the proposed algorithm. Also, the selected regressors are very well known and robust models as derived from the relevant literature.

Furthermore, two other parameters that have to be defined are the ratio of the initially labeled instances ( $T$ ) and the number of the conducted iterations. In order to keep up with all the included datasets, independently of their size, these parameters were fixed equal to 10% and 10 iterations respectively. Under this circumstance, at the end of the 10th iteration the final training set would contain two equal populations of instances, one initially collected and another one labeled by the human expert after its gradual selection by the proposed algorithm. To clarify, the role of the human expert inside this learning chain is simulated by an ideal oracle that returns each time the accurate target value of any asked instance, a process close enough to a real-case scenario. The experimental procedure that is followed is slightly automated but still reveals much about the predictive power of the proposed algorithm.

Table 1
Correlation coefficient ( $R=$ 20%)

Datasets	ALERegression	RandomL (M5P)	Super (M5P)	Super (SMOreg)	Super (RF)
auto93	0.739	0.694	0.601	0.757	0.770
autoMpg	0.919	0.918	0.900	0.895	0.908
auto_price	0.886	0.883	0.863	0.833	0.888
baskball	0.591	0.568	0.517	0.496	0.427
bolts	0.569	0.569	0.569	0.625	0.718
breastTumor	0.202	0.202	0.159	0.135	0.139
cholesterol	0.135	0.157	0.123	0.117	0.104
cleveland	0.699	0.654	0.597	0.625	0.658
cloud	0.920	0.916	0.900	0.892	0.837
cpu	0.984	0.968	0.951	0.920	0.894
echoMonths	0.677	0.650	0.564	0.587	0.565
elusage	0.882	0.869	0.877	0.802	0.563
fishcatch	0.979	0.974	0.951	0.951	0.914
housing	0.896	0.879	0.847	0.822	0.882
hungarian	0.641	0.634	0.604	0.581	0.622
lowbwt	0.773	0.767	0.753	0.668	0.753
machine_cpu	0.923	0.881	0.866	0.893	0.891
mbagrade	0.384	0.270	0.199	0.353	0.111
pbc	0.575	0.561	0.499	0.456	0.516
pollution	0.621	0.591	0.339	0.514	0.572
pwLinear	0.932	0.934	0.892	0.824	0.837
sensory	0.345	0.344	0.280	0.243	0.323
servo	0.882	0.881	0.812	0.764	0.820
strike	0.492	0.473	0.448	0.515	0.470
triazines	0.348	0.335	0.231	0.254	0.363

Table 2

Correlation coefficient ( $R=$ 30%)

Datasets	ALERegression	RandomL (M5P)	Super (M5P)	Super (SMOreg)	Super (RF)
auto93	0.780	0.766	0.667	0.746	0.797
autoMpg	0.925	0.925	0.916	0.907	0.916
auto_price	0.915	0.898	0.873	0.858	0.907
baskball	0.600	0.574	0.531	0.531	0.467
bolts	0.851	0.827	0.722	0.684	0.743
breastTumor	0.244	0.227	0.171	0.147	0.142
cholesterol	0.146	0.158	0.151	0.136	0.108
cleveland	0.704	0.694	0.645	0.664	0.676
cloud	0.925	0.918	0.909	0.910	0.861
cpu	0.991	0.975	0.956	0.929	0.924
echoMonths	0.693	0.673	0.623	0.613	0.582
elusage	0.881	0.879	0.883	0.807	0.634
fishcatch	0.985	0.984	0.967	0.958	0.935
housing	0.913	0.886	0.868	0.831	0.897
hungarian	0.651	0.646	0.599	0.579	0.629
lowbwt	0.779	0.781	0.763	0.712	0.756
machine_cpu	0.934	0.911	0.882	0.904	0.907
mbagrade	0.388	0.291	0.244	0.372	0.066
pbc	0.592	0.586	0.528	0.496	0.541
pollution	0.678	0.616	0.426	0.612	0.602
pwLinear	0.941	0.941	0.928	0.843	0.874
sensory	0.396	0.391	0.338	0.280	0.368
servo	0.925	0.907	0.841	0.804	0.863
strike	0.526	0.499	0.473	0.534	0.483
triazines	0.453	0.431	0.291	0.274	0.430

Table 3

RMSE ( $R=$ 20%)

Datasets	ALERegression	RandomL (M5P)	Super (M5P)	Super (SMOreg)	Super (RF)
auto93	6.771	6.832	7.824	6.628	6.860
autoMpg	3.080	3.072	3.416	3.544	3.322
auto_price	2697.377	2827.909	3079.315	3491.012	2922.005
baskball	0.087	0.090	0.096	0.097	0.099
bolts	23.256	23.256	23.256	21.916	19.140
breastTumor	10.433	10.529	10.962	12.563	10.728
cholesterol	52.022	52.413	54.241	59.412	52.693
cleveland	0.883	0.934	1.006	1.001	0.936
cloud	0.398	0.405	0.429	0.462	0.631
cpu	32.456	45.403	59.977	67.793	78.105
echoMonths	11.518	12.139	13.530	13.783	13.214
elusage	12.001	12.677	12.841	15.523	20.459
fishcatch	74.568	81.913	116.774	120.620	150.702
housing	3.969	4.267	4.898	5.327	4.394
hungarian	0.370	0.374	0.392	0.435	0.378
lowbwt	469.469	469.443	492.412	583.167	492.548
machine_cpu	55.460	71.880	80.723	75.560	78.520
mbagrade	0.295	0.303	0.327	0.307	0.356
pbc	903.871	924.887	991.118	1076.656	951.519
pollution	48.031	54.766	75.613	72.627	54.319
pwLinear	1.590	1.596	2.025	2.606	2.696
sensory	0.781	0.789	0.831	0.896	0.782
servo	0.722	0.740	0.938	1.139	0.931
strike	458.574	476.085	493.665	457.915	487.855
triazines	0.152	0.157	0.171	0.186	0.147

Table 4

RMSE ( $R=$ 30%)

Datasets	ALERegression	RandomL (M5P)	Super (M5P)	Super (SMOreg)	Super (RF)
auto93	5.949	6.006	7.382	6.632	6.433
autoMpg	2.939	2.943	3.130	3.332	3.135
auto_price	2301.942	2562.589	2916.127	3209.651	2618.810
baskball	0.085	0.088	0.092	0.092	0.095
bolts	8.951	10.327	17.801	17.729	16.364
breastTumor	10.206	10.355	10.736	11.699	10.789
cholesterol	51.772	51.860	52.811	56.264	52.645
cleveland	0.874	0.884	0.948	0.934	0.915
cloud	0.374	0.384	0.408	0.424	0.566
cpu	23.662	40.166	53.001	61.851	68.316
echoMonths	11.223	11.565	12.516	12.975	12.695
elusage	11.826	11.810	12.056	14.586	18.671
fishcatch	63.672	62.251	91.852	111.376	127.868
housing	3.649	4.138	4.428	5.146	4.091
hungarian	0.363	0.366	0.390	0.433	0.373
lowbwt	456.789	455.179	478.681	532.070	482.685
machine_cpu	51.353	62.859	75.461	71.066	69.985
mbagrade	0.288	0.301	0.314	0.302	0.367
pbc	888.696	895.796	954.194	1012.051	930.444
pollution	45.384	51.712	61.266	54.315	51.135
pwLinear	1.494	1.496	1.667	2.422	2.374
sensory	0.758	0.762	0.790	0.839	0.766
servo	0.592	0.650	0.842	1.033	0.819
strike	451.008	461.767	476.922	449.663	487.476
triazines	0.140	0.143	0.161	0.178	0.141

For determining the effectiveness of the algorithms used in the experiments, two widely used me- trics were employed: The correlation coefficient (CorCoef) and the Root Mean Square Error (RMSE). The produced results per labeled ratio are presented in Tables 1–4, created using the 10-cross-validation resampling technique, while the higher value per each dataset is bold highlighted. In general, it is observed that the majority of the datasets favor the performance of the proposed algorithm per each scenario. Another useful conclusion is the improvement that the RandomL (M5P) method scores in all cases compared to the supervised regressors, thus demonstrating the benefits of the active learning approach. However, the far better results achieved using a targeted query (RPO) and computed on the presence of the formatted committee, denote clearly the need for more oriented queries in order to better predict the target value, spending much computational resources and employing human expertise for acquiring true values, or at least values that represent the knowledge of the related human factor.

In order to compare the performance of the used methods, the Friedman Aligned Ranks non-parametric test [36] and the Hochberg post-hoc test [37] were applied (with a significance level $\alpha=$ 0.05). In the following Tables 5 and 6, the first column refers to the examined algorithms, the second column presents the Friedman test results and the third one contains the adjusted $p$ -values based on Hockberg’s post-hoc test. According to the Friedman’s test, the algorithms are ordered from the best performer (lowest ranking value) to the lowest one (highest ranking value). It has been noticed that the proposed algorithm outperforms all the other methods in both scenarios, indicating its effectiveness. Moreover, to detect the existing differences among the examined algorithms, the Hochberg post-hoc test was applied. The null hypothesis of equivalence of the medians of the algorithms is rejected in all the cases, as it is depicted in the last column.

Table 5

Statistical comparison ( $R=$ 20% and $\alpha=$ 0.05)

Algorithm	Friedman ranking	Hockberg post hoc test
		Adjusted $p$ -value	Null hypothesis
ALERegression	1.42	–	–
RandomL (M5P)	2.42	0.050000	Rejected
Supervised (M5P)	3.34	0.025000	Rejected
Supervised (SMOreg)	3.90	0.016667	Rejected
Supervised (RF)	3.92	0.012500	Rejected

Table 6

Statistical comparison ( $R=$ 30% and $\alpha=$ 0.05)

Algorithm	Friedman ranking	Hockberg post hoc test
		Adjusted $p$ -value	Null hypothesis
ALERegression	1.28	–	–
RandomL (M5P)	2.12	0.050000	Rejected
Supervised (M5P)	3.72	0.025000	Rejected
Supervised (SMOreg)	3.74	0.016667	Rejected
Supervised (RF)	4.14	0.012500	Rejected

Also, the contribution of the three base learners on the selection of the most confusing examples was indirectly analyzed considering the distribution of the middle model in the RPO measure while computing Eq. (3). The distribution of the middle model was extracted while evaluating 10 of the best benchmark performing datasets for the 2 labeled ratios (20% and 30%) that were examined above.

Figure 2.

The distribution of base regressors as middle model in the two scenarios ( $R=$ 20% and $R=$ 30%).

As can be concluded from Fig. 2, there is a strong alternation of the middle model and therefore between the models that make up the min and max predictions in RPO, thus suggesting that all three models are contributing to selection of the unlabeled data.

Finally, the corresponding supervised regression algorithms using the participants of committee as base regressors built on the full training set ( $R=$ 100%) is tested against the proposed ( $R=$ 30%). The produced results (Table 7), reveal that the proposed algorithm in this case outperforms RF and SMOreg, and its average prediction behavior is slightly worse than the corresponding M5P supervised regressor.

5.2 Experiments on artificial data

In addition, we experimented on artificially created data to gain a better insight into strengths and weaknesses of the proposed approach. Therefore, we investigated the influence of noise intensity and extreme outliers.

Table 7
Comparing correlation coefficient of proposed algorithm against supervised regressors on whole training set

Datasets	ALERegression ( $R=$ 30%)	Super (M5P) ( $R=$ 100%)	Super (SMOreg) ( $R=$ 100%)	Super (RF) ( $R=$ 100%)
auto93	0.780	0.848	0.817	0.879
autoMpg	0.925	0.937	0.922	0.932
auto_price	0.915	0.926	0.898	0.949
baskball	0.600	0.617	0.590	0.526
bolts	0.851	0.922	0.896	0.847
breastTumor	0.244	0.290	0.276	0.140
cholesterol	0.146	0.182	0.157	0.135
cleveland	0.704	0.711	0.707	0.702
cloud	0.925	0.925	0.929	0.909
cpu	0.991	0.986	0.968	0.985
echoMonths	0.693	0.702	0.679	0.637
elusage	0.881	0.884	0.839	0.831
fishcatch	0.985	0.988	0.970	0.984
housing	0.913	0.913	0.843	0.941
hungarian	0.651	0.678	0.579	0.651
lowbwt	0.779	0.792	0.767	0.768
machine_cpu	0.934	0.922	0.919	0.957
mbagrade	0.388	0.437	0.442	0.158
pbc	0.592	0.603	0.579	0.573
pollution	0.678	0.716	0.742	0.758
pwLinear	0.941	0.945	0.865	0.917
sensory	0.396	0.477	0.347	0.502
servo	0.925	0.935	0.836	0.930
strike	0.526	0.532	0.559	0.494
triazines	0.453	0.531	0.390	0.593

Table 8

Artificial dataset initial parameters

Parameter	Value
Sample size	100
Num. of features	20
Num. of informative features	8
Noise	0
Percentage of outliers	0

Figure 3.

The RMSE values regarding the gaussian noise applied (0.10–0.40) for $R=$ 20% and $R=$ 30%.

Figure 4.

The RMSE values regarding the percentage of outliers (4%–10%) for $R=$ 20% and $R=$ 30%.

Initially, we generated an artificial dataset utilizing a random regression problem [38]. To further complicate the relation between the response and the independent variables, we applied the cube root function on the latter. For the generated dataset we considered the following parameters: the number of samples, the number of features, the number of informative features (i.e. the number of features applied to build the linear model used to produce the output), the noise intensity and the percentage of outliers. These parameters were set as presented in Table 8.

Subsequently, we injected gaussian noise to the dataset, with a standard deviation value ranging from 0.10 to 0.40. The obtained RMSE results for $R=$ 20% and $R=$ 30% are illustrated in Fig. 3. It is therefore observed that ALERegression achieves substantially better results compared to all other methods in both scenarios.

Moreover, we investigated how the presence of outliers in the dataset affects the performance of the proposed algorithm. Therefore, a percentage of outliers, ranging from 4% to 10%, was added in the dataset. The obtained RMSE results for $R=$ 20% and $R=$ 30% are illustrated in Fig. 4. As the related figures suggest, the integration of RF as a base regression favors ALERegression, since they obtain similar results regardless of the percentage of outliers presented in the dataset.

6. Conclusions

Before we proceed to the conclusory remarks, some promising points are made here. Therefore, two attractive improvements that could be implemented inside the learning kernel of the proposed algorithm could be the integration of Local Weighted approaches per regressor, as it was proposed for SMOreg in [39], or apply dynamic selection methods for choosing accurate enough regressors inside ensemble schemes or improving Committee’s participants selection per dataset [40]. Another one perspective that should be considered is the demanded time resources, especially for vast amounts of data. Combination of clustering methods during pre-process steps has been employed in regression tasks and could be proven highly beneficial in case of AL schemes which are characterized by iterative structure. Feature engineering concepts should also be examined in the case that more sophisticated base regressors are going to be combined into different QBC approaches [41].

In a future work, we will also investigate the introduction of multiple query measures for the ranking of the unlabeled examples. New query measures can be relevant to the RPO method, which was introduced in the present work, like modifying RPO to ignore outlier predictions from the computed range, or completely different approaches than the range metric used. Another technical characteristic of the proposed algorithm is that its iterative nature in combination with the deployment of multiple learners renders the algorithm suitable for parallel execution using multi-threading and GPUs. Extending the structures and the procedures of the algorithm to support parallel execution may also be a vital feature. Use of incremental learners, such as those in case of [42], where a density criterion is used for detecting the most appropriate samples, could also offer time affordable approaches.

Another interesting topic that could extend this work derives from the combination of the different learning approaches and multiple learners. The first idea involves the combination of active learning [43] and semi-supervised regression [44] methods. The proposed algorithm, as an iterative method, can take advantage of schemes like SSL self-training to further enrich the labeled set in conjunction with the AL approach that is already embedded, thus reducing the need for labeled instances, which is considered as an expensive factor due to the need for human expertise in the process of labeling. The second one incorporates the field of deep learning [45]. To further increase the learning capacity of the proposed algorithm, convolutional neural networks can be embedded in the proposed algorithm if the appropriate pre-process procedures are implemented. Such an extension can make the algorithm suitable for use with big data, thus tackling a wide range of important real-world problems like computer vision and speech recognition.

In summary, this work presented a new wrapper algorithm for regression problems utilizing multiple learners and active learning techniques. A plethora of experiments were conducted for evaluating the efficacy of the proposed algorithm in a wide range of datasets against well-known learners using supervised and active methods in different labeled ratios. The experimental results on a plethora of benchmark datasets demonstrate the efficiency of the proposed method, since it prevails over the baseline active learning approach employing the random sampling strategy, as well as familiar supervised methods, as statistically confirmed by the Friedman Aligned Ranks nonparametric test and the Hockberg post-hoc test. To further promote the use of the algorithm, a WEKA package containing the wrapper method has been developed, while more details can be found in the Appendix section.

Footnotes

Appendix

The implementation of the proposed algorithm (ALERegression) was developed as a software package for the WEKA data mining tool, since WEKA is one of the most widespread and easy to use tools in the machine learning community. The package includes the java source code along with the precompiled package. The ALERegression package can be downloaded at the following link: http://ml.upatras. gr/aleregression/.

References

Roiger

R.J.

, Data mining: a tutorial-based primer, CRC Press, 2017.

Flach

, Machine learning: the art and science of algorithms that make sense of data, Cambridge University Press, 2012.

Ngo-Ye

T.L.

Sinha

A.P.

and Sen

, Predicting the helpfulness of online reviews using a scripts-enriched text regression model, Expert Syst. Appl. 71 (2017), 98–110.

Alpaydin

, Introduction to machine learning, MIT Press, 2014.

Wang

and Zhou

Z.-H.

, On multi-view active learning and the combination with semi-supervised learning, in: Proceedings of the 25th International Conference on Machine Learning, 2008, pp. 1152–1159.

Zhou

et al., An active learning radial basis function modeling method based on self-organization maps for simulation-based design problems, Knowledge-Based Syst. 131 (2017), 10–27.

Zhou

Z.H.

, A brief introduction to weakly supervised learning, National Science Review, 2018.

Settles

, Active learning, Synth. Lect. Artif. Intell. Mach. Learn. 6(1) (2012), 1–114.

RayChaudhuri

and Hamey

L.G.C.

, Minimisation of data collection by active learning, in: Neural Networks, 1995. Proceedings., IEEE International Conference on, Vol. 3, 1995, pp. 1338–1341.

10.

Burbidge

Rowland

J.J.

and King

R.D.

, Active learning for regression based on query by committee, in: International Conference on Intelligent Data Engineering and Automated Learning, 2007, pp. 209–218.

11.

McCallumzy

A.K.

and Nigamy

, Employing EM and pool-based active learning for text classification, in: Proc. International Conference on Machine Learning (ICML), 1998, pp. 359–367.

12.

Angluin

, Queries and concept learning, Mach. Learn. 2(4) (1988), 319–342.

13.

Seung

H.S.

Opper

Sompolinsky

, Q. by Committee and others, Computational Learning Theory, Google Scholar, 1992.

14.

Cai

Zhang

and Zhang

, Batch mode active learning for regression with expected model change, IEEE Trans. Neural Networks Learn. Syst. 28(7) (2017), 1668–1681.

15.

Son

and Lee

, Active learning using transductive sparse Bayesian regression, Inf. Sci. (Ny). 374 (2016), 240–254.

16.

Ramirez-Loaiza

M.E.

Sharma

Kumar

and Bilgic

, Active learning: An empirical study of common baselines, Data Min. Knowl. Discov. 31(2) (2017), 287–313.

17.

Geman

Bienenstock

and Doursat

, Neural networks and the bias/variance dilemma, Neural Comput. 4(1) (1992), 1–58.

18.

Mamitsuka

N.A.H.

and others, Query learning strategies using boosting and bagging, in: Machine Learning: Proceedings of the Fifteenth International Conference (ICML’98), Vol. 1, 1998.

19.

Krogh

and Vedelsby

, Neural Network Ensembles, Cross Validation, and Active Learning, in: Advances in Neural Information Processing Systems, Vol. 7, 1995.

20.

Sugiyama

and Rubens

, Active learning with model selection in linear regression, in: Proceedings of the 2008 SIAM International Conference on Data Mining, 2008, pp. 518–529.

21.

Ceperic

Gielen

and Baric

, Sparse multikernel support vector regression machines trained by active learning, Expert Syst. Appl. 39(12) (2012), 11029–11035.

22.

Demir

and Bruzzone

, A multiple criteria active learning method for support vector regression, Pattern Recognit. 47(7) (2014), 2558–2567.

23.

Cai

Zhang

and Zhou

, Maximizing expected model change for active learning in regression, in: Data Mining (ICDM), 2013 IEEE 13th International Conference on, 2013, pp. 51–60.

24.

Sabato

and Munos

, Active regression by stratification, in: Advances in Neural Information Processing Systems, 2014, pp. 469–477.

25.

Riquelme

Johari

and Zhang

, Online Active Linear Regression via Thresholding., in: AAAI, 2017, pp. 2506–2512.

26.

Lughofer

and Pratama

, Online active learning in data stream regression using uncertainty sampling based on evolving generalized fuzzy models, IEEE Trans. fuzzy Syst. 26(1) (2018), 292–309.

27.

Jiang

and Gruenwald

, Research issues in data stream association rule mining, ACM Sigmod Rec. 35(1) (2006), 14–19.

28.

Sugiyama

and Nakajima

, Pool-based agnostic experiment design in linear regression, in: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2008, pp. 406–422.

29.

Pereira-Santos

Prudêncio

R.B.C.

and de Carvalho

A.C.

, Empirical investigation of active learning strategies, Neurocomputing, 2017.

30.

Quinlan

J.R.

, Learning with continuous classes, Mach. Learn. 92 (1992), 343–348.

31.

Wang

and Witten

I.H.

, Inducing model trees for continuous classes, in: Proceedings of the Ninth European Conference on Machine Learning, 1997, pp. 128–137.

32.

Behnood

Olek

and Glinicki

M.A.

, Predicting modulus elasticity of recycled aggregate concrete using M5’ model tree algorithm, Constr. Build. Mater., 2015.

33.

Smola

A.J.

and Schölkopf

, A tutorial on support vector regression, Statistics and Computing, 2004.

34.

Shevade

S.K.

Keerthi

S.S.

Bhattacharyya

and Murthy

K.R.K.

, Improvements to the SMO algorithm for SVM regression, IEEE Trans. Neural Networks 11(5) (2000), 1188–1193.

35.

Breiman

, Random forests, Mach. Learn. 45(1) (2001), 5–32.

36.

Hodges

J.L.

Lehmann

E.L.

and others, Rank methods for combination of independent experiments in analysis of variance, Ann. Math. Stat. 33(2) (1962), 482–497.

37.

Hochberg

, A sharper Bonferroni procedure for multiple tests of significance, Biometrika 75(4) (1988), 800–802.

38.

Pedregosa

et al., Scikit-learn: Machine learning in python, J. Mach. Learn. Res. 12(Oct) (2011), 2825–2830.

39.

and Jiang

, Using Locally Weighted Learning to Improve SMOreg for Regression, PRICAI 2006 Trends Artif. Intell., 2006, 375–384.

40.

Dias

and Windeatt

, Dynamic ensemble selection and instantaneous pruning for regression, in: ESANN, 2014.

41.

Bolón-Canedo

and Alonso-Betanzos

, Recent Advances in Ensembles for Feature Selection, Springer, 2018.

42.

Yin

Wang

and Fan

, Active learning based support vector data description method for robust novelty detection, Knowledge-Based Syst. 153 (2018), 40–52.

43.

Reitmaier

Calma

and Sick

, Semi-Supervised Active Learning for Support Vector Machines: A Novel Approach that Exploits Structure Information in Data, arXiv Prepr. arXiv1610.03995, 2016.

44.

Kostopoulos

Karlos

Kotsiantis

and Ragos

, Semi-supervised regression: A recent review, J. Intell. Fuzzy Syst. 35(2) (2018), 1483–1500.

45.

Yang

Drake

Damianou

and Maarek

, Leveraging Crowdsourcing Data For Deep Active Learning An Application: Learning Intents in Alexa, in: Proceedings of the 2018 World Wide Web Conference on World Wide Web, 2018, pp. 23–32.

An active learning ensemble method for regression tasks

Abstract

Keywords

1. Introduction

2. Active learning for regression problems

4. Proposed method

5.1 Experiments on benchmark datasets

Table 1 Correlation coefficient ( R = 20%)

Table 7 Comparing correlation coefficient of proposed algorithm against supervised regressors on whole training set

Footnotes

Appendix

References

Table 1
Correlation coefficient ( $R=$ 20%)

Table 7
Comparing correlation coefficient of proposed algorithm against supervised regressors on whole training set