Automatic Classification of Pavement Type and Service Age Benchmarked with Standard Texture Databases Using the Machine Learning Method: A Pilot Study

Abstract

Pavement intelligent management systems have attracted considerable interest from researchers. However, various service conditions of pavement surface concerning the pavement type, texture service age, and so forth, inhibit a universal algorithm that is feasible for all cases. In this regard, the automatic classification of pavement type and service age is an essential premise to unblock the bottleneck stated above. Based on the surface texture data, a pilot study of the automatic classification approach to identify pavement surface textures using convolutional neural networks (CNNs) is presented. For comparison, the efficiency of the support vector machine (SVM) is also investigated. In total, three cases, (i) pavement types, (ii) texture service ages, and (iii) a combination of (i) and (ii), are involved in the automatic classification. The results indicate that the CNN outperforms the SVM, and the CNN models show a favorable classification accuracy for the above three cases with 93.0%, 81.1%, and 83.8%, respectively. In conclusion, the CNN demonstrates a high capability in expressing the pavement texture features and achieves satisfactory identification results for pavement surface types, but is inferior for texture service age. It is promising that the presented results could serve as a foundational exploration in the automatic identification of texture service conditions benchmarked with standard texture databases to facilitate pavement management systems.

Keywords

surface texture convolutional neural networks support vector machine automatic classification service conditions

Because of the increase of the heavy-duty traffic volume, many in-service pavements suffer from premature distresses, which might induce irreversible pavement deterioration until the failure of pavement performance. Therefore, appropriate pavement health monitoring is essential for maintenance and rehabilitation decisions ( 1 ). However, the existing methods to monitor pavement health are time- and labor-consuming ( 2 ). In this regard, artificial intelligence technology that can mimic human intelligence to perform tasks via data-driven approaches has been introduced to modify the whole process ( 3 ).

An ideal pavement management system should cover data collection and archiving, distress identification, distress condition assessment, pavement performance evaluation and prediction, and consequent maintenance decision-making ( 4 ). Data collection currently implements pavement inspection devices widely known as vehicles equipped with various sensors ( 5 – 7 ). Meanwhile, numerous studies have been carried out concerning distress detection and segmentation ( 8 – 12 ), damage assessment, and so forth ( 13 – 15 ). Evaluation and prediction methods of pavement performance concerning skid resistance ( 16 ), noise pollution ( 17 ), water drainage ( 18 ), and so forth based on the collected data have also been investigated comprehensively. With respect to data archiving of pavement service conditions, Pereira et al. ( 19 ) and Riid et al. ( 20 ) developed automatic classification models to identify whether the road is paved based on pavement image data. However, few studies have focused on the identification of pavement surface service conditions with respect to pavement types and texture service ages.

Undoubtedly, the service conditions of pavement surface textures serve as essential background data that aid in the diagnosis of distress and the maintenance decision-making procedure. On the one hand, the proper adoption of the algorithms for identifying distresses is constrained by an unclear recognition of pavement type information ( 21 – 23 ). On the other hand, the texture service age is also one of the critical consideration indexes for the decision-making of the pavement maintenance strategy ( 24 – 26 ). Admittedly, pavement service conditions can be obtained from related road construction records. However, in-service pavements, especially urban roads, will have undergone varying degrees of maintenance, making the distribution of pavement information complex. The manual comparison approach to determine the pavement information for a large pavement system is time-consuming and of questionable accuracy. To address this issue, this study designed a viable methodology for the automatic classification of service conditions of the pavement surface texture.

With the development of sensor technology, three-dimensional (3D) scanner technology has become a mainstream technology for collecting high-resolution 3D pavement surface data ( 27 ). Various achievements for crack detection ( 28 ), skid resistance evaluation ( 29 ), surface texture depth calculation ( 30 ), and so forth, have been achieved based on the pavement surface texture. Enlightened by previous studies, the 3D pavement surface data are selected as the database for classification. Unlike pavement surface image data, 3D point cloud data retain all height information of pavement surface textures, which are critical reflections of the texture service ages. However, the pavement surface texture has highly random distribution, making the classification procedure difficult and challenging.

Recently, the convolutional neural network (CNN), a branch of machine learning, has been successfully applied in image recognition ( 31 ), object detection, and segmentation tasks ( 32 , 33 ), and demonstrates a powerful capability to cultivate highly nonlinear representations of big data with strong robustness. Therefore, the CNN is incorporated into the automatic classification of service conditions of the pavement surface texture. The 3D pavement texture data was first collected from pavements with various pavement types and service ages. Correspondingly, different CNN classification models with respect to the pavement type, texture service ages, and combination of the first two were trained and tested. By comparing with the traditional machine learning method, the support vector machine (SVM), the effectiveness and advantages of the CNN models were further concluded.

Introduction to the Dataset

Data Collection

It should be mentioned that, to our best knowledge, this work is the first attempt to utilize CNNs to deal with the automatic classification of the service conditions of pavement surface textures. Therefore, there is no off-the-shelf dataset available for this case. Therefore, a dataset under the categorization criterion of texture serving conditions according to pavement types and service ages was built. The texture data were collected from five standard pavements, namely AC-13, AC-16, AC-20, emulsified asphalt mixture (EAM), and cement concrete (CC). For service ages, four condition states for AC-13 with service ages of one year, one and a half years, five years, and eight years, respectively, were collected. There were 256 samples collected, and the sample size is 126 mm × 76 mm. Detailed descriptions of the serving states are shown in Table 1.

Table 1.

Summary of Collected Data Information

No.	Sample symbol	Sample type	Age (years)	Sample number
1	AC13-1	AC-13	1	37
2	AC13-1(EAM)	Emulsified asphalt mixture	1	33
3	AC13-1.5	AC-13	1.5	29
4	AC13-5	AC-13	5	25
5	AC13-8	AC-13	8	32
6	AC16-1	AC-16	1	24
7	AC20	AC20 (pedestrian)	Unknown	32
8	CC	CC (pedestrian)	Unknown	44

Note: CC = cement concrete.

In this study, a 3D scanner was used to capture high-resolution pavement surface data. The point spacing ranges between 0.015 and 0.1 mm. The measurement resolution varies from 0.001 to 0.5 mm, depending on the distance between the lens and the captured sample. Before collection, the scanner was first calibrated to identify the coordinates of the texture sample. Subsequently, the magnetic contrast Aid (FC-5) was sprayed uniformly on the pavement surface. All relative coordinates of the scanned texture were determined based on the triangulation principle. During the collection process, the scanner was moved around the target sample to capture the points on the pavement surface. Examples of the collected 3D point cloud data for the eight pavement surface textures and their corresponding grayscale images are presented in Figure 1.

Figure 1.

Visualization of the collected pavement surface textures.

Data Preprocessing

The collected raw data requires two preprocessing steps before model training, namely homogenization and planarization. Point cloud data were first homogenized by interpolation method with 0.5 mm point spacing ( 34 ). The reference plane was then adjusted according to the mean square regression plane. Elevation values of the final processed point cloud were transformed into a square matrix for classification training. Figure 2 illustrates the entire process of data preprocessing.

Figure 2.

Schematic diagram of data preprocessing.

Data Annotation and Splitting

A reasonable model should balance both computational accuracy and computational efficiency, which are closely related to the size of the input samples. Considering that the maximum nominal particle size in the sample is 20 mm, the input sample size should be larger than 20 mm. On the other hand, a larger sample size would sacrifice computational efficiency and improve the requirement of dataset volume for training. In this study, the physical sample length of 32 mm × 32 mm is chosen as a balance of the contradiction ( 35 ). Considering that the point cloud data are homogenized with 0.5 mm point spacing, the input data matrix has a size of 65 × 65 accordingly.

To take full advantage of texture samples, a fixed 65 × 65 matrix with different angles and positions was randomly built into the samples to augment input samples, as shown in Figure 3. Original samples were split into two pieces to avoid the interference of datasets before data augmentation with a split ratio of 4:1 for the training-validation set and test set, respectively, which eliminates the overlapping of texture data.

Figure 3.

Input sample data annotation.

Methodology

This study investigated two representative machine learning methods to establish pavement classification models, namely the CNN and SVM. The advantage of the CNN is that the whole texture points can be used as the input data without preselecting the texture indicators, which avoids the loss of texture information. Meanwhile, the CNN shows good fault tolerance and powerful feature extraction and expression capabilities, which are qualities required for texture classification ( 36 – 38 ). For the SVM, as a typical traditional machine learning method, it does not make any assumptions about the distribution of the original data, which indicates that the SVM model has low requirements for data distribution. Since the texture distribution is random and irregular, the SVM is also a good choice for classification.

Convolution Neural Network

Architecture of the CNN

The CNN is a feed-forward neural network, which consists of an input layer, convolutional layers, activation layers, pooling layers, a full connection layer, and an output layer. Each neuron output of the network layers is locally connected to its input ( 39 ). A CNN model with 16 layers was built in this study for pavement surface texture classification. Figure 4 shows the CNN architecture.

Figure 4.

Convolutional neural network architecture.

The input layer serves as the entrance of the network for the preprocessed texture data, as aforementioned in the Data Preprocessing section, while the convolution layer is the key operation for feature extraction by taking the inner product of the input data and filters with defined weights. The nonlinear factors in the network are introduced via the activation layer, which boosts the capability of dealing with complex problems. To decrease the size of model parameters, the pooling layers are designed to further extract the main features. A fully connected layer is set before the output layer to integrate all weighted features into one representative feature, based on which the predicted labels are obtained and output via the output layer. To improve the computational accuracy, the network depth can be increased appropriately, which in turn leads to network overfitting and gradient dispersion issues ( 40 ). The dropout rate ( 41 ) and batch normalization operation ( 42 ) are then put forth. The networks are trained by adjusting the model parameters based on the gradient descent algorithm until the deviation between the predicted and real labels is within the allowance range, which indicates that the model is well-trained.

Considering that texture samples have various maximum nominal particle diameters, convolution kernels of three sizes are designed to capture the texture characteristics within different local receptive fields. It was proved that the model's training accuracy and test accuracy could reach a satisfying result. Table 2 lists the detailed dimensions of each layer and operation. The model was implemented using MATLAB and trained with a computer equipped with an Intel Core i7-8500U CPU, 8.00 GB RAM, and an NVIDIA GeForce MX 150 4 GB GPU.

Table 2.

Dimensions of Layers and Operations

Layer	Height	Width	Depth	Operation	Height	Width	Depth	Number	Stride
Input	65	65	1	Input	NA	NA	NA	NA	NA
C1	65	65	32	C	7	7	1	32	1
BN1	65	65	32	BN	NA	NA	NA	NA	NA
R1	65	65	32	R	NA	NA	NA	NA	NA
M1	32	32	32	M	NA	NA	NA	NA	2
C2	32	32	16	C	5	5	32	16	1
BN2	32	32	16	BN	NA	NA	NA	NA	NA
R2	32	32	16	R	NA	NA	NA	NA	NA
M2	16	16	16	M	NA	NA	NA	NA	2
C3	16	16	16	C	3	3	16	32	1
BN3	16	16	32	BN	NA	NA	NA	NA	NA
R3	16	16	32	R	NA	NA	NA	NA	NA
Dropout	16	16	32	D	NA	NA	NA	NA	NA
Fc	1	1	N	Fc	N	8192	NA	NA	NA
Softmax	1	1	1	Softmax	NA	NA	NA	NA	NA
Classoutput	1	1	1	Output	NA	NA	NA	NA	NA

Note: C = convolution; BN = batch normalization; R = rectified linear unit; M = max-pooling; Fc = full connection; N = the number of labels; NA = not available.

Protocols of the CNN Model

Before using the CNN model, it is necessary to determine the optimal parameters of the model. The hyper-parameter study of the initial learning rate, dropout, and gradient descent method was performed with 20 epochs based on the test accuracy. The batch size was set to 128. Early stopping was applied to select the model weights that obtained the best performance on the validation dataset during training. The validation dataset was not used to update the weights. The selected weights were then used for the following model training stage.

From Tables 3 and 4, it can be seen that the values of the learning rate and dropout settings do not have a significant effect on the accuracy when the learning rate and dropout are set between 0.005–0.05 and 0.1–0.9, respectively. Table 5 shows that the Adam algorithm has the best classification result. Comprehensively, the initial learning rate, dropout, and the gradient descent method are set to the best-performing 0.015, 0.8, and Adam, respectively.

Table 3.

Hyper-Parameter Study of the Initial Learning Rate (%)

Initial learning rate	0.005	0.01	0.015	0.02	0.025	0.03	0.035	0.04	0.045	0.05
Pavement types	88.9	89.8	88.3	90.0	88.9	90.0	90.0	89.1	89.3	89.6
Texture service ages	77.0	77.5	79.0	74.4	73.3	78.8	77.7	73.1	74.4	75.3
Pavement types & texture service ages	80.9	78.7	80.4	79.7	78.7	79.4	80.4	78.8	80.9	79.0

Table 4.

Hyper-Parameter Study of Dropout (%)

Dropout	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
Pavement types	88.1	90.4	90.2	90.4	91.2	91.1	90.7	91.8	92.2
Texture service ages	78.4	79.4	80.6	81.2	80.4	80.1	81.4	81.7	80.1
Pavement types & texture service ages	80.6	81.9	81.5	82.5	81.3	82.8	82.2	82.9	82.1

Table 5.

Determination of the Gradient Descent Method (%)

Gradient descent method	sdgm	rmsprop	Adam
Pavement types	92.0	91.4	92.1
Texture service ages	77.7	77.2	79.5
Pavement types & texture service ages	79.8	80.0	82.9

SVM

The SVM, a binary classification model, implements classification by finding linear separators on the defined feature space. Note that it is challenging to directly classify texture service conditions based on whole texture samples with an excessive number of coordinate points. To reduce the data dimension, texture features that can distinguish samples are preselected manually. Then the extracted feature vectors are input into the SVM for training. Here the wavelet theory was used for feature extraction.

Feature Vector Extraction

Wavelet Theory

Wavelet theory can characterize features both in the time and frequency domains and has the capability of multi-resolution. Generally, wavelet theory has been widely used in signal and image processing. The basic idea of two-dimensional discrete wavelet decomposition is to perform low-pass and high-pass filtering from horizontal and vertical directions. The decomposition process is based on a set of shifted and dilated wavelet functions ψ^H, ψ^Y, ψ^D, and scaling functions φ, which form an orthonormal basis for L² (R²). For a two-dimensional M×N matrix f(x,y), the discrete wavelet transform is performed as follows:

f (x, y) = \frac{1}{\sqrt{M N}} \sum_{m} \sum_{n} W_{φ} (0, m, n) φ_{0, m, n} x, y + \frac{1}{\sqrt{M N}} \sum_{j = 0}^{\infty} {\sum_{m} \sum_{n} W_{ψ}^{H} (j, m, n) ψ_{j, m, n}^{H} (x, y) + \sum_{m} \sum_{n} W_{ψ}^{V} (j, m, n) ψ_{j, m, n}^{V} (x, y) + \sum_{m} \sum_{n} W_{ψ}^{D} (j, m, n) ψ_{j, m, n}^{D} (x, y)}

(1)

W_{φ} (0, m, n) = \frac{1}{\sqrt{MN}} \sum_{x = 0}^{M - 1} \sum_{y = 0}^{N - 1} f (x, y) φ_{0, m, n} (x, y)

(2)

W_{ψ}^{i} (j, m, n) = \frac{1}{\sqrt{MN}} \sum_{x = 0}^{M - 1} \sum_{y = 0}^{N - 1} f (x, y) ψ_{j, m, n}^{i} (x, y) i = {H, V, D}

(3)

φ_{0, m, n} x, y = 2^{j / 2} φ (2^{j} x - m, 2^{j} - n)

(4)

ψ_{j, m, n}^{i} (x, y) = 2^{j / 2} ψ^{i} (2^{j} x - m, 2^{j} - n) i = {H, V, D}

(5)

where $φ_{j, m, n} x, y$ and $ψ_{j, m, n}^{i} (x, y)$ are the scaling function and wavelet function in different scales and positions. $W_{φ} (0, m, n)$ is a scaling coefficient, and $W_{ψ}^{i} (j, m, n)$ denotes the (m,n)th wavelet coefficient in scale j.

Wavelet Decomposition

Daubechies (Db) wavelets are widely used to process signals and images at various scales because of their orthogonality and compact support, which are ideal properties for iterative decomposition. The most common mother wavelet in transportation-related applications is the Db3 mother wavelet. Favorable results have been achieved by using this mother wavelet to analyze the surface texture ( 43 , 44 ), roughness ( 45 ), and vehicle–pavement interaction ( 46 ). Therefore, Db3 was chosen as the mother wavelet in this study to decompose the texture data. Since the minimum interval of texture is 0.5 mm and the upper limit of input texture wavelength is 32 mm, six-layer wavelet decomposition was used. The decomposition diagram is shown in Figure 5.

Figure 5.

Wavelet decomposition diagram of the pavement surface texture.

Feature Extraction

According to the coefficients decomposed by the wavelet, the feature indexes at each scale can be extracted, and the feature vector can be constructed by transforming the coefficient. Commonly used indicators are the mean (M), entropy (EN), energy (E), and standard deviation (S) ( 47 ). Here they are collectively referred to as EEMSD:

M = \frac{1}{mn} \sum_{i = 1}^{m} \sum_{j = 1}^{n} p (i, j)

(6)

EN = - \frac{1}{mn} \sum_{i = 1}^{m} \sum_{j = 1}^{n} p (i, j) lo g_{2} p (i, j)

(7)

S = \sqrt{\frac{1}{mn} \sum_{i = 1}^{m} \sum_{j = 1}^{n} {(p (i, j) - M)}^{2}}

(8)

E = \sum_{i = 1}^{m} \sum_{j = 1}^{n} (p (i, j)^{2})

(9)

where p(i,j) is the amplitude of the decomposed m×n subband at (i, j).

SVM Classification

Based on the preselected features obtained by wavelet decomposition, the further classification of the service conditions of the pavement surface texture can be achieved by the SVM. The texture classification process used a multi-class SVM ( 48 ) based on the standard one-versus-one approach to reduce the single multi-class problem into multiple binary classification problems. For each texture sample, the extracted feature vectors x ∈ R _N based on wavelet theory have N = 96 elements. According to the standard one-versus-one approach, the label y∈Y={−1,1} is set as the output, in which 1 indicates that the sample belongs to the target category, and not vice versa. The dataset is then denoted as T = {( x ₁, y₁), ( x ₂, y₂), …, ( x _n, y_n)}, where n is the dataset size. The aim of the SVM algorithm is to find a function, g: R _N→ R , serving as a maximum-margin hyper-plane (H:w^T x +b = 0) to separate the sample category accurately by solving an optimization problem, where w is the normal vector and b is the bias. Undoubtedly, there is no satisfactory linear hyper-plane to segment nonlinear distributed features. To address this issue, various kernel functions ϕ(·) are introduced to transform the original texture feature vectors into high-dimensional space, in which the samples become linearly separable, as shown in Figure 6.

Figure 6.

Transformation using kernel functions.

In this study, a quadratic kernel function has been chosen to achieve the best accuracy when compared with linear, cubic, and Gaussian kernel functions. The SVM adopted a cross-validation method when performing classification training in this study.

Experiment Design

Considering the stochastic complexity of the surface texture distribution, 10,000 augmented samples were used for each pavement to fully cover the collected texture information. The large input dataset size also reduces the number of training epochs accordingly. Specifically, 2000 of them were used to test the training results and 7200 samples were subjected to the model training, while the remaining 800 samples were used for the validation. Pavement types and texture service ages were used as classification objects to discuss the classification performance of the CNN and SVM. For further comparative analysis of the impact of increased categories on the classification accuracy, the CNN and SVM were also used to train a universal model for the combined classification objects, including pavement type (AC13, AC13(EAM), AC16, AC20, and CC) and texture service age (1, 1.5, 5, and 8) for AC13. For convenience, the combined classification case is referred to as both pavement type and texture service age.

Metrics

Confusion Matrix

The confusion matrix was used to visualize the classification results with combinations of predicted and actual values. As shown in Figure 7, in a confusion matrix M = ( $m_{ij}$ ) ∈N^n×n (i, j = 1, 2, …, n), for class j $m_{jj}$ presents the true positives (TP_j). The other elements in column j are false negatives (FN_j), in row j false positives (FP_j), and the remaining elements are true negatives (TN_j).

Figure 7.

Schematic of the confusion table.

In addition, the precision, recall, and accuracy for the classification of multi-class pavement surface service conditions were calculated:

Precisio n_{j} = \frac{m_{jj}}{\sum_{i = 1}^{n} m_{ji}} = \frac{T P_{j}}{T P_{j} + F P_{j}}

(10)

Recal l_{j} = \frac{m_{jj}}{\sum_{i = 1}^{n} m_{ij}} = \frac{T P_{j}}{T P_{j} + F N_{j}}

(11)

Accuracy = \frac{\sum_{i = 1}^{n} m_{ii}}{\sum_{i = 1}^{n} \sum_{j = 1}^{n} m_{ij}}

(12)

where Precision_j is the proportion of prediction results where the actual value is also type j, Recall_j indicates the proportion of samples with the actual value j that are accurately predicted to be type j, and Accuracy represents the comprehensive classification accuracy of the model.

Receiver Operating Characteristic

To compare the relative performance of the trained classifier based on the CNN and SVM and validate whether the classifiers perform better than random guessing, receiver operating characteristic (ROC) analysis, which measures the classifier by examining the trade-off between the successful identification of positive samples and the misdetection of negative samples, was introduced in this study ( 49 ). In the ROC space, the true positive rate, TPR, and false positive rate, FPR, of a classifier are plotted on the Y- and X-axes, respectively. Considering that the involved classifications are multi-class tasks and the sample labels are uniformly distributed, the average ROC across all categories in the classification case was calculated to evaluate the model performance generally, as shown in Figure 8. Specifically, let p(P∣I) be the probability that instance I is estimated to be positive on a test set. The TPR and FPR of a classifier are as follows:

TPR = \frac{\sum_{j = 1}^{n} p_{j} (Classify Positive ∣ Positive)}{n} = \frac{\sum_{j = 1}^{n} precisio n_{j}}{n}

(13)

FPR = \frac{\sum_{j = 1}^{n} p_{j} (Classify Positive ∣ Negative)}{n}

(14)

Figure 8.

Schematic diagram of the receiver operating characteristic (ROC).

These two statistics vary together as the threshold of the classifier is altered between its extremes, and then the resulting ROC curve is drawn. For orientation, the point (0, 0) in the ROC space corresponds to an extremely conservative classifier, which yields no true positive. Similarly, the point (1, 1) represents a classifier that declares every sample to be positive. Since the point (0, 1) corresponds to the ideal classifier in which both positive and negative samples are issued perfectly, points closer to the upper-left corner of ROC space are generally better classifiers. In contrast, a random classifier produces points that reside along the diagonal line connecting the bottom-left to the upper-right corner of the ROC graph.

Results and Discussion

Classification Model of the Pavement Type

The loss curves for the CNN during the training process are shown in Figure 9. It can be seen that the loss curves for the training and validation dataset are close and show a continuously downward trend. Notably, validation samples are different from training samples yet partially contain textures overlapped with the training samples induced by the sample data annotation method. In this regard, overfitting during the training process is diagnosed as not obvious. Meanwhile, the loss for both training and validation datasets decreases continuously with iterations and shows a slow downward trend after about 25 epochs. Based on the trade-off between computation load and efficiency, the 30 epochs were finally adopted. Table 6 summarizes the precision, recall, and accuracy for different pavement types using the CNN and SVM.

Figure 9.

The loss curves during the convolutional neural network training process for classification of pavement types.

Table 6.

Model Merits of Classification of Pavement Types (%)

Merits	Methods	AC13-8	AC13-1 (EAM)	AC16-1	AC		CC
Precision	CNN	90.5	93.2	97.3	87.6		96.4
Precision	SVM	88.0	81.2	83.4	88.3		98.8
Recall	CNN	80.6	92.5	95.4	96.5		99.8
Recall	SVM	86.0	72.0	89.9	93.0		99.3
Accuracy	CNN	Training				96.4
		Validation				93.2
		Test				93.0
	SVM	Train and validation				84.4
	SVM	Test				88.0

Note: EAM = emulsified asphalt mixture; CC = cement concrete; CNN = convolutional neural network; SVM = support vector machine.

The trained CNN classification model has a satisfying outcome, as it can be seen that the training, validation, and test accuracy all exceed 90%. It demonstrates that the CNN model could extract features effectively from randomly distributed textures. As for the SVM, it shows relatively low accuracy, as it is observed that the accuracy results are limited between 80% and 90%. Concerning the average precision and recall for different pavement types, it can be found that the overall performance of the CNN classification model for different textures follows the sequence of CC > AC16-1 > AC13-1(EAM) > AC20 > AC13-8 (Figure 10).

Figure 10.

Confusion matrixes of the convolutional neural network (CNN) and support vector machine (SVM) for classification of pavement types: (a) CNN training set, (b) CNN validation set, (c) CNN test set, (d) SVM training and validation set, and (e) SVM test set.

The misclassification of the SVM is distributed among types AC13-8, AC13-1 (EAM), AC16-1, and AC20. It may be attributed to the loss of important data information when manually extracting features after wavelet decomposition. In addition, the differentiation for type CC is still better than others, which may be because type CC has clearly distinguishing features. In conclusion, for those with distinct features, both the CNN and SVM could obtain the desired classification results. However, the CNN outperforms the SVM when it comes to categorizing samples with indistinct features.

Classification Model of the Texture Service Age

Concerning the adequacy of the texture data source and its diversity of service ages, AC13 was adopted as an example to explore the classification performance of the CNN and SVM for texture service ages with the same aggregate gradation. The loss curves for the CNN during the training process are shown in Figure 11. Table 7 lists the classification results of the CNN and SVM. It can be seen that the SVM fails in classifying the years of service. On the other hand, the CNN could still achieve satisfactory results for service age classification, indicating good capability of dealing with complex random data with no distinct features. Compared to the Classification Model of the Pavement Type section, the accuracy of the CNN model in classifying the texture service ages decreases, which is in line with our general perception. It is because the variability of the same type of pavement caused by the service age is lower than that of aggregate gradation. From Table 7, it can be found that the overall performance of the CNN for classifying the four types of pavements is in the order of AC13-1 > AC13-8 ≈ AC13-1.5 > AC13-5.

Figure 11.

Loss curves during the convolutional neural network training process for the classification of service ages.

Table 7.

Model Merits of Classification of Texture Service Age (%)

Merits	Methods	AC13-1	AC13-1.5	AC13-5		AC13-8
Precision	CNN	94.5	70.1	89.5		75.5
Precision	SVM	70.3	55.9	55.0		51.2
Recall	CNN	97.0	83.7	61.1		82.7
Recall	SVM	78.7	59.5	46.4		49.8
Accuracy	CNN	Training			97.4
		Validation			95.5
		Test			81.2
	SVM	Training & validation			66.0
	SVM	Test			58.6

Note: CNN = convolutional neural network; SVM = support vector machine.

By observing the distribution of misclassified samples in Figure 12, it can be found that the main misclassification was that some type AC13-5 samples are misclassified as types AC13-8 and AC13-1.5, which may be because texture samples from different locations had undergone varying degrees of abrasion. Besides, there is a small amount of mutual misclassification between types AC13-1.5 and type AC13-8, as some areas with similar features inevitably exist between the same pavement surface texture. However, most samples could be well classified using the CNN model.

Figure 12.

Confusion matrixes of the convolutional neural network (CNN) and support vector machine (SVM) for classification of service ages: (a) CNN training set, (b) CNN validation set, (c) CNN test set, (d) SVM training and validation set, and (e) SVM test set.

Classification Model of Both Pavement Type and Service Age

The loss curves for the CNN during the training process are shown in Figure 13. Table 8 lists the classification results of the CNN and SVM on the service state of the eight textures. It can be seen that the classification accuracy of the SVM method for both the training and validation dataset and the test dataset is very low. In contrast, the CNN has a comprehensive accuracy rate of 83.8%. Concerning the average precision and recall for different texture types, the overall performance of the CNN classification model follows the sequence of CC > AC16-1 > AC20 > AC13-1 (EAM) ≈ AC13-1 > AC13-5 > AC13-8 > AC13-1.5.

Figure 13.

Loss curves during the convolutional neural network training process for classification of both pavement type and texture service age.

Table 8.

Model Merits of Classification of Both Pavement Type and Texture Service Age (%)

Merits	Methods	AC13-1	AC13-1 (EAM)	AC13-1.5	AC13-5	AC13-8		AC16-1	AC20	CC
Precision	CNN	80.3	84.0	77.0	91.7	82.0		89.8	72.4	98.0
Precision	SVM	55.6	77.7	46.3	55.9	49.3		70.3	84.8	98.6
Recall	CNN	83.9	80.3	74.7	67.8	73.1		97.4	94.3	99.1
Recall	SVM	58.1	63.5	52.6	46.2	47.7		82.3	87.2	99.2
Accuracy	CNN	Training					93.2
		Validation					90.5
		Test					83.8
	SVM	Training & validation					69.9
	SVM	Test					67.1

Note: EAM = emulsified asphalt mixture; CC = cement concrete; CNN = convolutional neural network; SVM = support vector machine.

By comparing Tables 6–8, it can be seen that the classification accuracy of types AC13-8, AC13-1 (EAM), AC16-1, AC13-1, AC13-1.5, AC20, and CC, which have apparent features, are all reduced. It can be inferred that when the number of classification categories increases, the feature extraction process takes all of the characteristics contained in each type into account simultaneously, leading to a universal decrease in classification accuracy. Based on this, the necessity of classifying various types of pavement surface textures before analyzing the performance of pavement surface textures is indirectly confirmed. However, it is worth mentioning that the classification results of type AC13-5 have been improved, as shown in Figure 14. This may be because the increase of the texture category promotes the further extraction of effective features from type AC13-5 samples whose features are not evident. Therefore, when classifying texture service conditions with significant differences, reducing classification categories can result in obtaining a better classification performance. On the other hand, when classifying texture service conditions with few differences, increasing the categories of input samples helps to improve the classification accuracy.

Figure 14.

Confusion matrixes of the convolutional neural network (CNN) for classification of both pavement type and texture service age: (a) CNN training set, (b) CNN validation set, (c) CNN test set, (d) support vector machine (SVM) training and validation set, and (e) SVM test set.

Performance Evaluation Between the CNN and SVM Methods

Figure 15 shows the ROC of the CNN and SVM models for the three classification cases based on the test datasets. It can be seen that the designed CNN classifiers are discrete-valued output mapped to a single point in the ROC graph because the involved parameters were trained to find the only one optimal solution in the self-supervised learning process, whereas the SVM based on the standard one-versus-one approach produces numeric-valued outputs forming a piecewise linear curve by varying the threshold that determines whether a new sample belongs to the target class. Obviously, the ROC curves of the SVM are above the random guessing line, which indicates that the classification results of the SVM classifiers are not achieved only by chance. In addition, it should be mentioned that the one-versus-one calculation principle of the SVM caused skewed class distribution issues. The slope of the iso-performance line for best-performing SVM model adoption was then adjusted from 1 to 4 (type), 3 (age), and 7 (type and age), respectively according to the category number ( 50 ). The optimum SVM classifiers are marked in Figure 15. Consistent with the results of the confusion matrix, the SVM and CNN perform best in the type classification case, followed by the classification case of both type and age. The performance of age classification is the worst. Moreover, compared to the SVM classifiers for each case, the ROCs of the CNN classifiers are closer to the upper-left corner in the ROC space, proving a superior performance of the CNN compared with the SVM in the texture type and age identification tasks.

Figure 15.

Receiver operating characteristic of the convolutional neural network (CNN) and support vector machine (SVM) models.

Conclusions

The classification of the service condition of pavement surface textures is an essential premise for the evaluation of texture-related pavement performance in pavement management systems. However, there is still a lack of effective methods for classifying the service conditions of common pavement surface textures with respect to pavement types and texture service ages. Considering the unclear recognition of texture features induced by the random irregular distributions, the CNN was adopted to explore an automated service condition classification model based on pavement surface texture. Three classification cases, case 1: pavement type classification, case 2: texture service age classification, and case 3: the classification of texture service conditions involved in the above two cases, were studied consecutively. To illustrate the advantages of the CNN models, the typical machine learning method, namely the SVM, was used simultaneously for the same classification cases. Considering the limitations of this study, the results show promising outcomes.

(1) The proposed CNN model achieves favorable classification accuracies of 93.0%, 81.1%, and 83.8% in the three cases, respectively.

(2) For the classification of pavement types with distinct texture features, both the CNN and SVM achieve good classification results of 93.0% and 88.0%, respectively. However, for the classification of service ages, the CNN shows good classification results of 81.2%, while the SVM fails with an accuracy of 58.6%.

(3) For the classification of pavement types with distinct texture features, such as AC13-8, AC13-1 (EAM), AC16-1, AC13-1, AC13-1.5, AC20, and CC, the increased classification categories tend to act as interference information, thus reducing the classification accuracy. However, for the classification of service ages with indistinct texture features (AC13-5), the increased classification categories could guide a further extraction of texture features, therefore increasing the classification accuracy.

(4) The ROC of both the SVM and CNN are above the random guessing line in the ROC space, proving that the classification results of the classifiers are not achieved only by chance. Compared to the SVM classifiers for each case, the ROCs of the CNN classifiers are closer to the upper-left corner.

Limitations and Future Work

It should be mentioned that the models in this study were trained under a limited variety of texture service conditions. Meanwhile, the label of texture ages is a rough mark without comprehensively considering the actual texture wearing status. Therefore, with respect to the practical application in pavement management systems, the automatic identification models for texture data still deserve further investigation and this study is supposed to serve as a foundational exploration. Based on this initial study, future work may include the following aspects.

(1) The interpretability of the texture feature latent extracted by machine learning methods should be studied further to improve the CNN architectures.

(2) The practical performance of texture samples with a finer or coarser grid should be studied further in the future to achieve a balance between classification accuracy and efficiency.

(3) A standard texture database should be established and expanded by involving factors beyond pavement type and surface age, such as aggregate type, shape, surface texture, and gradation, as well as surface condition and skid resistance measurement methods to further enhance the generality of the CNN models. Once new texture samples are input into the models, the texture condition benchmarked with the standard texture database could be obtained.

(4) Methodologies to embed the well-trained CNN model into the follow-up process of texture performance evaluation and maintenance decision-making still deserves further investigation.

Footnotes

Acknowledgements

We gratefully acknowledge the highway engineering laboratories at Dalian University of Technology.

Author Contributions

The authors confirm their contribution to the paper as follows: study conception and design: J. Lu, B. Pan, Q. Liu; data collection: J. Lu; analysis and interpretation of results: J. Lu; draft manuscript preparation: J. Lu., B. Pan, Q. Liu, P. Liu, M. Oeser. All authors reviewed the results and approved the final version of the manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Quan Liu

Pengfei Liu

Data Accessibility Statement

Some or all data, models, or codes that support the findings of this study are available from the corresponding author on reasonable request.

References

Huang

Dong

Wang

LCA and LCCA Based Multi-Objective Optimization of Pavement Maintenance. Journal of Cleaner Production, Vol. 283, 2021, p. 124583. https://doi.org/10.1016/j.jclepro.2020.124583

Rodriguez-Lozano

F. J.

León-García

Gámez-Granados

J. C.

Palomares

J. M.

Olivares

Benefits of Ensemble Models in Road Pavement Cracking Classification. Computer-Aided Civil and Infrastructure Engineering, Vol. 35, 2020, pp. 1194–1208. https://doi.org/10.1111/mice.12543

Dong

Meng

Liu

A Framework of Pavement Management System Based on IoT and Big Data. Advanced Engineering Informatics, Vol. 47, 2021, p. 101226. https://doi.org/10.1016/j.aei.2020.101226

Elbagalati

Elseifi

M. A.

Gaspard

Zhang

Development of an Enhanced Decision-Making Tool for Pavement Management Using a Neural Network Pattern-Recognition Algorithm. Journal of Transportation Engineering, Part B: Pavements, Vol. 144, 2018, p. 04018018. https://doi.org/10.1061/jpeodx.0000042

Fernando

E. G.

Crockford

Comparative Evaluation of Pavement Macrotexture Measurements from Different Devices. Transportation Research Record: Journal of the Transportaion Research Board, Vol. 2677, 2022, pp. 782–796. https://doi.org/10.1177/03611981221117818

Hong

Huang

Measurement and Characterization of Asphalt Pavement Surface Macrotexture Using Three Dimensional Laser Scanning Technology. Journal of Testing and Evaluation, Vol. 42, 2014, pp. 881–890. https://doi.org/10.1520/JTE20130147

Serigos

P. A.

Prozzi

J. A.

Smit

A. D.

Murphy

M. R.

Evaluation of 3D Automated Systems for the Measurement of Pavement Surface Cracking. Journal of Transportation Engineering, Vol. 142, 2016, p. 05016003. https://doi.org/10.1061/(ASCE)TE.1943-5436.0000841

Chun

P. J.

Yamane

Tsuzuki

Automatic Detection of Cracks in Asphalt Pavement Using Deep Learning to Overcome Weaknesses in Images and Gis Visualization. Applied Sciences, Vol. 11, 2021, pp. 1–15. https://doi.org/10.3390/app11030892

Famili

Sarasua

W. A.

Shams

Davis

W. J.

Ogle

J. H.

Application of mobile terrestrial LiDAR scanning systems for identification of potential pavement rutting locations. Transportation Research Record: Journal of Transportation Research Board, 2021. 2675(9): 1063–1075. https://doi.org/10.1177/03611981211005777

10.

Chen

Seo

Zhao

A Novel Pavement Transverse Cracks Detection Model Using WT-CNN and STFT-CNN for Smartphone Data Analysis. International Journal of Pavement Engineering, Vol. 23, 2021, pp. 4372–4384. https://doi.org/10.1080/10298436.2021.1945056.

11.

Nappo

Mavrouli

Nex

van Westen

Gambillara

Michetti

A. M.

Use of UAV-Based Photogrammetry Products for Semi-Automatic Detection and Classification of Asphalt Road Damage in Landslide-Affected Areas. Engineering Geology, Vol. 294, 2021, p. 106363. https://doi.org/10.1016/j.enggeo.2021.106363

12.

Llopis-Castelló

Paredes

Parreño-Lara

García-Segura

Pellicer

Automatic Classification and Quantification of Basic Distresses on Urban Flexible Pavement through Convolutional Neural Networks. Journal of Transportation Engineering, Part B: Pavements, Vol. 147, 2021, p. 4021063. https://doi.org/10.1061/JPEODX.0000321.

13.

Tang

Zhang

A. A.

Luo

Wang

Yang

Pixel-Level Pavement Crack Segmentation with Encoder-Decoder Network. Measurement, Vol. 184, 2021, p. 109914. https://doi.org/10.1016/j.measurement.2021.109914

14.

Han

Huyan

Huang

Zhang

, CrackW-Net: A Novel Pavement Crack Image Segmentation Convolutional Neural Network. IEEE Transactions on Intelligent Transportation System, Vol. 23, 2021, pp. 22135–22144. https://doi.org/10.1109/tits.2021.3095507

15.

Polovnikov

Alekseev

Vinogradov

Lashkia

G. V.

DAUNet: Deep Augmented Neural Network for Pavement Crack Segmentation. IEEE Access, Vol. 9, 2021, pp. 125714–125723. https://doi.org/10.1109/ACCESS.2021.3111223.

16.

Dan

H. C.

L. H.

Experimental Investigation on Skid Resistance of Asphalt Pavement Under Various Slippery Conditions. International Journal of Pavement Engineering, Vol. 18, 2017, pp. 485–499. https://doi.org/10.1080/10298436.2015.1095901.

17.

Mikhailenko

Piao

Kakar

M. R.

Bueno

Athari

Pieren

Heutschi

Poulikakos

Low-Noise Pavement Technologies and Evaluation Techniques: A Literature Review. International Journal of Pavement Engineering, Vol. 23, 2020, pp. 1911–1934. https://doi.org/10.1080/10298436.2020.1830091.

18.

Luo

Wang

K. C. P.

Wei

Surface Drainage Evaluation of Asphalt Pavement Using A New Analytical Water Film Depth Model. Road Material and Pavement Engineering, Vol. 21, 2019, pp. 1985–2004. https://doi.org/10.1080/14680629.2019.1590220.

19.

Pereira

Tamura

Hayamizu

Fukai

Classification of Paved and Unpaved Road Image Using Convolutional Neural Network for Road Condition Inspection System. 5th International Conference on Advanced Informatics - Concepts, Theory and Applications (ICAICTA), Krabi, Thailand, IEEE, New York, 2018, pp. 165–169.

20.

Riid

Manna

D. L.

Astapov

Image-Based Pavement Type Classification with Convolutional Neural Networks. 24th IEEE International Conference on Intelligent Engineering Systems (INES), Reykjavík, Iceland, IEEE, New York, 2020, pp. 55–60.

21.

You

Zhang

Jia

Zhang

Zhao

Yan

Long-Term Performance Deterioration Models for Semi-Rigid Asphalt Pavement in Cold Region. International Journal of Pavement Research and Technology, Vol. 14, 2021, pp. 697–707. https://doi.org/10.1007/s42947-020-0044-x.

22.

Shi

Mukhopadhyay

Zollinger

Huang

Performance Evaluation of Jointed Plain Concrete Pavement Made With Portland Cement Concrete Containing Reclaimed Asphalt Pavement. Road Materials and Pavement Design, Vol. 22, 2021, pp. 59–81. https://doi.org/10.1080/14680629.2019.1616604.

23.

Zhao

Fan

A Brief Review of Sealants for Cement Concrete Pavement Joints and Cracks. Road Materials and Pavement Design, Vol. 23, 2021, pp. 1467–1491. https://doi.org/10.1080/14680629.2021.1898452

24.

Han

Chen

Asphalt Pavement Maintenance Plans Intelligent Decision Model Based on Reinforcement Learning Algorithm. Construction and Building Materials, Vol. 299, 2021, p. 124278. https://doi.org/10.1016/j.conbuildmat.2021.124278

25.

Moayedfar

Sajjadifard

Prioritization of Pavement Restoration and Maintenance Strategies in Airports Using APMS Technique. International Journal of Pavement Research and Technology, Vol. 14, 2021, pp. 327–333. https://doi.org/10.1007/s42947-020-0244-4.

26.

Jiang

Selecting the Optimal Network-Level Pavement Maintenance Budget Scenario Based on Sustainable Considerations. Transportation and Research Part D: Transport and Environment, Vol. 97, 2021, p. 102919. https://doi.org/10.1016/j.trd.2021.102919

27.

Kogbara

R. B.

Masad

E. A.

Kassem

Scarpas

Anupam

A State-of-the-Art Review of Parameters Influencing Measurement and Modeling of Skid Resistance of Asphalt Pavements. Construction and Building Materials, Vol. 114, 2016, pp. 602–617. https://doi.org/10.1016/j.conbuildmat.2016.04.002.

28.

Tsai

Y. C. J.

Critical Assessment of Detecting Asphalt Pavement Cracks under Different Lighting and Low Intensity Contrast Conditions Using Emerging 3D Laser Technology. Journal of Transportation Engineering, Vol. 138, 2012, pp. 649–656. https://doi.org/10.1061/(ASCE)TE.1943-5436.0000353.

29.

Zhang

Liu

Chen

Research on Skid Resistance of Asphalt Pavement Based on Three-Dimensional Laser-Scanning Technology and Pressure-Sensitive Film. Construction and Building Materials, Vol. 69, 2014, pp. 49–59. https://doi.org/10.1016/j.conbuildmat.2014.07.015.

30.

Ding

Wang

K. C. P.

Yang

Zhan

Influence of Effective Texture Depth on Pavement Friction Based on 3D Texture Area. Construction and Building Material, Vol. 287, 2021, p. 123002. https://doi.org/10.1016/j.conbuildmat.2021.123002.

31.

Tong

Gao

Han

Wang

Recognition of Asphalt Pavement Crack Length Using Deep Convolutional Neural Networks. Road Materials and Pavement Design, Vol. 19, 2018, pp. 1334–1349. https://doi.org/10.1080/14680629.2017.1308265.

32.

Dung

Anh

Autonomous Concrete Crack Detection Using Deep Fully Convolutional Neural Network. Automation in Construction, Vol. 99, 2019, pp. 52–58. https://doi.org/10.1016/j.autcon.2018.11.028.

33.

Han

Huang

Zhang

CrackW-Net: A Novel Pavement Crack Image Segmentation Convolutional Neural Network. IEEE Transactions on Intelligent Transportation Systems, Vol. 23, 2022, p. 22135. https://doi.org/10.1109/TITS.2021.3095507

34.

Pan

Liu

Sun

Liu

Oeser

A Novel Noncontact Method for the Pavement Skid Resistance Evaluation Based on Surface Texture. Tribology International, Vol. 165, 2022, p. 107311. https://doi.org/10.1016/j.triboint.2021.107311.

35.

Pan

Ren

Liu

Oeser

Regeneration of Pavement Surface Textures Using M-Sigmoid-Normalized Generative Adversarial Networks. Computer-Aided Civil and Infrastructure Engineering, Vol. 38, 2023, pp. 2225–2241. https://doi.org/10.1111/mice.12987.

36.

Pan

Deng

Shen

Kang

Pavement Distress Detection and Classification Based on YOLO Network. International Journal of Pavement Engineering, Vol. 22, 2021, pp. 1659–1672. https://doi.org/10.1080/10298436.2020.1714047.

37.

Liu

Yang

Lau

Wang

Luo

Lee

V. C. S.

Ding

Automated Pavement Crack Detection and Segmentation Based on Two-Step Convolutional Neural Network. Computer-Aided Civil and Infrastructure Engineering, Vol. 35, 2020, pp. 1291–1305. https://doi.org/10.1111/mice.12622.

38.

Bang

Park

Kim

Encoder–Decoder Network for Pixel-Level Road Crack Detection in Black-Box Images. Computer-Aided Civil and Infrastructure Engineering, Vol. 34, 2019, pp. 713–727. https://doi.org/10.1111/mice.12440.

39.

LeCun

Bottou

Bengio

Haffner

Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, Vol. 86, 1998, pp. 2278–2323. https://doi.org/10.1109/5.726791.

40.

Simonyan

Zisserman

Very Deep Convolutional Networks for large-Scale Image Recognition. 3rd International Conference on Learning Representations ICLR, San Diego, CA, 2015. Ithaca, NY: ArXiv, Cornell Tech, 2015, arXiv:1409.1556.

41.

Krizhevsky

Sutskever

Hinton

G. E.

ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing System (Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, eds.), Curran Associates, Inc., The MIT Press, 2012. https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf.

42.

Szegedy

Liu

Jia

Sermanet

Reed

Anguelov

Erhan

Vanhoucke

Rabinovich

Going Deeper With Convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, IEEE, New York, 2015, pp. 1–9. https://doi.org/10.1109/CVPR.2015.7298594

43.

Yang

Q. J.

Zhan

Y. J.

Wang

K. C. P.

Wang

Wavelet Based Macrotexture Analysis for Pavement Friction Prediction. KSCE Journal of Civil Engineering, Vol. 22, 2018, pp. 117–124. https://doi.org/10.1007/s12205-017-1165-x.

44.

Zelelew

Khasawneh

Abbas

Wavelet-Based Characterisation of Asphalt Pavement Surface Macro-Texture. Road Materials and Pavement Design, Vol. 15, 2014, pp. 622–641. https://doi.org/10.1080/14680629.2014.908137

45.

Liu

Fwa

T. F.

Zhao

Wavelet Analysis and Interpretation of Road Roughness. Journal of Transportation Engineering, Vol. 131, 2005, pp. 120–130. https://doi.org/10.1061/(ASCE)0733-947X(2005)131:2(120).

46.

Papagiannakis

A. T.

Zelelew

H. M.

Muhunthan

A Wavelet Interpretation of Vehicle-Pavement Interaction. International Journal of Pavement Engineering, Vol. 8, 2007, pp. 245–252. https://doi.org/10.1080/10298430701309378.

47.

Faziludeen

Sabiq

P. V.

ECG Beat Classification Using Wavelets and SVM, 2013 IEEE Conference on Information and Communication Technologies ICT, Thuckalay, India, IEEE, New York, 2013, pp. 815–818. https://doi.org/10.1109/CICT.2013.6558206.

48.

Vural

J. G.

A Hierarchical Method for Multi-Class Support Vector Machines. Proceedings of the Twenty-First International Conference on Machine Learning ICML, Banff, Alberta, Association for Computing Machinery, New York, 2004, pp. 831–838. https://doi.org/10.1145/1015330.1015427.

49.

Park

S. H.

Goo

J. M.

Receiver Operating Characteristic (ROC) Curve: Practical Review for Radiologists. Korean Journal of Radiology, Vol. 5, 2004, pp. 11–18.

50.

Provost

Fawcett

Analysis and Visualization of Classifier Performance with Nonuniform Class and Cost Distributions. Proceedings of the the Fourteenth National Conference on Artificial Intelligence and The Ninth Annual Conference on Innovative Applications of Artificial Intelligence, Providence, Rhode Island, AAAI Press, Washington, DC, U.S. 1997, pp. 57–63.