Imaging-based crack detection on concrete surfaces using You Only Look Once network

Abstract

The detection of cracks in concrete structures is a pivotal aspect in assessing structural robustness. Current inspection methods are subjective, relying on the inspector’s experience and mental focus. In this study, an ad hoc You Only Look Once version 2 object detector was applied to automatically detect concrete cracks from real-world images, which were taken from diverse concrete bridges and contaminated with handwriting scripts. A total of 3010 cropped images were used to generate the dataset, labelled for two different detection classes, that is, cracks and handwriting. The proposed network was then trained and tested using the generated image dataset. Three full-scale images that contained disturbing background information were used to evaluate the robustness of the trained detector. The influence of labelling handwriting as an object class for network training on the overall crack detection accuracy was assessed as well. The results of this study show that the You Only Look Once version 2 could automatically locate crack with bounding boxes from raw images, even with the presence of handwriting scripts. As a comparative study, the proposed network was also compared with faster region-based convolutional neural network. The results showed that You Only Look Once version 2 performed better in terms of both accuracy and inference speed.

Keywords

Neural network object detector crack detection concrete structures machine learning

Introduction

Concrete structures are heavily influenced by environmental conditions. Chemical erosion, neutralisation, frost-salt scaling and alkali aggregate reaction are some common degradation mechanisms that result in cracks.¹ In particular, concrete cracks pose a threat to the health of the structure and the safety of its users. In the United States, civil infrastructure systems are estimated to be worth US$20 trillion.² Inspections of these prevalent systems allow agencies to predict future conditions, to support investment planning and to allocate their limited maintenance and repair resources.³ Several incidents have been directly related to insufficient inspection and condition assessment of concrete structures. For example, 10 people died as a result of the sudden collapse of the New York State Thruway Bridge over Schoharie Creek in 1987. In the final accident report, the New York State Thruway Authority identified major safety issues including lack of inspection guidance for the initiation and unstable propagation of the curvilinear crack in the pier.⁴ Current concrete inspection methods include manual visual assessment using a crack scale. Typically, condition assessment procedures are performed manually by certified inspectors or structural engineers, either at regular intervals or after disasters. This, however, is both time and labour consuming and relies on the inspector’s experience and mental focus. Areas of structures that are difficult to examine may be inspected using binoculars, or dangerously disregarded altogether. Moreover, in an extreme event, such as terrorism or earthquake, real-time structural monitoring would be paramount to quickly identify the change in structural soundness, such as cracking characteristics, which may pose a threat to the public safety if it is not analysed on time. With increases in labour costs, the need for automatic and real-time damage detection is escalating.

Recently, to address the shortcomings of manual inspection, computer vision (CV) has been integrated into bridge inspection and evaluation.^5–8 CV, which enables the computer to recognise and understand the images and scenes, has become an integral part of structural health monitoring (SHM) systems of bridges. For faster and more reliable asset management, automatic crack detection systems have been developed to minimise the subjectivity of traditional human inspection procedures, in particular with the involvement of unmanned aerial vehicles (UAVs).^9,10

A number of approaches have been proposed to address the challenge of manual inspection using CV-based crack detection. From the aspect of manual inputs, existing approaches can be divided into rule-based and machine learning–based (ML-based) methods.^11,12 Rule-based or image processing techniques (IPTs) use predefined features to identify cracks from images such as edge detection methods and pattern recognition techniques.^13–18 The edge detection technique is based on the assumption that crack pixels are generally darker than their surroundings, although some techniques based on the gradient of pixel values and brightness are available, which may be more robust to wider cracks and more complex geometries.¹⁹ IPT approaches are fast and easily embedded in vision systems, but results are greatly affected by brightness, shading and resolution of images.²⁰ Standard rule-based feature crack detections were generally combined with different IPTs and different researchers have used several combinations of the available IPTs to achieve the optimal automated system so as to produce useful and real-time results, with a reliable level of precision and consistency. Conventional IPTs mostly deal with the pixels and regions inside the images directly and generate the regions of interest (RoIs). However, the results based on these methods heavily rely on the predefined parameters and are prone to the environmental effect in real-life situations. In addition, these methods all require some form of manual input.

However, ML-based approaches such as convolutional neural networks (CNNs), which are able to conduct highly nonlinear mapping that outputs the target features in a specified form based on the input image, show more potential in real-time detection of structural damages.^21–26 In this respect, deep neural networks (DNNs) have shown remarkable potential in image-based classification, detection and recognition.^27–33 DNNs have more layers and parameters than typical ML-based techniques, which promote more accurate object detection.³⁴ Furthermore, a fully convolutional network (FCN)-based algorithm has been developed to detect different types of concrete surface damage, such as cracks, spalling, efflorescence and holes, which is an end-to-end, pixel-to-pixel convolutional network for semantic segmentation.^12,35–38 The proposed FCN successfully detected the damage at pixel level and showed low levels of noise. However, the method required a large number of images to train a robust model and the training process can be laborious in terms of time as well.^33,39 Labelling the images manually is hugely time-consuming and is therefore another limitation of this method. CrackNet, an efficient architecture based on CNN, was proposed for automatic crack detection on three-dimensional (3D) asphalt surfaces with high accuracy at the pixel level.⁴⁰ However, CrackNet required substantial processing time and had more difficulty in detecting hairline cracks successfully.⁴⁰ The biggest challenge of automated crack detection is maintaining consistent accuracy under complex environments, such as rough asphalt. Subsequently, CrackNet-V was developed using a Visual Geometry Group (VGG) network to automatically detect cracks on 3D asphalt pavement images.⁴¹ Compared to CrackNet, CrackNet-V has a deeper architecture but fewer parameters, resulting in improved accuracy and efficiency. CrackNet-V is better at detecting fine cracks and is three to four times faster. However, performance on wide cracks is less accurate than that of CrackNet.

Determining the location of structural defects such as cracks from images using region-based methods has also achieved remarkable results.^42–45 However, they all require an external method (i.e. a selective search) to generate region proposals. To address this issue, a faster region-based convolutional neural network (faster R-CNN) was proposed to identify different types of damage: concrete cracks, steel corrosion (medium and high), bolt corrosion, steel delamination and other surface damages.^46–48 Moreover, Single Shot MultiBox Detector (SSD) is a CNN object detector to predict different object classes with bounding boxes by offsetting the predefined anchor.⁴⁹ SSD was used to detect eight types of road damages from smartphone images.⁵⁰ Most of the training images used for concrete crack detection as a particular type of damage in the reported studies were relatively clean and showed little disturbance, although some studies have been conducted to deal with complex background issues. For example, some algorithms were proposed for the identification of cracks with complex background information (e.g. hand marked information) on a steel structure, which however cannot predict the location of the cracks.^32,51 The extraction of cracks still required certain complex post-processing techniques. A more advanced ML algorithm is therefore required that can recognise cracks in concrete structures with complex backgrounds.

You Only Look Once (YOLO) is a single-stage object detection network that is capable of predicting the location of objects through bounding boxes with high detection accuracy at considerable speed.^52,53 The YOLO detection network was performed to detection potholes and damage with relatively clean concrete surface with promising results, showing the potential for more complicated scenarios.^54–56 With the introduction of anchor boxes in You Only Look Once version 2 (YOLO v2), it provided much better and stronger detection performance compared to its predecessor (i.e. You Only Look Once version 1 (YOLO v1)). To the best of the authors’ knowledge, no research has been conducted on detecting concrete cracks using YOLO v2 network from real-world images with complex backgrounds (e.g. handwriting scripts), which are commonly seen in reality. Inspired by the region-based CNN, this study proposed a modified YOLO v2 network for crack identification using real-world images, where concrete cracks and handwritten texts co-exist. No predefined filter was required, different from the traditional image-based methods. The main challenge to implement the YOLO v2 was to differentiate the concrete cracks from the crack-like features (e.g. handwriting scripts) which have similar patterns. In this respect, the anchor box was predefined carefully, and the algorithm was fine-tuned.

The contributions of this study are as follows.

An ad hoc YOLO v2 algorithm was implemented for fast concrete crack detection in real-world situations.

Experiments were conducted on different scenarios of images including those with poor lighting (e.g. images with bright and dark regions).

YOLO v2 and faster R-CNN were compared in terms of accuracy and computational cost.

Influence of labelling handwriting for network training on the overall crack detection accuracy was highlighted.

This article is organised as follows. The proposed methodology is introduced in section ‘Methodology of YOLO for crack detection’, followed by the network database and implementation details in section ‘Image dataset setup and implementation’. The evaluation results and detailed discussion are addressed in section ‘Experimental results and discussion’. Finally, a conclusion is presented in section ‘Conclusion and future work’.

Methodology of YOLO for crack detection

Figure 1 shows the overall schematic mechanism of the YOLO v2 object detector, where a single-stage object detection network is used as opposed to two-stage detectors such as faster R-CNN. A base feature extraction network CNN, acting as object detector, is implemented to generate bounding boxes through decoding the predicted outcomes from the CNN network. Bounding boxes with the highest intersection over union (IoU) will be selected as the detection results, so as to reduce the number of redundant boxes for more precise results. The structure of the YOLO v2 detector was ad hoc modified in this study to better detect cracks on concrete structures from real-world images in which handwriting scripts co-existed as the background. The implementation details of the YOLO v2 network are explained in the following sections.

Figure 1.

Concrete crack detection using YOLO v2 object detector.

The base extraction CNN (VGG16)

In this section, the fundamental architecture of the YOLO v2 detector network is presented, where the base extraction CNN was used to extract object features and pass them to the final YOLO layers. The detailed structure of the YOLO v2 is included in Table 1, consisting of convolution, rectified linear unit (ReLU), max pooling, batch normalisation and YOLO prediction layers. Many well-established networks for CNNs were available, such as ResNet-152, Alexnet, GoogleNet and Zeiler-Fergus network (ZF-net).^57,58 However, the original VGG16 series network⁵⁷ was used due to its excellent performance in improving AlexNet in ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC-2014). VGG16 was then modified in this study to generate the YOLO v2 network for the detection of a crack in images with a complex background (i.e. handwriting scripts). The architecture of the modified VGG16 network consisted of 39 layers: 13 convolutional, 15 ReLU, 5 max pooling, 2 dropouts, 3 fully connected and 1 softmax. To establish a damage detector that could distinguish concrete cracks from complex background, the layers of the VGG16 were replaced by YOLO detection layers after the feature extraction layer ReLU 5-3 in Table 1. A good feature extraction layer for YOLO v2 should have an output size between 8 and 16 times smaller than the input image to facilitate a good trade-off between spatial resolution and quality of output features. In this study, the output size after ReLU 5-3 is [28 28] and the original image size is [448 448]. Therefore, the downsampling factor is 16, which is desirable for YOLO v2 detection. The number of filters in the last convolution layer is calculated as numAnchors × (numClasses + numPredictionsPerAnchor) = 182.

Table 1.

Overall architecture of the proposed YOLO v2 (based on VGG16).

Layer	Type	Depth	Filter size	Stride	Paddings	Output
1	Convolution 1-1	64	3	1	1	[448 448]
2	ReLU 1_1	–	–	–	–	[448 448]
3	Convolution 1_2	64	3	1	1	[448 448]
4	ReLU 1_2	–	–	–	–	[448 448]
5	Max pooling 1	64	2	2	0	[224 224]
6	Convolution 2-1	128	3	1	1	[224 224]
7	ReLU 2_1	–	–	–	–	[224 224]
8	Convolution 2_2	128	3	1	1	[224 224]
9	ReLU 2_2	–	–	–	–	[224 224]
10	Max pooling 2	128	2	2	0	[112 112]
11	Convolution 3-1	256	3	1	1	[112 112]
12	ReLU 3_1	–	–	–	–	[112 112]
13	Convolution 3-2	256	3	1	1	[112 112]
14	ReLU 3_2	–	–	–	–	[112 112]
15	Convolution 3-3	256	3	1	1	[112 112]
16	ReLU 3_3	–	–	–	–	[112 112]
17	Max pooling 3	256	2	2	0	[56 56]
18	Convolution 4-1	512	3	1	1	[56 56]
19	ReLU 4_1	–	–	–	–	[56 56]
20	Convolution 4-2	512	3	1	1	[56 56]
21	ReLU 4_2	–	–	–	–	[56 56]
22	Convolution 4-3	512	3	1	1	[56 56]
23	ReLU 4_3	–	–	–	–	[56 56]
24	Max pooling 4	512	2	2	0	[28 28]
25	Convolution 5-1	512	3	1	1	[28 28]
26	ReLU 5-1	–	–	–	–	[28 28]
27	Convolution 5-2	512	3	1	1	[28 28]
28	ReLU 5-2	–	–	–	–	[28 28]
29	Convolution 5-3	512	3	1	1	[28 28]
30	ReLU 5-3	–	–	–	–	[28 28]
31	YOLOv2_Conv 1	512	3	1	1	[28 28]
32	YOLOv2_Batch 1	–	–	–	–	[28 28]
33	YOLOv2_ReLU 1	–	–	–	–	[28 28]
34	YOLOv2_Conv 2	512	3	1	1	[28 28]
35	YOLOv2_Batch 2	–	–	–	–	[28 28]
36	YOLOv2_ReLU 2	–	–	–	–	[28 28]
37	YOLOv2_ClassConv	182	1	1	0	[28 28]
38	YOLOv2_Transform	–	–	–	–	–
39	YOLOv2_Output	–	–	–	–	–

YOLO v2: You Only Look Once version 2.

Figure 2 illustrates the schematic structure of the YOLO v2 detector for defect detection. The most important aspect of this network is that feature maps are computed through the base CNN and predicted by the YOLO layers. The detailed operations of the layers used in CNN can be found in the study by Ziou and Tabbone.²⁰ The YOLO transform layers are utilised to transform the raw CNN output into a form required for object detection and also to improve the stability of the network by constraining the location predictions of the bounding box to fall within the boundary of the ground truth.

Figure 2.

Schematic architecture of the YOLO v2 object detector network.

YOLO v2 anchor box

The input images were divided into a number of grid cells (28 × 28) by the YOLO v2 detector, and a number of object regions were proposed at each grid cell location with a defined number of reference boxes called anchors consisting of different scales and aspect ratios, as shown in Figure 3. The centre of the anchor boxes (the blue dots in Figure 3) fell within the grid cell. Therefore, each grid cell location in the images had k anchors associated with it. The combination of these anchors in each position was determined by the predefined geometries of the bounding boxes. Each bounding box included five parameters, that is [x, y, w, h] which specify the top left corner coordinate (x, y) and the width and height of the boxes, as well as a box confidence score of containing an object. Three attributes were predicted by the YOLO v2 network, namely IoU, anchor box offsets and class probability. An IoU concept was adopted to calculate the object score of each anchor box, whereas the bounding box offsets refined the location of the box. Each anchor box was assigned a corresponding class label. Due to the fact that the YOLO v2 network could make multiple detections for the same object at each grid cell, non-maximal suppression was applied to remove all bounding boxes predicted with low confidence scores and therefore to retain just one box with the highest IoU.

Figure 3.

Example of grid cell divisions.

YOLO v2 loss function

The YOLO v2 network was trained end-to-end using stochastic gradient descent with momentum (SGDM). As mentioned in the previous section, YOLO v2 predicts multiple bounding boxes per grid cell. However, only one box was required, which was determined by the highest IoU with the ground truth. The probability of objects in an image patch was updated during the training process to minimise the difference through offsetting. YOLO v2 introduced sum-squared errors between the predictions and the ground truth to calculate the loss for bounding box refinement. The geometrical coordinate and the confidence score of a bounding box were then predicted, and the network could be trained to map the predicted box to a ground truth box through the total loss function as

\begin{matrix} Total loss = classification loss + localisation loss + confidence loss \end{matrix}

(1)

\begin{matrix} Classification loss = \sum_{i = 0}^{S^{2}} I_{i}^{obj} \sum_{c \in classes} {(p_{i} (c) - {\hat{p}}_{i} (c))}^{2} \end{matrix}

(2)

where $I_{i}^{obj} = 1$ if an object is in cell i, otherwise = 0. ${\hat{p}}_{i} (c)$ represents the conditional class probability for class c in cell i

\begin{matrix} Localisation loss = λ_{coord} \sum_{i = 0}^{S^{2}} \sum_{j = 0}^{B} I_{ij}^{obj} [{(x_{i} - {\hat{x}}_{i})}^{2} + {(y_{i} - {\hat{y}}_{i})}^{2}] \\ + λ_{coord} \sum_{i = 0}^{S^{2}} \sum_{j = 0}^{B} I_{ij}^{obj} [{(\sqrt{w_{i}} - \sqrt{{\hat{w}}_{i}})}^{2} + {(\sqrt{h_{i}} - \sqrt{{\hat{h}}_{i}})}^{2}] \end{matrix}

(3)

where $I_{ij}^{obj} = 1$ if the jth bounding box detects an object in cell i, otherwise = 0. $λ_{coord} = 5$ to increase the weight for the loss in the bounding box coordinates in order to put more emphasis on the box accuracy

\begin{matrix} Confidence loss = \sum_{i = 0}^{S^{2}} \sum_{i = 0}^{B} I_{i j}^{o b j} {(C_{i} - {\hat{C}}_{i})}^{2} + λ_{n o o b j} \sum_{i = 0}^{S^{2}} \sum_{i = 0}^{B} I_{i j}^{n o o b j} {(C_{i} - {\hat{C}}_{i})}^{2} \end{matrix}

(4)

where $I_{ij}^{noobj}$ is the complement of $I_{ij}^{obj}$ . ${\hat{C}}_{i}$ represents the confidence score of the box j in cell i. $λ_{noobj} = 0.5$ to weight down the non-object loss to remedy the class imbalance issue, as most boxes do not include any objects (e.g. background).

Image dataset setup and implementation

Training and testing datasets

The YOLO v2 was trained in one step and its detection performance was evaluated by the generated test dataset. A single network was created for detection using the pre-trained network VGG16 as the base feature extraction network.

To generate a dataset of training and testing images, real-world images 3264 × 2448 pixels in size, taken by different digital single-lens reflex cameras at different angles and distances with complex background information from a variety of concrete bridges, were used and cropped to 448 × 448 pixels to increase the number of training and testing samples, while maintaining suitable resolution for concrete crack detection. These original full-scale images with handwriting scripts marked on the concrete surface for the cracks were all taken by inspectors during annual visual inspections. In total, 3010 images were generated for the database to detect objects as cracks and handwriting for both training and testing processes. Then 80% of the image data were randomly selected and used for the YOLO v2 training, while the remaining 20% of the images were used for the testing dataset.

The Image Labeler app in MATLAB was implemented to create the labelled ground truth data for all the previously prepared dataset. Examples of annotated images are shown in Figure 4. Each bounding box is in the format [x y w h] as described previously. Horizontal flipping of sub-images as data augmentation technique was used in this study to not only increase the number of data for training, but also help reduce a possibility of overfitting by increasing the diversity of the data, thereby creating a more robust classifier by preventing the network from memorising exact details of what was being trained.^58,59 Furthermore, horizontal flipping was also beneficial in situations where the available dataset is unbalanced. In addition, YOLO v2 featured batch normalisation on all of the convolutional layers, which helps regularise the model and avoids overfitting problem.⁵³

Figure 4.

Examples of ground truth images with boxes and labels.

The proportions of crack and handwriting in the dataset were kept similar to make sure that there was no bias towards any of the detection classes. Accordingly, the image dataset included three main image types: containing both crack and handwriting, only crack and only handwriting. Examples of the training samples are shown in Figure 5.

Figure 5.

Examples of training samples.

Implementation

All experiments were conducted in MATLAB 2019a, CUDA 9.2 on a computer with an Intel^® Core™ i7-6700 @ 3.40 GHz CPU, 64 GB installed memory (RAM) and a 4 GB GDDR5 memory NVIDIA Quadro P1000 graphics processing unit. This study used the Image Processing Toolbox™, Neural Network Toolbox™ and Computer Vision System Toolbox™ in MATLAB.

The YOLO v2 network was trained end-to-end with SGDM loss function by training one image per iteration. The hyperparameters selected in this study are summarised in Table 2. A momentum of 0.9 was chosen, which was used to update the parameters between iterations. A full pass through the entire training dataset was denoted as an epoch. In this study, the MaxEpochs of 200 was chosen due to the trade-off between detection accuracy and computational cost. The number of iterations in each epoch was always equal to the number of training images since the YOLO v2 network used the ‘image-centric’ training scheme. In this study, a mini-batch size of 10 was used for the network training, which controlled multiple RoIs that were sampled from each training image per iteration to evaluate the gradient of the loss function and to update the weights. Moreover, the training options were set to ensure that the network had a weight decay of 0.0005 and a learning rate of 0.001.

Table 2.

Table of confusion for prediction in a bounding box.

Actual class (true/false)	Predicted class (positive/negative)
Actual class (true/false)	Crack/handwriting	Non-crack/non-handwriting
Crack/handwriting	True positive (TP)	False negative (FN)
Non-crack/non-handwriting	False positive (FP)	True negative (TN)

The performance of the YOLO v2 was evaluated by adopting precision and recall concepts, based on a measure of relevance. As shown in Table 2 and equation (5), precision, also known as positive predictive value, was defined as the ratio of true positive (TP) instances to all positive detection results. Recall, also called sensitivity, was denoted by the ratio of TP to the sum of TP and false negative (FN). The concept of average precision (AP), defined as the area under the precision and recall curves (P-R curves), was calculated to assess the robustness of the proposed YOLO v2. The AP provided a single value demonstrating the capability of the detector to make correct classifications (Precision) and to find all corresponding objects (Recall). The mean average precision (mAP) was also computed, determined by the average value between APs for different classes

\begin{matrix} Precision = \frac{TP}{TP + FP}, Recall = \frac{TP}{TP + FN} \end{matrix}

(5)

Miss rate and false positive per image (FPPI) concepts were also adopted to generate missing rate curves for evaluating the performance of the YOLO v2. Log-average miss rate metrics were computed for both classes, indicating the percentage of objects missed during detection. In theory, a lower average miss rate signified fewer objected being missed

\begin{matrix} Miss rate = \frac{FN}{TP + FN} \end{matrix}

(6)

As the cracks and handwriting contained in the dataset were mostly elongated and of irregular shape in this study, a predefined number of anchor boxes was assigned for the network evaluation at each grid cell location, where redundant boxes were removed by the use of non-maximal suppression computed to discard prediction results with low probability. This allowed the network to include the predicted boxes that overlapped with the ground truth more accurately. In this way, the predicted bounding box that overlapped most heavily with the ground truth box had the highest IoU value. Examples of the predicted test images are shown in Figure 6 including the IoUs with the ground truth boxes.

Figure 6.

Examples of test results for images 448 × 448 pixels in size with confidence levels.

A parametric study was conducted to estimate anchor boxes using a clustering estimation method. The selection of appropriate anchor boxes provides important parameters for detecting elongated concrete cracks with a complex background. The size, aspect ratios and number of anchor boxes have significant impacts on the efficiency and accuracy of the detection network. For this reason, anchor boxes were estimated based on the training dataset using the K-Medoids clustering algorithm that attempts to minimise the distance between points in a group and a point (medoid) defined as the centre of that group.⁵³ The anchor boxes were then clustered with the custom IoU distance metric (equation (7)), which resulted in boxes of similar aspect ratio and size being clustered together

\begin{matrix} distance (box, medoid) = 1 - IoU (box, medoid) \end{matrix}

(7)

The number of anchors was chosen carefully. This is because the selection of this hyperparameter has a significant effect on the overall detection accuracy. As a result, the number of anchors was selected using empirical analysis, based on the mean IoU of the boxes in each cluster. The mean IoU was chosen to be greater than 0.8 to ensure that anchor boxes overlapped well with the boxes in the training data. In reality, increasing the number of anchors may increase the mean IoU measure, but this can result in an increase in computational cost and can lead to overfitting. Therefore, a trade-off between a larger number of anchors and a corresponding higher mean IoU is required. The K-Medoids clustering algorithm was applied to the training datasets with various numbers of anchors against the corresponding average IoU with the closest medoid, as shown in Figure 7. Using 26 anchors provided an average IoU above 0.8 and only minor improvement was observed after 40 anchors. As a result, the detector was trained and evaluated using 26 anchors. Given these results, the number of anchors and the mean IoU, where 26 anchors had the mean IoU of 0.80826, were adopted in this study to achieve a good balance between accuracy and computational cost.

Figure 7.

Selection of number of anchors in the training samples.

The original labelled ground truth box distribution should be studied to better understand the range of object sizes presented in the training dataset, where a few groups of objects may have similar size and shape, but they are dispersed. Therefore, through the use of the distance metric as described previously by grouping similar ground truth boxes together, the data were classified and grouped in accordance to size and shape into groups using 26 clusters as shown in Figure 8. In this way, boxes which had similar aspect ratios and size were clustered using the IoU distance metric, where the sizes of the predefined anchor boxes estimated and chosen automatically fitted well with the data. Consequently, the automatically generated anchor boxes were used in the training of the proposed detector.

Figure 8.

Visualisation of ground truth box distribution in the training data after K-medoids with 26 clusters.

Parametric study: dataset size

In order to determine the sufficiency of the dataset size for training, three different training datasets (A, B and C) were generated with 1204, 1806 and 2408 images, respectively. The same dataset with 602 images was used for testing the three trained networks. As can be seen from results in Table 3, dataset C demonstrates the best performance with the mAP of 77% compared with datasets A and B, a result which could be attributed to the larger and more diverse training set.

Table 3.

Size effect of the dataset.

Dataset	Training data	AP % (crack)	AP % (handwriting)	mAP %	Training time (h)	Inference speed (s/image)
A	1204	64	70	67	26	0.13
B	1806	72	72	72	39	0.11
C	2408	76	78	77	48	0.17

AP: average precision.

The P-R curves for these training datasets (A, B and C) were plotted in Figure 9 for crack (left) and handwriting (right). The corresponding APs which were calculated by integrating the area under the curves were 76% and 78% for crack and handwriting, respectively, based on dataset C, the highest among three dataset sizes. In addition, the log miss rate over FPPI curves for these training datasets was plotted in Figure 10 for crack (left) and handwriting (right). The corresponding log average miss rate was 34% and 30% for crack and handwriting, respectively, based on dataset C, which are the lowest among three dataset sizes. This also means that a certain number of objects were missed during the network testing, which could be due to tiny cracks on the blurred images.

Figure 9.

Precision and recall curves for crack (left) and handwriting (right).

Figure 10.

Miss rate curves for crack (left) and handwriting (right).

Experimental results and discussion

Full-scale image evaluation

Three new full-scale images (3264 × 2448 pixels) were utilised to evaluate the performance of the proposed YOLO v2 detector. The images were photographed by bridge engineers with handwriting scripts marked on them. Before images were assessed by the trained detector, each full-sized image was scanned with a window size of 448 × 448 pixels to provide the same level of detection accuracy. Each scanned image patch was then input into the trained network and output with corresponding bounding boxes for crack and handwriting with corresponding confidence scores as shown in Figure 11. It was evident that the proposed YOLO v2 could successfully localise concrete cracks despite the potential interference of pre-existing handwritten background. However, some undetected objects (FN instances) were shown in the images, which in general were acceptable because of the images with complex background (e.g. crack-like handwriting, stains). Moreover, the images were taken in various positions of bridges, and the various distances from the concrete surface where the photographs were taken could also have an impact on the detection results (e.g. lighting, shadow, object angle). The non-uniform sunlight also made it difficult for the detector to locate the real tiny cracks in the dirty concrete surface. However, these disturbing factors did not cause significant numbers of FP instances in this study and could be ameliorated by training the network with a larger and more diverse dataset.

Figure 11.

Detected crack damages and handwriting scripts; three new full-scale test images (left) and labelled images (right).

However, handwriting scripts presented in the images were in general correctly identified using the proposed YOLO v2. At some locations, the handwriting was not detected, which could be attributed to the aforesaid possible reasons. These false detections could also be mitigated by developing a larger dataset containing a greater variety of situations for more accurate detection.

Network performance on images with clustered background and poor lighting

In order to assess the robustness of the proposed network under dark environment, the trained YOLO v2 was evaluated with two images taken under poor lighting condition. As can be seen from Figure 12 (right), the distribution of the crack can be clearly captured by the trained YOLO v2. However, the results also showed that the evaluated images experienced more false detection. Therefore, the lighting condition and complexity of the background have a significant effect on the overall accuracy of the network for concrete crack detection. However, it is understood that the provision of lighting difference (e.g. lux) will be more informative for a quantitative study, as illuminance can be calculated, which measures the amount of light falling on a surface. However, since the images in this study were taken by different inspectors at different times, it is impossible to conduct such lighting measurements, where the specifications of the camera with which the image was taken must be available in advance, for example, shutter opening time, sensitivity of the charged coupled device (CCD), analog-to-digital units. Therefore, the lighting difference can only be perceived qualitatively in Figure 12. For example, the darker areas are those in the shaded regions. Quantitative information in terms of lighting difference will be a focus in the future study with well-controlled experimental settings.

Figure 12.

Detected crack damages and handwriting scripts under poor lighting condition; two full-scale test images (left) and labelled images (right).

Confidence levels of bounding boxes

For a better and more straightforward application of the proposed algorithm, a grading system (i.e. excellent, good, acceptable and bad) as shown in Table 4 was defined to group these prediction results based on the predicted confidence scores, where 80% confidence score was selected as a threshold due to the fact that the bounding boxes in general have high confidence scores to be correct. More details regarding the confidence level grading system can be found in Deng et al.⁶⁰ As can be seen from Table 5 for images shown in Figure 11, it can be concluded that the overall performance of the proposed network is satisfactory for all the three images and there is no bad prediction for the crack, as there are no incorrectly detected objects with high confidence scores.

Table 4.

Grading system used in Table 5.

Abbreviation	Positive instances	Description	Grading
CH	TP	Correct prediction with high confidence scores (>80%)	Excellent
CL	TP	Correct prediction with low confidence scores (<80%)	Good
IL	FP	Incorrect prediction with low confidence scores (<80%)	Acceptable
IH	FP	Incorrect prediction with high confidence scores (>80%)	Bad

TP: true positive; FP: false positive.

Table 5.

Relative confidence scores for images shown in Figure 11.

Image	Object	CH%	CL%	IL%	Total%
a	Crack	0	89	11	100
a	Handwriting	6	87	7	100
b	Crack	15	77	8	100
b	Handwriting	7	91	2	100
c	Crack	0	100	0	100
c	Handwriting	4	96	0	100

CH: correct prediction with high confidence scores; CL: correct prediction with low confidence scores; IH: incorrect prediction with high confidence scores; IL: incorrect prediction with low confidence scores.

It is worth mentioning that negative outcomes (i.e. TNs and FNs) are not shown in Table 4 as they cannot be directly quantified by counting the number of bounding boxes with corresponding confidence scores. In particular, an FN instance was normally the worst case for crack detection, because it meant that some crack regions were overlooked by the network, potentially leading to underestimation of bridge safety. As observed in Figure 11, certain tiny cracks were still undetected. Such an issue could be ameliorated by increasing the size of the training datasets so that the network was exposed to a wider range of scenarios and possibilities.

Comparison between faster R-CNN and YOLO v2

To evaluate the performance of the proposed YOLO v2, faster R-CNN was applied and trained on the same dataset C for comparison, where the pre-trained VGG16 was also used for more accurate comparison. The detected classes included crack and handwriting as well. As can be seen in Table 6, the mAPs are 74.5% and 77% for faster R-CNN and YOLO v2, respectively, which are comparable in terms of detection accuracy. However, the training time required by the faster R-CNN is more than fourfold that of YOLO v2. Therefore, YOLO v2 was superior, with fewer false detections at the remarkably faster training and inference speed. As can be seen from Figure 13 for two more new images, the faster R-CNN evinced more incorrect detection of crack compared to that of YOLO v2. The relative confidence levels for images shown in Figure 13 are summarised in Table 7. Although the faster R-CNN has a higher percentage of correct prediction with high confidence scores (CH) compared to that of YOLO v2, which actually comes at the cost of long training time, the percentage of incorrect detection with the high confidence score of the faster R-CNN in general is still higher than that of YOLO v2, further reaffirming that YOLO v2 is more reliable.

Table 6.

Comparison of YOLO v2 with faster R-CNN algorithm on crack detection.

Model	Network	AP (%)		mAP (%)	Training time (days)	Inference testing speed (s/image)
Model	Network	Crack	Handwriting	mAP (%)	Training time (days)	Inference testing speed (s/image)
Faster R-CNN	VGG16	74	75	74.5	9	0.23
YOLO v2	VGG16	76	78	77	2	0.17

AP: average precision; R-CNN: region-based convolutional neural network; YOLO v2: You Only Look Once version 2; mAP: mean average precision.

Figure 13.

Detected crack damages and handwriting scripts; two new full-scale test images with YOLO v2 (left) and faster R-CNN (right).

Table 7.

Relative confidence level for images shown in Figure 13.

YOLO v2						Faster R-CNN
Image	Object	CH%	CL%	IH%	IL%	CH%	CL%	IH%	IL%	Sum%
a	Crack	0	88	0	12	29	71	0	0	100
a	Handwriting	3	97	0	0	52	45	0	3	100
b	Crack	0	100	0	0	40	30	10	20	100
b	Handwriting	0	97	0	3	56	44	0	0	100

YOLO v2: You Only Look Once version 2; R-CNN: region-based convolutional neural network; CH: correct prediction with high confidence scores; CL: correct prediction with low confidence scores; IH: incorrect prediction with high confidence scores; IL: incorrect prediction with low confidence scores.

Effect of labelling handwriting for training

An experiment was further conducted to investigate the influence of labelling handwriting as an object class for network training on the overall crack detection accuracy. In detail, the YOLO v2 network was retrained with the same dataset C without considering handwriting as an object class. As can be seen from Table 8, the training time was threefolds with a significantly increased MaxEpochs value from 200 to 500 for considering crack only to reach similar crack detection accuracy; otherwise, the network cannot recognise any crack from images. It is thus obvious that considering handwriting as one kind of complex background and explicitly labelling it as an individual object class during the network training would avoid potential overlook of concrete crack and significantly improve the efficiency of correctly identifying the crack from complex background.

Table 8.

Comparison of YOLO v2 with/without handwriting as object class on crack detection.

Class labels	MaxEpochs	AP (%)		mAP (%)	Training time (days)	Inference testing speed (s/image)
Class labels	MaxEpochs	Crack	Handwriting	mAP (%)	Training time (days)	Inference testing speed (s/image)
Crack and handwriting	200	76	78	77	2	0.17
Crack only	500	76	–	76	6	0.14

YOLO v2: You Only Look Once version 2; AP: average precision; mAP: mean average precision.

It is also understood that bounding box techniques such as YOLO v2 provide an immediate mean for engineers to evaluate the structural soundness, so as to determine whether the structures require further assessments. This is usually the first and most important step to identify the conditions of a structure. However, the current trend also leads to the semantic segmentation, such as FCN, to allow pixel-level classification and damage quantification. Both bounding box techniques and FCN are comparable in terms of object detection. It is acknowledged that FCN is good for object localisation, but it may take more time to infer and bear higher computational costs due to the fact that its detection is at the pixel level. However, YOLO v2 network is faster and good for real-time monitoring, as it is at the image level while it is less precise as compared to FCN. It is therefore suggested that once the structure is quickly identified to be safety suspicious based on the outcomes from the YOLO v2 bounding boxes, semantic segmentation should come into the picture, allowing for more precise structural assessments and quantification.

Conclusion and future work

This study presented a novel application of YOLO v2 to detect the two most commonly seen structural features in onsite collected images, that is concrete crack and handwriting script, which are quite similar in shape and easily misclassified. Experiments were carried out to verify the ad hoc YOLO v2 trained with a dataset with these two different classes of objects. The results showed that although there were some FNs in the three full-scale evaluation images, the majority of the concrete cracks were correctly identified. As a result of that finding, it is considered that the proposed YOLO v2 network can be used to determine crack locations and efficiently differentiate cracks from co-existing handwriting script marked on the concrete surface during onsite inspection. A similar object detection network, faster R-CNN, was applied to the same dataset for comparison of the network performance. The results showed that YOLO v2 had similar accuracy in crack detection, but demonstrated much faster processing, indicating the potential for real-time crack detection using UAVs. As concrete cracking is usually the first sign of structural damage, this work will also allow us to track the progression of cracks as the crack develops over time, which will trigger early alarm for the bridge failures. Finally, the proposed algorithm can be extended to any other surface defects, for example, corrosion, spalling.

Although YOLO v2 has proven to be robust in concrete crack detection with relatively fast speed, the algorithm still requires further improvement to achieve real-time concrete crack assessment. First, further study on the influence of different hyperparameters used in the training stage should be conducted to achieve better performance of the network. Second, based on the outcomes from YOLO v2 where crack location can be determined, semantic segmentation algorithm, such as FCN should be deployed to quantify the damage, which will be another focus of future study. Moreover, the global locations of concrete cracks are to be determined in the future with the aid of UAVs with global positioning system (GPS) coordination when the photo is taken. A bridge information modelling (BrIM) platform will also be set up to integrate image database, by which concrete cracks can be identified and visualised globally, highlighting the corresponding location of the crack on the bridge.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iD

Ye Lu

References

Yamaguchi

Hashimoto

. Practical image measurement of crack width for real concrete structure. Electron Commun Jpn 2009; 92(10): 1–12.

Jahanshahi

Masri

Padgett

, et al. An innovative methodology for detection and quantification of cracks through incorporation of depth perception. Mach Vis Appl 2013; 24(2): 227–241.

Koch

Georgieva

Kasireddy

, et al. A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure. Adv Eng Inform 2015; 29(2): 196–210.

Swenson

Ingraffea

. The collapse of the Schoharie Creek Bridge: a case study in concrete fracture mechanics. In: Bažant

(ed.) Current trends in concrete fracture research. Dordrecht: Springer, 1991, pp. 73–92.

Abdel-Qader

Abudayyeh

Kelly

. Analysis of edge-detection techniques for crack identification in bridges. J Comput Civ Eng 2003; 17(4): 255–263.

Prasanna

Dana

Gucunski

, et al. Computer-vision based crack detection and analysis. In: Proceedings of the sensors and smart structures technologies for civil, mechanical, and aerospace systems, San Diego, CA, 6 April 2012, vol. 8345, p. 834542. Bellingham, WA: International Society for Optics and Photonics.

Jang

, et al. Bridge inspection robot system with machine vision. Automat Constr 2009; 18(7): 929–941.

Lim

Sheng

. A robotic crack inspection and mapping system for bridge deck maintenance. IEEE Trans Autom Sci Eng 2014; 11(2): 367–378.

Lei

Ren

Wang

, et al. Design of a new low-cost unmanned aerial vehicle and vision-based concrete crack inspection method. Struct Health Monit 2020; 19(6): 1871–1883.

10.

Reddy

Indragandhi

Ravi

, et al. Detection of cracks and damage in wind turbine blades using artificial intelligence-based image analytics. Measurement 2019; 147: 106823.

11.

Adeli

. Machine learning–neural networks, genetic algorithms and fuzzy systems. Kybernetes 1999; 28: 317–318.

12.

Yang

, et al. Automatic pixel-level crack detection and measurement using fully convolutional network. Comput Aid Civ Infrastruct Eng 2018; 33(12): 1090–1109.

13.

Adhikari

Moselhi

Bagchi

. Image-based retrieval of concrete crack properties for bridge inspection. Automat Constr 2014; 39: 180–194.

14.

Chen

Cao

, et al. A self organizing map optimization based image recognition and processing model for bridge crack inspection. Automat Constr 2017; 73: 58–66.

15.

Chen

Jahanshahi

, et al. A texture-based video processing methodology using Bayesian data fusion for autonomous crack detection on metallic surfaces. Comput Aid Civ Infrastruct Eng 2017; 32(4): 271–287.

16.

Tong

Guo

Ling

, et al. A new image-based method for concrete bridge bottom crack detection. In: Proceedings of the international conference on image analysis and signal processing, Wuhan, China, 21–23 October 2011. New York: IEEE.

17.

Yeum

Dyke

. Vision-based automated crack detection for bridge inspection. Comput Aid Civ Infrastruct Eng 2015; 30(10): 759–770.

18.

Zhang

Elaksher

. An unmanned aerial vehicle-based imaging system for 3D measurement of unpaved road surface distresses. Comput Aid Civ Infrastruct Eng 2012; 27(2): 118–129.

19.

Yamaguchi

Hashimoto

. Automated crack detection for concrete surface image using percolation model and edge information. In: Proceedings of the IECON 2006: 32nd annual conference on IEEE industrial electronics, Paris, 6–10 November 2006, pp. 3355–3360. New York: IEEE.

20.

Ziou

Tabbone

. Edge detection techniques: an overview. Pattern Recogn Image Anal 1998; 8: 537–559.

21.

Cha

Choi

Büyüköztürk

. Deep learning-based crack damage detection using convolutional neural networks. Comput Aid Civ Infrastruct Eng 2017; 32(5): 361–378.

22.

Cireşan

Meier

Masci

, et al. Flexible, high performance convolutional neural networks for image classification. In: Proceedings of the twenty-second international joint conference on artificial intelligence, Barcelona, 19–22 July 2011. New York: ACM.

23.

Huang

Zhang

. Deep learning based image recognition for crack and leakage defects of metro shield tunnel. Tunnell Undergr Space Technol 2018; 77: 166–176.

24.

Kim

Ahn

Cho

, et al. Comparative analysis of image binarization methods for crack identification in concrete structures. Cem Concrete Res 2017; 99: 53–61.

25.

Molina-Cabello

Laque-Baena

López-Rubio

, et al. Vehicle type detection by ensembles of convolutional neural networks operating on super resolved images. Integr Comput Aid Eng 2018; 25: 1–13.

26.

Wang

Bai

. Regional parallel structure based CNN for thermal infrared face identification. Integr Comput Aid Eng 2018; 25(3): 247–260.

27.

Chen

Jahanshahi

. NB-CNN: deep learning-based crack detection using convolutional neural network and Naïve Bayes data fusion. IEEE Trans Ind Electron 2017; 65(5): 4392–4400.

28.

Kang

Cha

. Autonomous UAVs for structural health monitoring using deep learning and an ultrasonic beacon system with geo-tagging. Comput Aid Civ Infrastruct Eng 2018; 33(10): 885–902.

29.

Yuan

Zhang

, et al. Unified vision-based methodology for simultaneous concrete defect detection and geolocalization. Comput Aid Civ Infrastruct Eng 2018; 33(7): 527–544.

30.

Liang

. Image-based post-disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization. Comput Aid Civ Infrastruct Eng 2019; 34(5): 415–430.

31.

Liao

. On-line vehicle routing problems for carbon emissions reduction. Comput Aid Civ Infrastruct Eng 2017; 32(12): 1047–1063.

32.

Bao

Chen

, et al. Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumer-grade camera images. Struct Health Monit 2019; 18(3): 653–674.

33.

Zhao

Zhou

. Automatic pixel-level multiple damage detection of concrete structure using fully convolutional network. Comput Aid Civ Infrastruct Eng 2019; 34(7): 616–634.

34.

Zhang

Yang

Zhang

, et al. Road crack detection using deep convolutional neural network. In: Proceedings of the IEEE international conference on image processing (ICIP), Phoenix, AZ, 25–28 September 2016. New York: IEEE.

35.

Alipour

Harris

Miller

. Robust pixel-level crack detection using deep fully convolutional neural networks. J Comput Civ Eng 2019; 33(6): 04019040.

36.

Dung

Ahn

. Autonomous concrete crack detection using deep fully convolutional neural network. Automat Constr 2019; 99: 52–58.

37.

Shelhamer

Long

Darrell

. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, 7–12 June 2015. New York: IEEE.

38.

Xue

. A fast detection method via region-based fully convolutional neural networks for shield tunnel lining defects. Comput Aid Civ Infrastruct Eng 2018; 33(8): 638–654.

39.

Liu

Cao

Wang

, et al. Computer vision-based concrete crack detection using U-net fully convolutional networks. Automat Constr 2019; 104: 129–139.

40.

Zhang

Wang

KCP

, et al. Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network. Comput Aid Civ Infrastruct Eng 2017; 32(10): 805–819.

41.

Fei

Wang

KCP

Zhang

, et al. Pixel-level cracking detection on 3D asphalt pavement images through deep-learning-based CrackNet-V. IEEE Trans Intell Transp Syst 2020; 21: 273–284.

42.

Adeli

Hung

. A concurrent adaptive conjugate gradient learning algorithm on MIMD shared-memory machines. Int J High Perform Comput Appl 1993; 7(2): 155–166.

43.

Girshick

Donahue

Darrell

, et al. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Columbus, OH, 23–28 June 2014. New York: IEEE.

44.

Girshick

. Fast R-CNN. In: Proceedings of the IEEE international conference on computer vision, Santiago, Chile, 7–13 December 2015. New York: IEEE.

45.

Zhang

Ren

, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 2015; 37(9): 1904–1916.

46.

Cha

Choi

Suh

, et al. Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Comput Aid Civ Infrastruct Eng 2018; 33(9): 731–747.

47.

Ren

Girshick

, et al. Faster R-CNN: towards real-time object detection with region proposal networks. In: Proceedings of the advances in neural information processing systems, Montreal, QC, Canada, 7–12 December 2015. New York: IEEE.

48.

Wang

Zhao

, et al. Automatic damage detection of historic masonry buildings based on mobile deep learning. Automat Constr 2019; 103: 53–66.

49.

Liu

Anguelov

Erhan

, et al. SSD: single shot multibox detector. In: Proceedings of the European conference on computer vision, Amsterdam, 8–16 October 2016. New York: Springer.

50.

Maeda

Sekimoto

Seto

, et al. Road damage detection and classification using deep neural networks with smartphone images. Comput Aid Civ Infrastruct Eng 2018; 33(12): 1127–1141.

51.

Zhang

, et al. Identification framework for cracks on a steel structure surface by a restricted Boltzmann machines algorithm based on consumer-grade camera images. Struct Control Health Monit 2018; 25(2): e2075.

52.

Redmon

Divvala

Girshick

, et al. You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, 27–30 June 2016. New York: IEEE.

53.

Redmon

Farhadi

. YOLO9000: better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, 21–26 July 2017. New York: IEEE.

54.

Murao

Nomura

Furuta

, et al. Concrete crack detection using UAV and deep learning, 2019, http://s-space.snu.ac.kr/handle/10371/153261

55.

Suong

Kwon

. Detection of potholes using a deep convolutional neural network. J Univ Comput Sci 2018; 24(9): 1244–1257.

56.

Zhang

Chang

Jamshidi

. Bridge damage detection using single-stage detector and field inspection images, 2018, https://arxiv.org/vc/arxiv/papers/1812/1812.10590v1.pdf

57.

Simonyan

Zisserman

. Very deep convolutional networks for large-scale image recognition, 2014, https://arxiv.org/pdf/1409.1556.pdf

58.

Zeiler

Fergus

. Visualizing and understanding convolutional networks. In: Proceedings of the European conference on computer vision, Zürich, 6–12 September 2014. New York: Springer.

59.

Krizhevsky

Sutskever

Hinton

. ImageNet classification with deep convolutional neural networks. In: Proceedings of the advances in neural information processing systems, Lake Tahoe, NV, 3–8 December 2012. New York: ACM.

60.

Deng

Lee

VCS

. Concrete crack detection with handwriting script interferences using faster region-based convolutional neural network. Comput Aid Civ Infrastruct Eng 2019; 35: 373–388.