Semi-supervised learning for concrete defect segmentation from images

Abstract

Deep learning-based concrete defect detection has emerged as a promising technology to overcome the limitations of manual visual inspection, which is often subjective, laborious, and time-consuming. However, most existing methods rely heavily on manual labels and adopt a fully supervised learning approach, which hinders their practical application in engineering. To tackle these concerns, we proposed a semi-supervised learning model, termed Semi-SegDefect. Our proposal utilizes a mean teacher architecture to assign pseudo-labels to the pixels of unlabeled images and performs pixel-level contrastive learning on a sparse set of hard negative pixels to achieve segmentation boundaries with the highest possible accuracy. Test results demonstrate that Semi-SegDefect outperforms fully supervised models significantly, especially when trained with limited labeled data. This method shows great promise for enhancing the accuracy and scalability of concrete defect segmentation and can contribute to the advancement of the construction industry.

Keywords

Deep learning semantic segmentation semi-supervised learning contrastive learning concrete defect

Introduction

Concrete structures are vulnerable to various defects caused by factors such as material aging, thermal stress, overloading, and inadequate maintenance, which can result in loss of functionality and safety hazards in infrastructure systems.^1–3 Regular inspections are crucial for evaluating the current condition of structures and guiding decisions about defect maintenance. Although expert-dependent visual inspection has traditionally been used for detecting defects in concrete structures, this method tends to be time-consuming, labor-intensive, subjective, and prone to errors in many situations.^4,5

Recently, the advancements in visual sensing technologies and machine learning have revolutionized operations across various industries. The civil infrastructure inspection and monitoring are also undergoing a transformation from manual visual inspection to data-driven automation, with the potential to achieve more efficient and automated infrastructure maintenance.⁶ Machine learning provides advanced mathematical frameworks and algorithms capable of discovering and modeling structural performance and conditions from rich monitoring data sources like images and videos.⁷ For concrete defect detection, computer vision techniques can facilitate long-distance, non-contact, low-cost, and automated detection, and many significant researches have been successfully explored. A proposal was made for a deep learning model that integrates One-Dimensional Convolutional Neural Network (1D-CNN) and Long Short-Term Memory (LSTM) to enhance computational efficiency.⁸ The input images were transformed into the frequency domain, followed by calibration using the flattened frequency algorithm, and LSTM was employed to enhance the model’s performance on long sequence data. Complex backgrounds may weaken the robustness of convolutional neural networks, such as shallow cracks in range images. Heterogeneous image fusion can reduce false detections and achieve better classification performance.⁹ Moreover, a robust augmentation strategy was proposed in Palevičius et al.¹⁰ that includes ray-tracing of shadows, augmenting shadow datasets, and blending shadows, which significantly enhanced the detection robustness. Li et al.¹¹ proposed a crack detection model that employs enhanced preprocessing, coarse crack localization, and fine crack edge identification progressively. To increase the model’s sensitivity to noise in crack images, a combination of convolutional layers and a cycle generative adversarial network (GAN) was employed.¹² During the training process, multi-scale and multi-level features were aggregated, resulting in improved crack detection capabilities. To facilitate the detection of failures and the assessment of damages in reinforced concrete structures, YOLOv4 was introduced by Zou et al.¹³ This model can detect and localize damage in these structures, thereby aiding in their quick and efficient assessment. Santos et al.¹⁴ employed an unmanned aerial system to capture images of exposed steel bars on concrete structures. These images were then analyzed using a region-convolutional neural network (R-CNN), demonstrating the high adaptability of the deep learning-based method. Zhang et al.¹⁵ presented a context-aware crack segmentation network that utilized a context-aware fusion technique with local cross-state and cross-spatial restrictions to integrate image patch predictions. Han et al.¹⁶ used a convolutional neural network to find surface cracks in concrete structures and then modified a threshold method to extract clear crack contours for quantification. To further enhance model performance, researchers have turned their attention toward the use of attention mechanisms. These mechanisms are crucial in effectively utilizing crack features while suppressing background features, resulting in improved overall performance of the model. ATCrack¹⁷ achieved pixel-level crack detection by applying channel spatial attention, increasing model accuracy and robustness. STRNet,¹⁸ a network consisting of a squeeze and excitation attention-based encoder, a multi-head attention-based decoder, coarse up-sampling, and a focal-Tversky loss, showed rapid processing speed on concrete crack images. In addition, Cui et al.¹⁹ proposed an improved attention mechanism model termed Att-Unet, which focused on critical areas, reconstructed semantic features, and significantly improved the pixel-level crack segmentation performance. MaDnet²⁰ simultaneously identified material types and fine/coarse structural damage via semantic segmentation. The network leverages shared filters learned through multi-objective optimization, improving accuracy. A modified fusion network was utilized to handle multi-level and multi-scale features. The method addresses real-world crack identification within steel box girders of bridges using images containing complex disturbances.²¹ To identify and locate various seismic damages in images of damaged reinforced concrete columns, the modified Faster R-CNN was proposed, and the model’s effectiveness in recognizing specific damage types was analyzed.²²

The deep learning models presented in the aforementioned studies heavily depend on high-quality training labels. Semi-supervised learning leverages both labeled data and unlabeled data to extract supervised information. This technique has garnered considerable interest in recent years owing to the availability of vast amounts of unlabeled data in various real-world applications, as well as the desire to minimize the expenses and time involved in manual annotation. Shin et al.²³ combined multi-scale segmentation networks, discriminator networks, and adversarial learning to achieve crack detection in underground concrete structures. By generating a new precisely labeled image from unlabeled images, this approach reduces the burden of manual annotation while enhancing the segmentation performance. They subsequently combined a super-resolution network with a semi-supervised model, using a super-resolution network to improve the quality of the input image.²⁴ Another semi-supervised approach²⁵ was to generate supervision signals for unlabeled pavement crack images and use them for model training to compensate for insufficient image labels. Then, an adversarial strategy and a fully convolutional discriminator were employed to distinguish ground truth from predictions. The semi-supervised model also exhibited good robustness in the detection of steel surface defects.²⁶ An effective improvement for GAN-based semi-supervised models is to boost the performance of the generator. Gao et al.²⁷ proposed CU-Net and FU-Net based on the U-Net, which served as a generator to generate synthetic crack images, and the binary network was utilized as a discriminator. Wang and Su²⁸ proposed a semi-supervised model that utilizes consistency regularization for concrete crack segmentation. A few-shot meta-learning paradigm was introduced for structural damage identification.²⁹ The method involves an external few-shot meta-learning module and an embedded internal attribute-based transfer learning model. Compared to traditional supervised learning, it exhibits higher accuracy and robustness. Xu et al.³⁰ proposed a task-aware meta-learning approach, which includes an interpretable task generation strategy and a dual-stage optimization framework. The method is evaluated on a multi-type structural damage dataset, showcasing great performance even with limited training images.

While open-source datasets have contributed significantly to the field of concrete defect identification based on images, and high precision has been achieved in recognizing various defects, our main objective focuses on addressing two key challenges, making the transition to practical engineering applications smoother: (1) Reducing annotation effort: Despite the availability of open-source datasets, labeling data for specific engineering projects can be time-consuming and costly, especially when dealing with large-scale or specialized datasets. Our semi-supervised approach, Semi-SegDefect, offers a solution to minimize the need for extensive manual labeling, making it more practical for real-world scenarios. (2) Enhancing performance: Semi-SegDefect has demonstrated superior performance, especially when trained with limited labeled data. This is significant because in practical engineering scenarios, acquiring a vast amount of labeled data may not always be feasible or cost-effective. Our approach improves the accuracy of defect identification even when data availability is limited. Furthermore, the model proposed in this study exhibits excellent scalability. In real-world engineering scenarios, when encountering new types of defects that were not originally considered during the model’s development, our approach offers a practical advantage. With only a small amount of labeled data specific to the new defect type, our model can efficiently adapt and generalize its defect identification capabilities. This adaptability to novel defect categories is a valuable feature, as it allows for the continuous improvement and versatility of the model in addressing emerging challenges within the construction industry. In this study, we proposed a semi-supervised model for pixel-level detection of crack, spalling, and rebar exposure, which improves model performance by providing appropriate contrastive supervision information. Our contributions can be summed as follows:

A semi-supervised learning framework is proposed for multi-class defect segmentation of concrete structures.

Pixel-level contrastive learning is utilized to help models learn feature information from both local context and global semantic class relationships across the entire dataset.

Active sampling strategy optimizes only a sparse set of queries and keys, avoiding expensive computation costs.

Compared with fully supervised methods, our proposal achieves superior results with very few labels.

Methodology

General description

Based on strategies such as consistency regularization,³¹ entropy minimization,³² and pseudo-labeling,³³ many semi-supervised models have been proposed for semantic segmentation. The essence of these strategies is that the model output should always maintain a low-entropy prediction for data perturbations and geometric transformations. Inspired by these approaches, we have developed a semi-supervised segmentation model, Semi-SegDefect, based on pixel-level contrastive learning. Figure 1 depicts the overall structure of our approach, which comprises three components: supervised loss calculation, unsupervised loss calculation, and pixel-level contrastive loss calculation. The computation of supervised loss is implemented by feeding the ground-truth data into the student model. The unsupervised loss calculation adopts Mean Teacher architecture,³⁴ which includes two fundamental networks: the student model and the teacher model. The pixel-level contrastive loss calculation of concrete defects is performed by projecting the prediction results of labeled and unlabeled data into a high-dimensional feature space. For the improvement of semi-supervised segmentation models, diverse data perturbations are crucial. The strong augmentation process involves the sequential use of color dithering, Gaussian filtering, and CutMix methods. Weak augmentation only includes random flipping and random cropping. Details of each component are elaborated as follows.

Figure 1.

The overall structure of Semi-SegDefect.

Mean teacher framework

The structure of Semi-SegDefect is modified based on Mean Teacher.³⁴ Mean Teacher architecture adopts the consistency regularization strategy, where the model predictions should not change significantly if data perturbations are applied to unlabeled data samples. In the Mean Teacher architecture, the same network plays dual roles: teacher and student. As a teacher, the goal is to generate objects for consistency loss computation with unlabeled data as input. The teacher model and student model do not share weights. Instead, the teacher model employs the exponential moving average weights of the student model, as depicted in formula (1). This method enables the model to collect information after each training iteration, thereby enhancing the output of all layers for an improved intermediate representation. During the early training stages, the student model progresses rapidly, necessitating the teacher model to swiftly discard outdated and inaccurate student weights. As training progresses and the student model’s progress slows down, the teacher model can benefit from a longer memory by adjusting the smoothing coefficient hyperparameter.

\begin{matrix} {θ'}_{t} = α {θ'}_{t - 1} + (1 - α) θ_{t} \end{matrix}

(1)

where $θ'_{t}$ is the weights of the teacher model at time $t$ ; $θ_{t}$ is the weights of the student model at time t; and $α$ is the smoothing coefficient hyperparameter.

Student model and teacher model

Both the student and teacher models adhere to the traditional encoder–decoder structure for semantic segmentation, as shown in Figure 2, and they share the same architecture. The encoder extracts image features by increasing the number of channels while decreasing the resolution, resulting in a pyramid feature structure with four distinct scales. The decoder then projects the semantically rich features into pixel space to produce dense classification results. An input image undergoes a convolution operation with a 4 × 4 kernel and a stride of 4 for a quick reduction in resolution (4× down-sampling) and an increase in channel number from 3 to 192, without information loss. The down-sampled feature maps are then processed by the ConvNext block³⁵ for further feature extraction, producing the output of the first encoder stage. The following three stages use a convolution operation with a 2 × 2 kernel and a stride of 2 for 2× down-sampling prior to feature extraction. The final feature map resolutions correspond to 1/4, 1/8, 1/16, and 1/32 of the input image, respectively. The ConvNext block, depicted in Figure 3, is the key component of the encoder and extracts image features by stacking multiple convolutional layers.

Figure 2.

Student model and teacher model structure.

Figure 3.

ConvNext block.

The ConvNext block first uses a depth-wise separable convolution with a kernel size of 7 × 7, a stride of 1, and a zero padding of 3 for feature extraction. Compared with standard convolution, depth-wise separable convolution greatly reduces the number of parameters. Then two convolution kernels with a size of 1 × 1 and a stride of 1 are used to realize dimension increase and reduction of feature maps, respectively. The dimension increase operation multiplies the number of input channels by four, allowing for the extraction of more semantically rich information from the feature space.

The decoder employs an atrous spatial pyramid pooling (ASPP)³⁶ for feature extraction across different receptive fields. It has five parallel branches, consisting of a 1 × 1 convolutional layer, three 3 × 3 dilated convolutional layers, and a global average pooling layer. The ASPP effectively captures multi-scale information using parallel convolution layers with different atrous rates, making the model perform better on multi-scale objects. The outputs from the five branches are concatenated along the channel direction and further fused through a 1 × 1 convolutional layer that adjusts the channel number. To utilize position information in low-dimensional features, a 1 × 1 convolution compresses the low-level feature channels from Stage 1 to 48 channels, providing a bias toward higher-level semantic information. After 4× up-sampling, high-dimensional features are concatenated with low-dimensional features. The output is obtained via a 3 × 3 convolution and bilinear up-sampling.

Pixel-level contrastive learning

Contrastive learning has gained significant success in recent years. It leverages the entirety of information in the dataset to learn a highly representative image representation space. In semantic segmentation, the labels of each image are already known, and positive samples can be considered as pixels of the same category while negative samples can be considered as pixels of different categories, regardless of whether they come from the same image or not.^37,38 Contrastive learning can be utilized to enhance the traditional cross-entropy loss by discovering the global semantic relationship between pixels across all the training images, thereby producing a structured semantic feature relationship.

The cross-entropy loss at the pixel level is a standard loss function in semantic segmentation. This loss guides the network on whether its predictions for individual pixel classes are accurate. Despite its usefulness, the cross-entropy loss does not consider the local information of neighboring pixels or the relationships between errors in different classes. To address this, a more effective approach would be to design the loss function to leverage the relationships between categories. Rather than simply teaching the model to identify individual pixels as cracks or spalling, a better approach would be to provide finer supervision on easily confused pixel categories. The availability of finer supervision allows the model to learn more discriminative features in a semi-supervised setting, even with limited labeled data, resulting in improved overall segmentation performance.

To improve the feature representations learned by the model in semi-supervised segmentation, we have incorporated pixel-level contrastive learning that offers insight into the relative relationships between various semantic categories. Suppose $(X, Y)$ is a training set, which contains training images $x \in X$ and corresponding pixel-level class labels $y \in Y$ (for labeled data, y is provided by the original label; for unlabeled data, y is provided by the pseudo-labels generated by the teacher model). The segmentation network f with parameter $θ$ is then optimized by learning a mapping function $f_{θ} : X \to Y$ . The segmentation network f can be classified into two parts: an encoder network $φ : X \to Z$ and a decoder classification network $φ_{c} : Z \to Y$ . To achieve pixel-level contrastive learning, we introduced a decoder representation network $φ_{r}$ parallel to the decoder classification network. This network is responsible for projecting the feature information learned by the encoder network into dense representations in a higher dimension, which enables further abstraction and compression: $φ_{r} : Z \to R, R \in R^{m}$ . The decoder representation network is only used in the training phase to provide pixel-level contrastive loss for the guidance of model learning.

For an anchor pixel $r_{q}$ in the training set, the goal of pixel-level contrastive learning is to compare the anchor pixel with the pixels of other categories in high-level space projected by the decoder representation network, trying to bring it closer to pixels of the same class $r_{k}^{+}$ (positive samples) as far as possible and force anchor pixel away from pixels of other classes $r_{k}^{-}$ (negative samples). Therefore, the training paradigm of pixel-level contrastive learning can fully consider the semantic similarity between all pixel categories in the training set, and improve the representation learning ability of the model. Using pixel-level contrastive loss, the similarity between anchor pixels $r_{q}$ , positive pixels $r_{k}^{+}$ , and negative pixels $r_{k}^{-}$ can be quantified by a normalized dot product, the form of contrastive loss $L_{contr}$ can be formulated as follows:

\begin{matrix} L_{contr} = \sum_{c \in C} \sum_{r_{q} ~ R_{q}^{c}} - \log \frac{\exp (r_{q} \cdot r_{k}^{c, +} / τ)}{\exp (r_{q} \cdot r_{k}^{c, +} / τ) + \sum_{r_{k}^{-} ~ R_{k}^{c}} \exp (r_{q} \cdot r_{k}^{-} / τ)'} \end{matrix}

(2)

where C is the set of all categories contained in the current batch size; $r_{q}$ is the anchor pixel; $R_{q}^{c}$ is the set of anchor pixels; $r_{k}^{-}$ is the negative pixel; $R_{k}^{c}$ is the set of negative pixels; $r_{k}^{c, +}$ is the average representation of the category $c$ ; and $τ$ is the temperature coefficient hyperparameter.

Suppose P is a set that contains all pixel coordinates with the same resolution as R, these anchor pixels, negative pixels, and average representation used for the contrastive loss calculation can be formulated as follows:

\begin{matrix} R_{q}^{c} = ⋃_{[u, v] \in P} 1 (y_{[u, v]} = c) r_{[u, v]} \end{matrix}

(3)

\begin{matrix} R_{k}^{c} = ⋃_{[u, v] \in P} 1 (y_{[u, v]} \neq c) r_{[u, v]} \end{matrix}

(4)

\begin{matrix} r_{k}^{c, +} = \frac{1}{| R_{q}^{c} |} \sum_{r_{q} \in R_{q}^{c}} r_{q} \end{matrix}

(5)

Loss function

Two data sources are available for training semi-supervised semantic segmentation models: a limited number of labeled samples $X_{l} = {x_{l}, y_{l}}$ and a vast number of unlabeled samples $X_{u} = {x_{u}}$ . Typically, the loss function comprises two components: a supervised loss $L_{\sup}$ and an unsupervised loss $L_{unsup}$ . Supervised loss and unsupervised loss adopt cross-entropy loss function. The proposal introduces pixel-level contrastive loss $L_{contr}$ for the contrastive learning of anchor pixels. The overall loss function $L_{total}$ of the semi-supervised model is a linear combination of supervised loss $L_{\sup}$ , unsupervised loss $L_{unsup}$ , and contrastive loss $L_{contr}$ :

\begin{matrix} L_{total} = L_{\sup} + ω \cdot L_{unsup} + λ \cdot L_{contr} \end{matrix}

(6)

\begin{matrix} L_{\sup} = L_{unsup} = L_{crossentropy} = - \sum_{i = 0}^{C - 1} y_{i} \log (p_{i}) \end{matrix}

(7)

where $ω, λ$ are hyperparameters that control the balance between the different loss functions; C is the number of categories; $p_{i}$ is the probability that each sample belongs to a category $i$ ; $y_{i}$ is the one-hot representation of the sample label. If the sample belongs to a category $i$ , $y_{i} = 1$ ; otherwise, $y_{i} = 0$ .

Labeled data are used for the calculation of supervised loss during training, which compares the model’s predictions on labeled data to the ground truth labels and is crucial for training the model to make accurate predictions on these labeled examples. The unsupervised loss is calculated based on unlabeled data, with the purpose of encouraging the model to produce consistent predictions on different views or augmentations of the same unlabeled data point. This consistency regularization helps improve the model’s robustness and generalization. Finally, the pixel-level contrastive loss is calculated based on the dense representations predicted from labeled and unlabeled images. Contrastive loss is applicable to all pixels in labeled samples and valid pixels in unlabeled samples. Valid pixels in unlabeled samples are those located in the pseudo-labels with category confidence greater than a threshold $δ_{w} = 0.7$ since their pseudo-labels may be correct and useful in training the model. By contrast, pixels with category confidence below the threshold are likely incorrect pseudo-labels and should be ignored to prevent negative impacts on model training. This is especially important at the beginning of training when most pixels in predicted pseudo-labels are incorrect. By screening for effective pixels, incorrect pseudo-labels can be avoided, leading to improved model performance.

Data preparation and augmentation

The dataset is a crucial component in the development and training of segmentation models, as the model performance is closely tied to the quality and quantity of the data it is trained on. In recent years, many concrete defect datasets have been made public, which facilitates the conduct of experiments.^39–42 We collected relevant datasets, cropped the original image into sub-images with a resolution of 512 × 512 pixels, and only kept the images containing concrete defects. Cropping to a smaller size can ensure that the model only learns the most important information, reducing the risk of overfitting and improving its generalization capabilities. In addition, cropping to a smaller size also reduces the computational cost of training as it requires less memory and processing power. In this way, 1785 images were finally obtained. Unlike classification tasks that rely on extensive datasets like Imagenet or COCO, the focus of our research is defect segmentation, where the emphasis is on pixel-level precision rather than recognizing object classes in entire images. The dataset size depends not only on the number of images but also on the number of pixels within each image, which determines the quantity of training samples available during model training. In the image semantic segmentation task, each pixel can be considered an individual training sample because our goal is to assign the correct semantic label to each pixel. This means that even if the dataset contains limited images, the data volume is still ensured, as every image comprises hundreds of thousands of pixels, each of which provides valuable training information. Hence, even with a dataset containing limited images, considering each pixel as an individual training sample provides the model with a sufficient amount of information to learn various semantic features and boundaries accurately. This fine-grained level of annotation is essential for achieving precise image semantic segmentation, and as such, the data volume is indeed guaranteed. To evaluate the model performance, the dataset is randomly split into training, validation, and test sets in a ratio of 7:1:2. Table 1 summarizes the details of each type of defect in the dataset. In the dataset, crack defects tend to have fewer crack pixels per image, making them more challenging to detect, and therefore, fewer training samples are available compared to the other two defect types. By contrast, spalling and rebar exposure defects typically cover larger areas within the images, making them relatively simpler to detect. Therefore, there are more training samples for crack detection to facilitate comprehensive feature extraction. This distribution accounts for the varying complexities of defect types and aims to ensure effective model training for each category. Figure 4 shows several samples in the concrete defect dataset.

Table 1.

Detailed information after the partitioning of concrete defect dataset.

Defect types	Training set	Validation set	Test set	Total
Crack	603	86	172	861
Spalling	284	41	81	406
Rebar exposure	363	52	103	518
Total	1250	179	356	1785

Figure 4.

Samples in the concrete defect dataset.

Traditional data augmentation techniques, while valuable for enhancing model generalization, often fall short in the context of semantic segmentation. These methods primarily apply isolated transformations to individual images, which can disrupt the spatial relationships crucial for accurate segmentation. By contrast, CutMix⁴³ addresses these limitations by seamlessly blending segments of different images. This approach yields several notable advantages: First, CutMix introduces a contextual understanding that traditional data augmentation methods cannot match. By combining segments of multiple images, the model learns to recognize how different objects and regions within an image relate to one another. This heightened contextual awareness leads to improved semantic segmentation. Second, the regularization effect of CutMix is remarkable. The act of replacing segments from one image with those from another enforces consistency within the mixed regions. This regularization discourages overfitting and encourages the model to produce stable predictions. In a semi-supervised setting where labeled data are often scarce, this regularization proves invaluable for effectively leveraging the limited labeled examples. Moreover, by blending regions from distinct images, it guides the model toward learning more accurate and refined segmentation boundaries. CutMix mixes two images along with their corresponding labels to form a new, combined image and label, as shown in Figure 5. By training on a diverse set of images and labels, including those created through CutMix, the model can learn to handle ambiguous and challenging data more effectively. The combined image is a mix of two input images, and the combined label is a mix of their corresponding segmentation masks. The segmentation masks can be combined in the same way as the images, by randomly selecting two regions and merging them to form a single mask. Assuming that $x_{a}$ and $x_{b}$ represent two unlabeled images, the goal of the CutMix strategy is to use the mask M to mix the two images and generate a new training sample. Then, the prediction results $g_{\emptyset} (x_{a})$ and $g_{\emptyset} (x_{b})$ of teacher model on the input data $x_{a}$ and $x_{b}$ are mixed to generate pseudo-labels. Finally, the model is optimized by the unsupervised loss (consistency loss). This process can be simplified as:

\begin{matrix} mix (a, b, M) = (1 - M) ⊙ a + M ⊙ b \end{matrix}

(8)

L_{unsup} = {‖ mix (g_{\emptyset} (x_{a}), g_{\emptyset} (x_{b}), M) - f_{θ} (mix (x_{a}, x_{b}, M)) ‖}^{2}

(9)

where $a, b$ are the input unlabeled data; $M \in {0, 1}^{W \times H}$ is the binary mask, indicating the area to be discarded and selected from the two images; 1 is the mask filled with 1; $g_{\emptyset}$ is the teacher model; $f_{θ}$ is the student model.

Figure 5.

Schematic diagram of CutMix.

In the CutMix algorithm, $α$ is a hyperparameter that controls the strength of the CutMix operation. Specifically, it is used to generate the $β$ distribution that determines the size of the random bounding box used to crop out a portion of one image and replace it with a portion of another image. In this study, $α$ is set to 0.5, meaning that the $β$ distribution used to generate the random bounding box will have a shape that is relatively balanced between the two images, combining both images in a more balanced way.

Implementation details

To assess the efficacy of semi-supervised learning, we trained and tested the Semi-SegDefect model on the dataset established in the section “Data preparation and augmentation.” The model was trained and tested on a PC running the Win10 operating system with an Intel® Core™ i7-8700 CPU and a GeForce GTX 1080 Ti 11GB GPU. To ensure fair comparisons, the validation and test sets were kept consistent throughout the experiments. During training, the AdamW⁴⁴ optimizer was utilized with an initial learning rate of 1e-3, and a batch size of 2. During the early stage of training, a higher learning rate is recommended to allow the network to converge quickly. Conversely, a lower learning rate is recommended during the later stage of training to enable the network to converge toward an optimal solution more accurately. To achieve this, the Poly strategy was employed to dynamically adjust the learning rate using a power of 0.95.

The quantitative evaluation of the segmentation model adopts two popular metrics: Intersection over Union (IoU) and Mean Intersection over Union (MIoU). IoU is the area of overlap between the predicted segmentation and the ground truth divided by the area of union between the predicted segmentation and the ground truth. MIoU is the average IoU across all classes in a dataset. This metric provides a more comprehensive view of model performance, as it takes into account the performance of individual classes. Figures 6 and 7 show the learning rate and MIoU curves during training. MIoU first increases over time. At some point, the model reaches a saturation point, where the MIoU does not improve significantly despite further training. This indicates that the model has reached its maximum performance capacity, and further training may not result in significant improvements, which will be used for further estimation of model performance:

\begin{matrix} Lr = ini t_{lr} \times {(1 - \frac{epoch}{nu m_{epoch}})}^{power} \end{matrix}

(10)

\begin{matrix} IoU = \frac{TP}{TP + FP + FN} \end{matrix}

(11)

\begin{matrix} MIoU = mean (IoU) \end{matrix}

(12)

where TP refers to the number of pixels that are correctly classified by the model as belonging to a certain defect class; FP refers to the number of pixels that are wrongly classified by the model as belonging to a certain defect class when they actually do not; FN refers to the number of pixels that belong to a certain defect class but are misclassified as another class or as background by the model.

Figure 6.

The curve of learning rate during training.

Figure 7.

Curves of MIoU during training.

Test results and discussion

Selection of initial learning rate

Learning rate is a hyperparameter that determines the step size at which the model updates its parameters during the training process. It is an essential factor in determining the model’s convergence and performance. The ideal learning rate value depends on the problem and the dataset at hand. A learning rate that is too small can result in slow convergence of the model, while a learning rate that is too large can cause the model to overshoot the optimal solution and lead to divergence. This section tests the performance of the model by setting different initial learning rates. Table 2 presents the segmentation results in terms of IoU scores for each category at different learning rates. The MIoU is also calculated to provide an overall performance measure. According to the values in the table, it can be found that when the learning rate is set to 0.1, the performance of the model is the worst. This is mainly because a large learning rate makes the training process of the model divergent and cannot achieve the global optimum. As the learning rate decreases, the optimization process converges more slowly but is more likely to find the global optimum, achieving the best performance with an MIoU of 69.63% when the learning rate is set to 0.001. However, if the learning rate is too small, the optimizer may take tiny steps that cause the model to converge too slowly or get stuck in a local minimum, leading to underfitting. This can result in poor segmentation performance, where the model fails to capture the necessary details in the input data for accurate segmentation. Therefore, the initial learning rate is uniformly set to 0.001 in the following experiments.

Table 2.

Test results with different learning rates.

Learning rate	IoU for crack (%)	IoU for spalling (%)	IoU for rebar exposure (%)	MIoU (%)
0.1	63.12	67.54	67.88	66.18
0.01	67.94	69.08	69.12	68.71
0.001	68.48	70.85	69.56	69.63
0.0001	69.02	68.74	68.49	68.75

IoU: Intersection over Union; MIoU: Mean Intersection over Union.

Impact of contrastive loss weight on model performance

The weight assigned to the contrastive loss function in semi-supervised learning is crucial in determining the balance between the supervised and unsupervised components. The objective of contrastive loss is to enhance the similarity between positive pairs of instances while minimizing the similarity between negative pairs of instances within the feature space. It is a measure of similarity between pixels of different defect categories. In practice, the weight of the contrastive loss function is typically set through experimentation and tuning. To balance the contribution of contrastive loss and the other two losses to the total loss, a hyperparameter study is carried out by setting different loss weights, and the results are summarized in Table 3. It indicates that as the weight of the contrastive loss increases, the total loss gives more weight to the differences between pixel categories. However, if the weight of the contrastive loss becomes too large, it may dominate the total loss and cause a decrease in the overall segmentation performance of the model. Therefore, the contrastive loss weight is set to 0.20 in the experiment.

Table 3.

Test results with different contrastive loss weight.

Contrastive loss weight	IoU for crack (%)	IoU for spalling (%)	IoU for rebar exposure (%)	MIoU (%)
0.02	64.33	68.83	66.08	66.41
0.20	68.48	70.85	69.56	69.63
0.40	68.12	68.54	68.70	68.45
0.60	66.70	68.01	66.77	67.16

IoU: Intersection over Union; MIoU: Mean Intersection over Union.

Impact of unsupervised loss weight on model performance

It is beneficial for training that the unsupervised loss weight starts close to zero because the initial model prediction accuracy is low, and the pseudo-labels at this time will not be a reasonable object to provide additional supervision information. Therefore, the contribution of unsupervised loss to the total loss should not be too big. As the training progresses, the model performance improves, and the weight can be increased at this time. In the experiments, the unsupervised loss weight is set as the percentage of the number of pixels with a confidence greater than 0.97 in the predicted pseudo-labels, as a sufficient representative of the prediction quality. The effect of unsupervised loss weights on model performance is shown in Table 4. The results indicate that when the unsupervised loss weight is not set, the segmentation performance of the model for crack, spalling, and rebar exposure drops by 3.08%, 2.16%, and 1.41%, respectively. Unsupervised loss contains much uncertain information, when the unsupervised loss weight is not set, all predicted pixel information will be used for model training, resulting in a great interference to the normal model training.

Table 4.

Test results with and without unsupervised loss weight.

Unsupervised loss	IoU for crack (%)	IoU for spalling (%)	IoU for rebar exposure (%)	MIoU (%)
With weight	68.48	70.85	69.56	69.63
Without weight	65.40	68.69	68.15	67.41

IoU: Intersection over Union; MIoU: Mean Intersection over Union.

Impact of sampling strategy on model performance

An image contains thousands of pixels, and sampling all the pixels when calculating pixel-level contrastive loss will bring huge computational costs and is unnecessary. The segmentation network is usually only indistinguishable for the pixels at the junction of different categories. To solve the above issue, an active sampling strategy is adopted. The active sampling strategy requires that the selection of anchor pixels must meet the following requirements: (a) under weak data augmentation, the model’s prediction confidence is greater than 0.7 and (b) under strong data augmentation, the model’s prediction confidence is less than 0.97. The anchor pixel first needs to be guaranteed to have the correct label. For the pixels from the labeled dataset, the correctness of the label can be guaranteed; for the pixels from the unlabeled dataset, the requirement (a) can ensure that those pixels with reliable prediction are selected. Requirement (b) can ensure that the model can sample indistinguishable pixels, and only the gradient update of these pixels with insufficient confidence can really help improve performance. If we sample anchor pixels with confidence greater than 0.97 for comparative learning under strong data augmentation operations, the model will not learn any useful information because models have learned well at the beginning for simple samples.

When sampling negative samples, it is better to select those pixels that are easy to confuse, and the definition of these confusion categories obviously cannot be artificial because the confusion categories may be different in different training stages. In the experiment, non-uniform sampling is performed by calculating the relative distance between defect pixels of different categories. If the distance is close, it is considered to be a confusion category. The calculated distances can form a probability distribution where negative samples are sampled. This process assigns more negative samples to categories that are easy to confuse, helping the model learn more accurate decision boundaries.

Table 5 summarizes the model performance when trained with different sampling strategies. Compared with the active sampling strategy, when the random sampling strategy is used, the IoU of crack, spalling, and rebar exposure drops by 9.74%, 6.97%, and 4.91%, respectively, mainly due to that the randomly sampled pixels contain a large number of simple samples. The model fails to learn useful knowledge from these simple samples through contrastive learning, and the effect of contrastive loss cannot be reflected. If the random key is replaced by the activate key, the performance improvement of the model is still small, and the MIoU only improves by about 1.17%. The activate query strategy has a major impact on the model performance, which further confirms that most queries are redundant.

Table 5.

Test results with different sampling strategies.

Sample strategy	IoU for crack (%)	IoU for spalling (%)	IoU for rebar exposure (%)	MIoU (%)
Random query/random key	58.74	63.88	64.65	62.42
Activate query/random key	62.42	67.51	66.29	65.41
Random query/activate key	61.36	64.15	65.26	63.59
Activate query/activate key	68.48	70.85	69.56	69.63

IoU: Intersection over Union; MIoU: Mean Intersection over Union.

Influence of sampling quantity on model performance

The pixels used for contrastive loss calculation are sampled from the decoder representation network, including three categories: anchor pixels, positive pixels, and negative pixels. Positive pixels are represented by the average representation of the class, calculated by Equation (5). Anchor pixels are selected for comparison with other pixels in an image to learn representations that keep the anchor pixels close to other pixels in the same class and far from pixels in different classes. The number of anchor pixels and negative samples chosen can impact the performance of contrastive loss functions in semi-supervised learning. Typically, the number of anchor pixels is smaller than the number of negative samples in practical applications since negative samples often contain difficult-to-classify pixels that can improve model performance. This ensures that the contrastive loss function has enough negative samples to learn from. As the number of pixel samples increases, the computational cost also increases. The number of pixel samples required depends on the dataset’s complexity and the neural network architecture used for training. This part studies the change in model performance by setting different numbers of anchor pixels and negative pixels. The number of negative samples is set to be twice of anchor pixels. The results are shown in Table 6. It can be observed from Table 6 that when the number of anchor pixels is too small, the model performance is not ideal due to the difficulty of contrastive loss to fully evaluate the difference between pixels of different defect categories. When sampling more pixels, the model performance will be better, but after a certain point increasing the number of sample pixels will have a small improvement in the segmentation performance. Considering the performance improvement and computer hardware requirements, the number of anchor pixels and negative samples is selected as 256/512. It is worth noting that in the four sampling settings, even with the smallest number of pixels (32/64), the segmentation result (MIoU 66.85%) is still better than the DeepLabv3+ shown in Table 9 (MIoU 60.47%), which further verifies the effectiveness of pixel-level contrastive learning.

Table 6.

Test results with different numbers of queries and keys.

Number of queries/number of keys	IoU for crack (%)	IoU for spalling (%)	IoU for rebar exposure (%)	MIoU (%)
32/64	60.93	70.11	69.50	66.85
64/128	67.51	69.84	67.70	68.35
128/256	66.95	71.03	69.55	69.18
256/512	68.48	70.85	69.56	69.63

IoU: Intersection over Union; MIoU: Mean Intersection over Union.

Comparison results across different dimensions of contrastive loss calculation

The encoding features extracted from the encoder network often reside in a lower-dimensional space, which might not capture the subtle nuances and semantic information crucial for tasks like similarity learning or contrastive loss effectively. The decoder representation network begins with a 2D convolutional layer using a kernel size of 3 × 3 to extract features. Subsequently, batch normalization is applied to stabilize training. An activation function, ReLU (Rectified Linear Unit), introduces non-linearity into the network. Lastly, another 2D convolutional layer with a 1 × 1 kernel further refines the features, transforming each pixel’s feature vector independently, resulting in the extraction of higher-level feature representations. Table 7 shows the segmentation performance when using high-dimensional features and encoder features for contrastive loss calculation. It can be seen that when high-dimensional features are used to calculate the contrastive loss, the IoU for crack, spalling, and rebar exposure are improved by 4.35%, 3.05, and 3.13%, respectively. By employing a separate decoder representation network that projects these features into a higher-dimensional space, we enrich the feature representations with more expressive power. This higher-dimensional space allows the model to distinguish subtle differences and similarities more effectively, leading to enhanced discriminative capabilities. Furthermore, using high-dimensional features before applying contrastive loss fosters better separability between positive and negative pairs. It provides the model with a broader feature space to work with, reducing the risk of feature collapse or over-regularization that can occur when using lower-dimensional representations directly. The increased dimensionality amplifies the model’s ability to capture fine-grained patterns. Therefore, employing a decoder representation network to project features into a higher-dimensional space before applying contrastive loss is both effective and necessary. It empowers the model with richer and more expressive feature representations and enhances its discriminative capabilities.

Table 7.

Test results of calculating contrastive loss using features of different dimensions.

Contrastive loss calculation	IoU for crack (%)	IoU for spalling (%)	IoU for rebar exposure (%)	MIoU (%)
With low-dimension	64.13	67.80	66.43	66.12
With high-dimension	68.48	70.85	69.56	69.63

IoU: Intersection over Union; MIoU: Mean Intersection over Union.

Effect verification of different loss branches

In this study, the loss function comprises three components: supervised loss, unsupervised loss, and contrastive loss. To assess the influence of each loss on model performance, we conducted ablation experiments, and the test results are summarized in Table 8. It can be seen that removing the supervision loss from the total loss will lead to a significant decrease in model performance, with the IoU of crack, spalling, and rebar exposure being reduced by 5.84%, 2.43%, and 3.43%, respectively. The supervised loss serves as a crucial source of task-specific guidance, ensuring that the model accurately predicts labeled data. In addition, the supervised loss aids in generalization, helping the model extend its learning from labeled examples to unseen data. Without it, the model will struggle to adapt to new, unobserved instances, reducing its overall effectiveness. The unsupervised loss leverages the information from unlabeled data, aiding in feature learning, and improving the model’s generalization capabilities. Its absence can lead to a reduction in the model’s ability to utilize unlabeled data effectively, resulting in overfitting to the labeled data. The contrastive loss encourages the model to bring similar samples closer together in feature space while pushing dissimilar samples apart. This mechanism helps the model better discriminate between different classes or objects and enhances its ability to capture subtle differences and similarities in the data. Without the contrastive loss, the model will struggle to learn discriminative features, which can lead to a reduction in its overall performance. When unsupervised loss and contrastive loss are not considered, the overall performance of the model decreases by 2.68% and 3.59%, respectively, in terms of IoU. A balanced combination of all three losses is critical for optimizing the performance of the model in concrete defect segmentation.

Table 8.

Test results with different loss branches.

Supervised branch	Unsupervised branch	Contrastive branch	IoU for crack (%)	IoU for spalling (%)	IoU for rebar exposure (%)	MIoU (%)
X	√	√	62.64	68.42	66.13	65.73
√	X	√	65.13	68.71	67.01	66.95
√	√	X	63.19	68.02	66.91	66.04
√	√	√	68.48	70.85	69.56	69.63

IoU: Intersection over Union; MIoU: Mean Intersection over Union.

Comparative studies

To ensure the fairness of the comparative studies, we took several measures to mitigate the risk of models getting stuck in the local optimum. First, we employed a consistent experimental setup for both the semi-supervised model (Semi-SegDefect, UniMatch,⁴⁵ and Duo-SegNet⁴⁶) and the fully supervised models (FPN,⁴⁷ DeepLabv3+⁴⁸) by training them on the same subsets of the dataset—specifically, 2%, 5%, 10%, and 20%. This ensured that all models were exposed to equivalent data proportions, minimizing any potential bias stemming from varying dataset sizes. In addition, we implemented the same training strategies designed to help models escape local optimum, including using appropriate learning rate schedules and data augmentation techniques. These strategies aimed to encourage model convergence to more favorable solutions and avoid overfitting.

As shown in Table 9, Semi-SegDefect greatly outperforms DeepLabv3+ when training with 2%, 5%, 10%, and 20% of labeled data. The MIoU are increased by 9.82%, 4.00%, 4.01%, and 3.69%, respectively, which fully demonstrates that the proposed semi-supervised model can mine more effective semantic information from a large amount of unlabeled data to provide additional supervision information for the classification of defect pixels. When the model is trained with very few labeled data (2%, 25 labeled images), the performance gap between the fully supervised model and the semi-supervised model, Semi-SegDefect, is the largest, 18.33% for crack, 3.98% for spalling, and 7.15% for rebar exposure, respectively. The deep learning model has a large number of model parameters, and the model is prone to overfitting when only a few data are used for training, which affects the model’s robustness and generalization performance. Semi-SegDefect makes predictions on unlabeled data after training with a small amount of data and provides supervision information for the successive model training by selecting pixels with reliable predictions in pseudo-labels. When trained with a limited amount of labeled data, semi-supervised models consistently demonstrate superior performance compared to fully supervised models. While UniMatch and Duo-SegNet exhibit competitiveness, they generally exhibit lower IoU scores and MIoU compared to Semi-SegDefect, especially in scenarios with few labeled data. Under the conditions of 2%, 5%, 10%, and 20% labeled data, UniMatch exhibits MIoU scores that are lower than Semi-SegDefect by 1.87%, 1.55%, 1.24%, and 0.44%, respectively. Similarly, Duo-SegNet demonstrates MIoU scores that are lower than Semi-SegDefect by 3.81%, 2.03%, 1.83%, and 0.71%, respectively. The performance gap between the semi-supervised models narrows as the training set size increases, which can be attributed to the distinct strategies these models employ in leveraging unlabeled data. Each semi-supervised model relies on different mechanisms and assumptions about the distribution of unlabeled data, and these discrepancies become more evident when labeled data is scarce. With a larger portion of labeled images participating in the training, the models benefit more from the direct supervision provided by these labeled samples. In the case of spalling and rebar exposure, UniMatch may achieve higher IoU scores than Semi-SegDefect at times. However, its MIoU for all three defects is lower.

Table 9.

Comparison of different segmentation models on the test set.

Training set	Model	IoU for crack (%)	IoU for spalling (%)	IoU for rebar exposure (%)	MIoU (%)
2%	FPN	43.35	65.64	54.30	54.43
	DeepLabv3+	50.15	66.87	62.41	59.81
	UniMatch	67.65	68.17	67.46	67.76
	Duo-SegNet	63.38	66.85	67.23	65.82
	Semi-SegDefect	68.48	70.85	69.56	69.63
5%	FPN	72.30	75.47	65.64	71.14
	DeepLabv3+	73.81	75.87	71.02	73.57
	UniMatch	69.04	80.13	78.89	76.02
	Duo-SegNet	68.77	79.03	78.84	75.54
	Semi-SegDefect	72.13	82.25	78.33	77.57
10%	FPN	72.66	77.45	74.30	74.80
	DeepLabv3+	75.61	77.48	76.21	76.43
	UniMatch	72.32	84.23	81.04	79.20
	Duo-SegNet	71.42	83.44	80.96	78.61
	Semi-SegDefect	74.45	84.11	82.76	80.44
20%	FPN	74.50	84.96	77.00	78.82
	DeepLabv3+	75.62	84.88	78.77	79.76
	UniMatch	74.64	88.97	85.42	83.01
	Duo-SegNet	74.27	89.02	84.93	82.74
	Semi-SegDefect	75.31	89.65	85.38	83.45
100%	FPN	74.85	90.88	87.46	84.40
	DeepLabv3+	82.10	92.29	89.40	87.93

IoU: Intersection over Union; MIoU: Mean Intersection over Union.

Figure 8 shows the segmentation results of the Semi-SegDefect model on the test set when using 20% labeled data for training. For visual comparison, the segmentation results of DeepLabv3+ trained with all labeled data are also given in Figure 8. Regarding the mis-segmented areas, both models tended to have difficulties in accurately detecting small or faint defects, resulting in under-segmentation. The semi-supervised model also had some mis-segmentation in areas with complex textures or ambiguous boundaries, possibly due to the limited amount of labeled data in the training set. By contrast, the fully supervised model was able to capture more details of the defects, but it still suffered from mis-segmentation in areas with similar edge texture or color to the background. Overall, it can be concluded that Semi-SegDefect can achieve satisfactory segmentation of defects with limited labeled data, greatly reducing the dependence on labeled data, and has a good application prospect in practical engineering. The training process was based on cropped sub-images. Once the model has been trained, it can be used to automate the detection of surface defects in concrete structures.

Figure 8.

Samples of segmentation results.

Robustness test

Comprehensive robustness testing is paramount to assess the model’s capacity to perform effectively under various challenging conditions. These challenging images were instrumental in thoroughly evaluating the model’s performance and its ability to handle diverse and complex scenarios commonly encountered in concrete defect detection applications. We selected a set of images for model testing, some of which were captured in complex scenes, some exhibited coupled damages, and others contained interference factors such as damage-like edges/regions, variations in illumination, shadows, and motion blurs. Figures 9 and 10 depict the segmentation results of single and coupled defects by the model in complex scenarios, respectively. It can be seen that the proposed model has good adaptability to diverse and complex scenarios frequently encountered in concrete defect detection applications. Notably, the model demonstrates stronger generalization capabilities when it comes to identifying spalling and rebar exposure. This is primarily attributed to the pronounced characteristics of these two defect types, where the model adeptly captures the relevant distinctive features and patterns. In comparison, when it comes to detecting cracks, the model is prone to false negatives or false positives, particularly in scenarios with pronounced interference factors. These findings will motivate ongoing research and improvements to enhance the model’s capabilities for real-world applications.

Figure 9.

Samples of single defect segmentation results.

Figure 10.

Samples of coupled defect segmentation results.

Visualization of the model decision process

In semi-supervised segmentation, the encoder is a critical component for extracting significant features from both labeled and unlabeled data. Leveraging the knowledge obtained from a vast amount of unlabeled data, the encoder can extract more robust features that are generalizable to new, unseen data. To gain insight into which features Semi-SegDefect uses to make predictions during the inference phase, and to determine whether the model focuses on the correct regions of the image, we utilized gradient-weighted class activation mapping (Grad-CAM) to visualize the encoder. The Grad-CAM algorithm operates by calculating the gradients of the predicted class score relative to the feature maps of the last convolutional layer in the encoder. These gradients are utilized to assign weights to the feature maps, producing a coarse localization map that indicates the regions of the image that contributed the most to the decision. The visualization results of Grad-CAM are shown in Figure 11, demonstrating that the encoder can activate the complete object of interest region while suppressing the irrelevant background region for concrete defect feature extraction, resulting in reliable predictions. The response areas in various scenarios are relatively concentrated, accurately capturing semantically related areas and focusing on important targets. This is crucial in concrete defect detection, as precise and dependable predictions are essential for ensuring the safety and structural integrity of concrete structures.

Figure 11.

Visualization results of the encoder with Grad-CAM.

Discussion of semi-supervised and fully supervised learning

Semi-supervised learning and fully supervised learning are two approaches used to train models for image segmentation tasks. While both methods aim to improve the model’s accuracy in identifying objects within an image, they differ in the amount of labeled data required. Fully supervised learning requires a large amount of labeled data, which includes both the image and its corresponding ground truth segmentation masks, to train the model effectively. On the other hand, semi-supervised learning can achieve high performance even with a small amount of labeled data because it leverages a large amount of unlabeled data to improve performance. The model is trained using both the labeled data and the unlabeled data, where the unlabeled data are used to learn the underlying patterns and relationships in the data. By learning from both labeled and unlabeled data, the model can better understand the complex relationships within the data, resulting in better performance. The choice between semi-supervised and fully supervised learning depends on the availability of labeled data. If sufficient labeled data are available, fully supervised learning may be the better choice, as it has the potential to achieve higher performance. However, if labeled data are scarce, semi-supervised learning may be a more practical approach, as it can still achieve good performance with limited labeled data.

Conclusion

In this study, a semi-supervised segmentation model termed Semi-SegDefect was proposed to automatically perform pixel-level detection for concrete defects. Semi-SegDefect consists of a student model and a teacher model with the same structure. To achieve pixel-level contrastive learning, a decoder representation network is added in parallel with the decoder classification network, and the encoder output features are projected into a high-dimensional space to select anchor pixels and negative pixels. The hyperparameter selection and applicability study of the proposed Semi-SegDefect model was carried out on a public dataset of concrete defects. In view of the experimental study, several conclusions can be drawn:

The settings of learning rate, contrastive loss weight, and unsupervised loss weight play an important role in achieving better performance.

Most queries are redundant in pixel sampling space, and the activate sampling strategy is quite essential.

Sampling more anchor pixels and negative pixels means better model performance. After a certain point, the performance improvement becomes marginal, along with higher computational cost.

Semi-SegDefect can reach a higher MioU than a fully supervised segmentation model with a small number of labeled data. When using the 2%, 5%, 10%, and 20% training dataset, the MIoU reaches an improvement of 9.82%, 4.00%, 4.01%, and 3.69%, respectively.

Our proposal overcomes the dependence on large-scale labeled data, makes full use of unlabeled data to provide reliable supervision information for model training, and achieves the goal of improving model performance at a low cost. However, we chose to investigate the effects of each hyperparameter independently for efficiency and simplicity in this study. In future work, we will delve deeper into potential coupling influences and fine-tune hyperparameters for specific use cases. Furthermore, we plan to expand our dataset to include a wider range of defect types. This expansion will not only enrich the dataset but also contribute to a more comprehensive evaluation of our model’s performance across various real-world scenarios and structural conditions. In addition, the development of a structural health management platform will also be considered, benefiting from the advantages of deep learning and unmanned aerial vehicles to improve the efficiency of inspectors.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The authors gratefully acknowledge the financial support of this research comes from the National Natural Science Foundation of China (grant no. 51579089).

Data availability

The code can be downloaded through the following link:

ORCID iD

Wenjun Wang

References

Flah

Nunez

Ben Chaabene

, et al. Machine learning algorithms in civil structural health monitoring: a systematic review. Arch Comput Methods Eng 2021; 28: 2621–2643.

Reagan

Sabato

Niezrecki

Feasibility of using digital image correlation for unmanned aerial vehicle structural health monitoring of bridges. Struct Health Monit 2018; 17: 1056–1072.

Taheri

A review on five key sensors for monitoring of concrete structures. Constr Build Mater 2019; 204: 492–509.

Cha

Choi

Büyüköztürk

Deep learning-based crack damage detection using convolutional neural networks. Comput-Aided Civ Infrastruct Eng 2017; 32: 361–378.

Deng

Lee

VCS

. Concrete crack detection with handwriting script interferences using faster region-based convolutional neural network. Comput Aided Civ Infrastruct Eng 2020; 35: 373–388.

Spencer

Jr Hoskere

Narazaki

Advances in computer vision-based civil infrastructure inspection and monitoring. Engineering 2019; 5: 199–222.

Bao

Machine learning paradigm for structural health monitoring. Struct Health Monit 2021; 20: 1353–1372.

Zhang

Barri

Babanajad

, et al. Real-time detection of cracks on concrete bridge decks using deep learning in the frequency domain. Engineering 2020; 7: 1786–1796.

Zhou

Song

Deep learning–based roadway crack classification with heterogeneous image data fusion. Struct Health Monit 2021; 20: 1274–1293.

10.

Palevičius

Pal

Landauskas

, et al. Automatic detection of cracks on concrete surfaces in the presence of shadows. Sensors-Basel 2022; 22: 3662.

11.

Niu

, et al. Tunnel crack detection using coarse-to-fine region localization and edge detection. Wiley Interdiscip Rev Data Min Knowl Discovery 2019; 9: e1308.

12.

Munawar

Ullah

Heravi

, et al. Inspecting buildings using drones and computer vision: a machine learning approach to detect cracks and damages. Drones 2021; 6: 5.

13.

Zou

Zhang

Bai

, et al. Multicategory damage detection and safety assessment of post-earthquake reinforced concrete structures using deep learning. Comput-Aided Civ Infrastruct Eng 2022; 37: 1188–1204.

14.

Santos

Ribeiro

Lopes

, et al. Detection of exposed steel rebars based on deep-learning techniques and unmanned aerial vehicles. Automat Constr 2022; 139: 104324.

15.

Zhang

Rajan

Story

Concrete crack detection using context-aware deep semantic segmentation network. Comput-Aided Civ Infrastruct Eng 2019; 34: 951–971.

16.

Han

Zhao

Chen

, et al. Structural damage-causing concrete cracking detection based on a deep-learning method. Constr Build Mater 2022; 337: 127562.

17.

, et al. Automatic pixel-level detection method for concrete crack with channel-spatial attention convolution neural network. Struct Health Monit. Epub ahead of print 5 July 2022. DOI: 10.1177/14759217221109496.

18.

Kang

Cha

Y-J.

Efficient attention-based deep encoder and decoder for automatic crack segmentation. Struct Health Monit 2022; 21: 2190–2205.

19.

Cui

Wang

Dai

, et al. Intelligent crack detection based on attention mechanism in convolution neural network. Adv Struct Eng 2021; 24: 1859–1868.

20.

Hoskere

Narazaki

Hoang

, et al. MaDnet: multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure. J Civ Struct Health Monit 2020; 10: 757–773.

21.

Bao

Chen

, et al. Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumer-grade camera images. Struct Health Monit 2019; 18: 653–674.

22.

Wei

Bao

, et al. Automatic seismic damage identification of reinforced concrete columns from images by a region-based deep convolutional neural network. Struct Control Health Monit 2019; 26: e2313.

23.

Shim

Kim

Cho

G-C

, et al. Multiscale and adversarial learning-based semi-supervised semantic segmentation approach for crack detection in concrete structures. IEEE Access 2020; 8: 170939–170950.

24.

Shim

Kim

Lee

S-W

, et al. Road damage detection using super-resolution and semi-supervised learning with generative adversarial network. Automat Constr 2022; 135: 104139.

25.

Wan

, et al. Semi-supervised semantic segmentation using adversarial learning for pavement crack detection. IEEE Access 2020; 8: 51446–51459.

26.

Zhang

Pan

Zhang

Semi-supervised learning with GAN for automatic defect detection from images. Automat Constr 2021; 128: 103764.

27.

Gao

Peng

, et al. Generative adversarial networks for road crack image segmentation. In: 2019 International joint conference on neural networks (IJCNN), Budapest, Hungary, 2019, pp. 1–8. IEEE.

28.

Wang

Semi-supervised semantic segmentation network for surface crack detection. Automat Constr 2021; 128: 103786.

29.

Bao

Zhang

, et al. Attribute-based structural damage identification by few-shot meta learning with inter-class knowledge transfer. Struct Health Monit 2021; 20: 1494–1517.

30.

Fan

Bao

, et al. Task-aware meta-learning paradigm for universal structural damage segmentation using limited images. Eng Struct 2023; 284: 115917.

31.

French

Laine

Aila

, et al. Semi-supervised semantic segmentation needs strong, varied perturbations. arXiv preprint arXiv:190601916, 2019.

32.

Chen

Jin

, et al. Semisupervised semantic segmentation by improving prediction confidence. IEEE Trans Neural Networks Learn Syst 2021; 33: 4991–5003.

33.

Chen

Yuan

Zeng

, et al. Semi-supervised semantic segmentation with cross pseudo supervision. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Nashville, TN, USA, 2021, pp. 2613–2622.

34.

Tarvainen

Valpola

Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. Adv Neural Inf Process Syst 2017; 30: 1195–1204.

35.

Liu

Mao

C-Y

, et al. A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, New Orleans, LA, USA, 2022, pp. 11976–11986.

36.

Chen

L-C

Papandreou

Schroff

, et al. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:170605587, 2017.

37.

Le-Khac

Healy

Smeaton

AF.

Contrastive representation learning: a framework and review. IEEE Access 2020; 8: 193907–193934.

38.

Jaiswal

Babu

Zadeh

, et al. A survey on contrastive self-supervised learning. Technologies 2020; 9: 2.

39.

Zhao

Zhou

Automatic pixel-level multiple damage detection of concrete structure using fully convolutional network. Comput Aided Civ Infrastruct Eng 2019; 34: 616–634.

40.

Yang

, et al. Semantic metric 3d reconstruction for concrete inspection. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, Salt Lake City, UT, USA, 2018, pp. 1543–1551.

41.

Yang

, et al. Automatic pixel-level crack detection and measurement using fully convolutional network. Comput Aided Civ Infrastruct Eng 2018; 33: 1090–1109.

42.

Zhang

Chang

C-C

Jamshidi

Simultaneous pixel-level concrete defect detection and grouping using a fully convolutional model. Struct Health Monit 2021; 20: 2199–2215.

43.

Yun

Han

, et al. Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF international conference on computer vision, Seoul, Korea (South), 2019, pp. 6023–6032.

44.

Loshchilov

Hutter

Decoupled weight decay regularization. arXiv preprint arXiv:171105101, 2017.

45.

Yang

Feng

, et al. Revisiting weak-to-strong consistency in semi-supervised semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Vancouver, BC, Canada, 2023, pp. 7236–7246.

46.

Peiris

Chen

Egan

, et al. Duo-SegNet: adversarial dual-views for semi-supervised medical image segmentation. In: Medical image computing and computer assisted intervention–MICCAI 2021, Strasbourg, France, September 27–October 1, 2021, pp. 428–438. Springer.

47.

Kirillov

Girshick

, et al. Panoptic feature pyramid networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, CA, USA, 2019, pp. 6399–6408.

48.

Chen

L-C

Zhu

Papandreou

, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV), Munich, Germany, 2018, pp. 801–818.