Hybrid connected attentional lightweight network for gangue intelligent segmentation in top-coal caving face

Abstract

The estimation of gangue content is the main basis for intelligent top coal caving mining by computer vision, and the automatic segmentation of gangue is crucial to computer vision analysis. However, it is still a great challenge due to the degradation of images and the limitation of computing resources. In this paper, a hybrid connected attentional lightweight network (HALNet) with high speed, few parameters and high accuracy is proposed for gangue intelligent segmentation on the conveyor in the top-coal caving face. Firstly, we propose a deep separable dilation convolution block (DSDC) combining deep separable convolution and dilation convolution, which can provide a larger receptive field to learn more information and reduce the size and computational cost of the model. Secondly, a bridging residual learning framework is designed as the basic unit of encoder and decoder to minimize the loss of semantic information in the process of feature extraction. An attention fusion block (AFB) with skip pathway is introduced to capture more representative and distinctive features through the fusion of high-level and low-level features. Finally, the proposed network is trained through the expanded dataset, and the gangue image segmentation results are obtained by pixel-by-pixel classification method. The experimental results show that the proposed HALNet reduces about 57 percentage parameters compared with U-Net, and achieves state-of-the art performance on dataset.

Keywords

Gangue intelligent segmentation the top-coal caving face depthwise separable dilation convolution attention mechanism

1 Introduction

The top-coal caving mining method that causes the top coal to collapse under the action of gravity is a high efficiency and high yield method for mining thick coal seams [1, 2]. In order to balance the quality and output of coal, it is inevitable that a large amount of gangue is mixed into the raw coal in the process of top coal caving mining. The latest research found that when the mixed gangue rate is 10% 15%, the coal recovery rate reaches the maximum [3]. At present, workers usually control the hydraulic support by identifying the gangue content on the scraper conveyor to realize the decision of “discharge” or “stop discharge” [4]. With the rapid development of artificial intelligence technology, the intelligent top coal caving mining technology [5] can effectively solve the problems of “under discharge” and “over discharge” in manual top coal caving mining, improve the coal recovery rate and reduce the cost of gangue washing. Therefore, the automatic identification of gangue content is the key technology to realize intellectualization of top coal caving mining process.

As a frontier problem in the field of top coal caving mining, many coal gangue recognition technologies have been widely studied in the early stage. At present, the popular methods mainly include infrared detection [6 –8], vibration signal [2 , 9–11] and dielectric characteristics [12, 13]. These methods use different feature vectors of coal and gangue (such as sound pressure signal and capacitance value) and classification model to identify coal and gangue. However, these methods can only distinguish the two stages of “coal discharge” and “gangue discharge”, but cannot identify the gangue content information. At present, the top coal caving process completely depends on the visual experience of workers to identify the content of gangue [2, 14]. Therefore, computer vision instead of human vision has self-evident advantages in automatic detection of gangue content in complex environment.

So far, a number of literatures are related to the research of coal gangue image recognition. However, the related research mainly focuses on the coal and gangue image classification of the washing plant (i.e. detecting whether there is gangue or coal in the image) and the coal rock interface recognition of the coal mining face (i.e. locating the spatial position of coal seam and rock stratum in the image). The results of these studies cannot provide the content information of gangue in the image. Image segmentation can provide the location and regional characteristics of the object of interest, which is an important task in the field of computer vision. It is mainly divided into traditional segmentation methods [15 –19] and deep learning methods [20 –23]. Compared with traditional segmentation technology, deep learning can automatically learn feature representation from images without introducing manual coding rules or human domain knowledge [24]. Because of its strong adaptability and accuracy, it has become one of the most popular scientific research trends [25]. Although these methods reported in relevant literature have obtained acceptable recognition accuracy in sparse gangue images, they have some limitations in the application of complex gangue images randomly stacked on scraper in fully mechanized top coal caving face with low illumination, high noise and uneven dust concentration distribution. Considering the fuzzy boundary and complex gradient of gangue image, more high-resolution information is needed for accurate segmentation. In addition, its structure and semantic information are relatively simple, and low-resolution information is helpful for recognition. Therefore, a deep learning framework that can integrate low-resolution information and high-resolution information is preferred.

U-shaped full convolution neural network (U-Net) has achieved great success in the field of small sample medical image segmentation [26] and coal gangue image segmentation [22]. However, the simple feature fusion of U-Net will lead to poor feature extraction ability. In addition, the large-scale and slow reasoning speed limit its application in practice. Therefore, it is an urgent problem to design a small-scale, fast and efficient gangue segmentation network. The potential of lightweight network in practical application stimulate our exploration of segmenting gangue image of top coal caving face through deep learning algorithm. As far as we know, this is the first time to consider intelligent gangue segmentation through network learning in the process of top coal caving mining. In order to obtain higher accuracy of gangue segmentation with limited hardware resources, a hybrid connected attentional lightweight network (HALNet) was proposed for gangue intelligent segmentation on the conveyor of top-coal caving face. Depthwise separable dilation convolution (DSDC) factorizes a standard dilation convolution into two parts to expand the receptive field and reduce computational cost and model size [27]. The design of bridging residual learning framework can minimize the loss of semantic information in the process of feature extraction. The attention fusion block (AFB) can highlight the gangue features of the scraper area and suppress irrelevant areas in the image with little computational cost. The redesigned skip path further reduces the semantic gap between feature mapping and more comprehensive segmentation of gangue. The results of the present research indicated that the proposed method can effectively segment the coal gangue on the scraper in the top coal caving face, which provides a theoretical basis for the intelligent top coal caving mining technology.

The contributions of our work are as follows:

A DSDC module combining deep separable convolution and dilation convolution with different dilation rates is proposed, which can effectively obtain receptive field information of different sizes and extract features more efficiently while reducing the amount of model calculation.

A bridging residual learning framework is designed to improve the feature extraction ability of deep network and obtain more representative features.

An AFB module with skip path connection is introduced to model semantic dependencies between channels and emphasize the gangue regions, which contributes to localization and semantic segmentation of gangue on the scraper conveyor.

This paper is organized as follows. In Section 2, we review state-of-the-art research of related work. The proposed method for gangue segmentation will be detailed in section 3. The detailed experimental results and analysis are presented in Section 4. Finally, Section 5 provides some conclusion and future exploration work.

2 Related works

The detection of gangue content on scraper conveyor is a key step in the intellectualization of top coal caving mining process. In recent years, researchers have studied methods for coal gangue identification, such as γ-ray detection, radar detection, cutting force response, infrared detection, vibration signal, dielectric characteristics and image detection [4]. At present, the research of top coal caving face mostly focuses on the application of vibration signal to identify coal gangue. This method only realizes the strategy of “stop discharging in case of gangue”, and cannot judge the mixed gangue rate [3]. So far, automatic coal caving still needs to control the coal caving process in combination with workers’ visual observation of the content of gangue in the released mixture [28]. As a non-contact detection technology, computer vision instead of human vision has been widely used in driverless vehicles, medical diagnosis, mineral processing and many other fields.

Wang et al. [29] proposed a gangue image recognition method combining traditional spatial processing with wavelet transformation, and the recognition rate reached 95.12%. In order to further improve the recognition accuracy, some typical deep learning networks are introduced to recognize coal and gangue. Pu et al. [23] established a self-defined convolutional neural network (CNN) model to distinguish coal and gangue by introducing the idea of transfer learning. Li et al. [21] and Liu et al. [20] respectively applied the improved yolov3 and yolov4 algorithms for intelligent and high-precision identification of coal and coal gangue. These studies mainly focus on the classification of coal and gangue, which cannot obtain the gangue content information in the image, and the image segmentation technology can effectively solve this problem.

Sun et al. [15] proposed a single target super-pixel segmentation algorithm of coal and gangue based on color, spatial position and texture fusion. Wang et al. [30] combined the edge detection method of star algorithm with morphological method to complete the fine segmentation of multi-objective coal gangue image with simple background and sparse distribution. Li et al. [18] developed a positioning and recognition system for segmenting coal gangue under solid color background. These traditional image segmentation methods are effective for simple background image or single target image segmentation. In order to improve the segmentation performance of multi-target image in complex background, Gao et al. [22] used U-Net model to segment coal gangue from raw coal images collected under complex conditions, and the segmentation accuracy reached 0.93. However, the simple feature fusion of U-Net leads to the poor feature extraction ability of gangues randomly stacked and buried in the degradation scene of fully mechanized top coal caving face. In addition, large-scale redundant parameters and slow reasoning speed limit its deployment in hardware equipment and practical application. So far, there is no relevant literature report that can process the coal gangue image on the conveyor in this complex environment.

In order to improve the feature learning ability of U-Net, the residual block [31] replaces the convolution block to overcome the degradation problem of deeper network feature extraction. The segmentation accuracy has been significantly improved. However, the direct data fusion brings in a large number of redundant parameters, which increases the computational intensity and complexity. In addition, the attention module helps the network focus on key areas by imitating human attention mechanism. Guo et al. [32] and Zhang et al. [33] embedded the channel attention module into U-Net and residual U-Net respectively, which improved the segmentation performance of retinal vessels and brain tumors. This method only focuses on the local important feature mapping, which is easy to produce semantic gap. Without reducing the resolution of feature mapping, Ma et al. [34] used dilation convolution to increase the receptive field to obtain rich multi-scale information of remote sensing images and improve the feature representation ability of the U-Net.

In order to reduce the computational efficiency and the amount of parameters of the network, the lightweight network named MobileNet [27] introduced depthwise separable convolution instead of the traditional convolution to reduce the model size and computational cost. Due to the degradation of image and the limitation of computing resources, these works cannot achieve better segmentation performance for gangue image in top coal caving face. Therefore, in order to meet the demand of obtaining higher coal gangue segmentation accuracy under limited hardware resources, the HALNet is proposed for gangue intelligent segmentation on the conveyor of top-coal caving face.

3 Materials and methods

3.1 Materials

3.1.1 Data collection

The sample images were randomly captured at a resolution of 1280×720 pixels from a video taken by an explosion-proof cameras (IPG-85HE20PY-S) in 8105 working faces of the Zhuxianzhuang Coal Mine, Huaibei Mining Group. These included 200 training images and 25 verification images, and all gangues on the scraper in these randomly captured images are labeled. The data collection and the captured image are shown in Fig. 1. The images captured by the camera were transmitted to the computer through Gigabit Ethernet protocol for display and storage. In order to ensure the diversity of sample images, there are 9200 training images and 75 verification images are generated via data augmentation operations (the specific description can be seen in section 3.1.3), eventually.

Fig. 1

Data collection and the captured image: (a) Data collection system; (b) The representative images captured randomly; (c) The ground truth of (b).

3.1.2 Data preparation

Due to the high dust concentration in the top coal caving face, the collected image has low contrast, high haze and serious noise. It is difficult to label and identify gangue without preprocessing. Here, the Retinex-based [35] image enhancement method is used to remove dust and enhance image contrast. A visual comparison of dusty image and enhanced image is shown in Fig. 2. It can be clearly seen that the enhanced images (Fig. 2b) provide richer, more detail information and have a better visual effect than the dusty image with a gray layer (Fig. 2a). Considering the speed of model training, the image size is reset to 512×512 pixels during training.

Fig. 2

Example of image preprocessing. (a) Original image. (b) Enhanced image.

In order to imitate the strong recognition capabilities of human vision for gangue and teach the HALNet to distinguish the gangue on the scraper from the interferents, we mark the image as gangue or background through pixel level annotation tool. Due to the serious degradation of image quality and the time-consuming of labeling, the ground truth was not perfect, which is also in line with human visual perception. In the experiment, 225 pairs of original images and ground truth images were utilized to train and test the proposed HALNet.

3.1.3 Data augmentation

It is time-consuming and laborious to manually label gangue images of the top coal caving face. In order to increase the diversity of data, reduce the over-fitting in model training and improve the generalization ability of the model, data augmentation is an optimization strategy. Therefore, we performed data enhancement operations on the training data. It includes basic geometric transformation (such as rotation, flip, movement and scale transformation) and scene migration [36]. Scene migration enhances the robustness of the algorithm in changeable environments and improved the generalization ability of the model by exchanging the low-frequency spectrum of training images and different scene images. Figure 3 shows the data augmentation results under scene migration.

Fig. 3

Data augmentation under scene migration. (a) Original image. (b) Image of scene A. (c) Image of scene B. (d) Augmented image in scene A style. (e) Augmented image in scene B style

3.1.4 Implementation

The proposed HALNet model is trained using captured images and the corresponding ground truth. Our network experiment is implemented in Pycharm software environment using Keras library and TensorFlow backend of Ubuntu 16.04. The Adam optimizer (learning rate = 0.001, exponential decay rates = 1e-6) is used to minimize the binary cross-entropy loss. The batch size of the training is set to 4 and the training epochs is 50. All experiments are performed on the hardware platform of Intel(R)Core(TM) i7-5960x CPU@3.00 GHz, 2TB memory, and the NVIDIA GeForce GTX1080(X2) GPU. Data augmentation is performed to increase the diversity of samples, contributing to improving network performance.

3.2 Proposed methods

Due to the degradation of data and the limitation of computing resources, the application of deep learning model in intelligent mining of coal mine is very difficult, especially in top coal caving face. Therefore, the HALNet was proposed to segment the gangue on the scraper in the top coal caving face and obtain the gangue content information indirectly. The encoder-decoder architecture of the HALNet is presented in Fig. 4. The deep convolution neural network with residual learning framework [37] can extract the deep features of the image and effectively prevent the gradient disappearance/explosion and accuracy degradation during network training. Therefore, in order to reduce the computational cost, the lightweight residual block composed of depthwise separable dilation convolution designed as bridging residual structure is used to replace the ordinary convolution in the original U-net network, so as to improve the learning ability of the network, as shown in Fig. 4(b).

Fig. 4

The architecture of proposed hybrid connected attentional lightweight network and its components. (a) Overall architecture. (b) Lightweight residual block (c) Attention fusion block.

The encoder mainly focuses on extracting the semantic features of gangue by gradually reducing the resolution of the image through the down sampling operation. Here, the convolution operation with step size of 2 and the maximum pool operation are combined to reduce the size of feature map, rather than one of them.

The depthwise separable dilation convolution module replaces the ordinary convolution to extract features intensively. Different dilation rates (r = 1, 2, 4, 6 and 8) are set in different convolution layers to expand the receptive field and effectively obtain context information of different scales, as shown in Fig. 4(a). The decoder mainly up-samples the feature image through bilinear interpolation and convolution, and gradually restores the resolution to the level of the original image. The lightweight attention block and the re-designed skip pathways are combined to replace the simple skip connection of the original U-net to obtain richer semantic and spatial information (Fig. 4(a) and (c)). It only needs little computational cost, which is helpful to the deployment of gangue segmentation algorithm. The lightweight residual block and attention fusion block will be introduced in detail next.

Fig. 5

Depthwise separable convolution.

3.2.1 Lightweight residual block

In depth learning, the depth network designed as residual connection structure can better learn abstract features and prevent the disappearance of gradient in the training process [24 , 38]. Therefore, feature extraction through the combination of depthwise separable convolution and residual connection is a common method in lightweight networks [39, 40]. The encoder and decoder use the depthwise separable dilation convolution with different residual structure as the basic unit instead of the standard convolution in the proposed HALNet model, as shown in Fig. 4(b). In this structure, the gradient of the upper layer can quickly propagate back to the lower layer. In addition, the down sampling process uses the residual block of the combination of convolution operation with step size of 2 and maximum pool operation to fuse the feature data and extract the features more fully. The depthwise separable convolution decomposes the standard convolution into a depthwise convolution and a pointwise convolution [27], which greatly reduces the parameters and computation in the network, as shown in Fig. 5.

Assuming that the feature map of H × H × D₁ is taken as the input, the convolution operation is carried out through the convolution kernel with the size of h × h to generate the feature map of M × M × D₂, where D₁ and D₂ is the number of feature map channels. The number of parameters and computational cost of standard convolution are D₁ · h · h · D₂ and h · h · D₁ · D₂ · H · H, respectively. The number of parameters and computational cost of depthwise separable convolution are D₁ · h · h + D₁ · D₂ and h · h · D₁ · H · H + D₁ · D₂ · H · H, respectively. The ratio of the depthwise separable convolution and the standard convolution parameter is shown in Equation (1). When the convolution kernel size h = 3, the parameters and computation of deep separable convolution are reduced by about 9 times compared with standard convolution, which can greatly reduce the number of parameters and improve the operation efficiency. $\begin{matrix} \frac{D_{1} \cdot h \cdot h + D_{1} \cdot D_{2}}{D_{1} \cdot h \cdot h \cdot D_{2}} = \frac{1}{D_{2}} + \frac{1}{h^{2}} \\ = \frac{h \cdot h \cdot D_{1} \cdot H \cdot H + D_{1} \cdot D_{2} \cdot H \cdot H}{h \cdot h \cdot D_{1} \cdot D_{2} \cdot H \cdot H} \end{matrix}$ (1)

In addition, the dilation convolution with different dilation rates (r = 1,2,4,6,8) is used instead of depthwise convolution to expand the receptive field, which can effectively fuse multi-scale information without increasing the amount of parameters, as shown in Fig. 4(a) and 4(b).

3.2.2 Attention fusion block

The decoder of the original U-net network fuses the low-level features with rich spatial information and the high-level features with rich semantic information through the up-sampling and skip connection structure. However, the low-level features of gangue image contain a lot of useless background information, which may interfere with the segmentation of target objects through simple fusion. In order to solve this problem, the AFB module is designed to emphasize the target location details in the low-level feature maps and capture the global context and semantic information in the high-level feature maps, so as to improve the representation of features, as shown in Fig. 4(c). AFB employs global average pooling (GAP) to capture global context and computes an attention vector to guide the feature learning, which helps to emphasize key features and filter background information [41], as described in Equation (2) for the feature map size of W × H × D. $G (x_{k}) = \frac{1}{W \times H} \sum_{i = 1}^{W} \sum_{j = 1}^{H} x_{k} (i, j)$ (2) where x refers to input feature map, k = [1, 2, . . . . . , D] and x = [x₁, x₂, . . . . . , x_D], i and j are the coordinate index of pixel in the feature map.

Formally, the high-level features x_h are firstly performed up-sampling and the features X_h are generated. Then input low-level features x_l and the features X_h through the average-pooling generate G (X_h) and G (x_l), respectively. Then a convolutional layer followed by the sigmoid activation function on the concatenated feature descriptor is used to generate the attentive vector, as shown in Equation (3). The output feature map x^s is generated by Equation (4). $A_{c} = σ (C^{1 \times 1} (δ (C^{1 \times 1} (G (X_{h})) ⊙ C^{1 \times 1} (G (x_{l})))))$ (3) where σ (·) represents the sigmoid function and δ (·) refers to ReLU function; C^1×1 (·) and ⊙ denote a 1×1 convolution operation and the concatenation operation, respectively. $x^{5} = A_{c} \otimes x_{l} \oplus X_{h}$ (4) where ⊗ and ⊕ denotes broadcast element-wise multiplication and addition, respectively.

Finally, the redesigned skip path fuses the output features x^s with low-level features x_l to reduce the semantic gap between the feature mapping of encoder and decoder sub networks. The proposed AFB module can suppress the interference of gangue image in non-scraper area and scraper components, highlight more gangue feature information, and is conducive to rapid gangue positioning in scraper area. In addition, the global average pool compresses the global information into a vector, which greatly reduces the computational cost, so the module dose not add too many parameters.

3.2.3 Evaluation metrics

In order to evaluate the performance of our model, we compare the segmentation results with the corresponding ground truth and divide the results of each pixel comparison into true positive (TP), false positive (FP), true negative (TN) and false negative (FN). In gangue image segmentation, the proportion of target and background areas is seriously unbalanced. The Matthews correlation coefficient (MCC) is suitable for the performance measurement of binary classifications for two categories with different sizes [32]. Therefore, the Precision, Dice similarity coefficient (Dice), Matthews Correlation Coefficient (MCC), Accuracy (ACC), Parameters and Frames Per Second (FPS) are used to evaluate the performance of the model. In addition to the parameters, the larger the value of other evaluation indexes, the better the segmentation effect. In the experiment, the mean value of all test pictures was used as the evaluation index. The calculations are shown below.

The precision is described as: $Precision = TP / (TP + FP)$ (5)

Dice represents the similarity between the segmentation result and the ground truth, which is described as $Dice = 2 TP / (2 TP + FP + FN)$ (6)

MCC and ACC are defined as: $\begin{matrix} MCC = \\ \frac{TP \cdot TN - FP \cdot FN}{\sqrt{(TP + FP) \cdot (TP + FN) \cdot (TN + FP) \cdot (TN + FN)}} \end{matrix}$ (7) $A CC = \frac{TP + TN}{FN + FP + TP + TN}$ (8)

FPS represents the algorithm speed, which is defined as follows: $FPS = N / \sum_{j}^{N} T_{j}$ (9) where N is the number of images, and T_j is the time taken to process the j-th image.

4 Results and discussion

4.1 Ablation experiments

In order to prove that each component of the proposed HALNet can improve the performance of gangue segmentation, ablation experiments were performed to validate the effect of each component of HALNet. The same training environments for all models. Table 1 show the segmentation performance of Unet, D-Unet (i.e. Unet+depthwise separable convolution), DR-Unet (i.e. D-Unet+bridging residual structure) DRA-Unet (i.e. DR-Unet+AFB), DRAD-UNet (i.e. DRA-Unet+dilation convolution) and ours (i.e. DRAD-UNet+re-designed skip connections) from top to bottom, respectively.

Table 1
Detailed performance comparison of each component in our proposed HALNet

Method Precision Dice MCC ACC Parameters (KB)

Unet 0.7361 0.7887 0.7751 0.9579 23516

D-Unet 0.7159 0.7503 0.7309 0.9508 3298

DR-Unet 0.8022 0.791 0.7735 0.9609 8230

DRA-Unet 0.8068 0.7932 0.7761 0.9624 8934

DRAD-UNet 0.8276 0.8111 0.7934 0.9659 8934

Ours 0.8684 0.8415 0.8291 0.9726 10211

Method	Precision	Dice	MCC	ACC	Parameters (KB)
Unet	0.7361	0.7887	0.7751	0.9579	23516
D-Unet	0.7159	0.7503	0.7309	0.9508	3298
DR-Unet	0.8022	0.791	0.7735	0.9609	8230
DRA-Unet	0.8068	0.7932	0.7761	0.9624	8934
DRAD-UNet	0.8276	0.8111	0.7934	0.9659	8934
Ours	0.8684	0.8415	0.8291	0.9726	10211

The results show that, (1) although the segmentation accuracy of D-Unet is slightly lower than that of original Unet, the amount of parameters is far lower than that of Unet, which is helpful for timely segmentation on the platform with limited calculation in practical application. (2) DR-Unet has better performance compared with the D-Unet and Unet, although the parameter amount is increased (but still far lower than Unet), which indicates that adding the bridging residual structure can improve the network performance to a certain extent. (3) The DRA-Unet, DRAD-Unet and HALNet (ours) all achieve better segmentation performance than Unet for gangue segmentation, which proves the strategy of the introduction of attention mechanism and dilation convolution are effective. In particular, compared with the original Unet, our HALNet not only reduces the parameters by half, but also the Precision, Dice and MCC are significantly increased by 0.1323, 0.0528 and 0.054, respectively. Therefore, HALNet is a lightweight and effective network for the gangue segmentation task on the scraper in the top coal caving face.

4.2 Comparisons with state-of-the-art methods

Finally, we compare the performance of HALNet with other state-of-the-art methods currently applied in gangue segmentation task and related frameworks (i.e., Unet [22], Res-Unet [31], ARes-Unet [33] and SA-Unet [32] on our test dataset. The comparison of gangue segmentation results is shown in Fig. 6. As can be seen in every green/red rectangle in Fig. 6, our method makes the segmentation results closer to the ground truth without too many over segmented and under segmented regions, which confirms the advantages of our method in degraded image segmentation. Although the result of Unet segmentation seems to be more comprehensive than other methods, this large-area excessive segmentation seriously interferes with the judgment of the actual gangue content, especially when the low content of coal gangue mixture leads to the exposure of scraper components similar to the gray scale of gangue, such as the second column and third column in Fig. 6. Due to the integration of more semantic information in the global context, the attention of important features and the suppression of unimportant features, the over segmentation phenomenon of Res-Unet, ARes-Unet and SA-Unet is significantly improved compared with that of Unet, but it still exists, and even a large-area undersegmentation phenomenon appears, such as column 4 and column 5 in Fig. 6.

Fig. 6

Image segmentation results of different methods.

Fig. 7

The loss (left) and accuracy (right) curves of different methods during training.

By contrast, our proposed HALNet almost yielded very comparative and even better results than these competitive methods. Although the results also have a small proportion of under segmentation and over segmentation (compared with the ground truth in Fig. 6), it has little impact on the estimation of the proportion of gangue content. This is also in line with the global estimation characteristics of human vision. In addition, we summarize the performance of different methods through six quantitative evaluation indexes in Table 2.

Table 2

Performance comparison of our method against other state-of-the-art methods

Method	Precision	Dice	MCC	ACC	Parameters (KB)	FPS
Unet	0.7361	0.7887	0.7751	0.9579	23516	9
Res-Unet	0.8168	0.8239	0.8087	0.9678	55703	6
ARes-Unet	0.822	0.825	0.8103	0.9685	56405	6
SA-UNet	0.8263	0.817	0.8017	0.9665	23977	9
HALNet	0.8684	0.8415	0.8291	0.9726	10211	11

From the results in the table, it can be concluded that HALNet has achieved the best performance. It achieves the highest Precision of 0.8684, the highest Dice of 0.8415, the highest MCC of 0.8291, and the highest ACC of 0.9726. Most notably, the number of parameters is only about 2/5 of Unet and 1/5 of Res-Unet, which greatly improves the operation efficiency and reaches 11FPS. The running speed of scraper in 8105 working faces of the Zhuxianzhuang Coal Mine is 1.6 m/s, and the capture range of the camera is about 1 m. Therefore, there is no need to process each frame image, and the inter frame processing can meet the requirements. Since Res-Unet and ARes-Unet are not satisfied, they are omitted in the following analysis.

To further observe the performance of the proposed HALNet, we compare the training accuracy and training loss curve of our method with other methods, which is shown in Fig. 7. Comparatively speaking, our proposed HALNet almost has achieved the highest accuracy values and the lowest loss values than other methods, and the convergence rate of HALNet is the fastest.

Based on the above analysis, it indicates that our proposed method achieves better gangue segmentation performance in the complex environment of top coal caving face, and meets the real-time segmentation effect under special needs.

5 Conclusions

In this paper, we propose a hybrid connected attentional lightweight network named HALNet for the intelligent segmentation of gangue in top coal caving face. Gangue image dataset is a typical small sample dataset. Coupled with the serious image degradation in harsh environment, it is difficult to obtain a large number of annotated training samples. Firstly, a set of comprehensive data augmentation is applied to increase the diversity of training samples to prevent over-fitting during training. Then a novel lightweight network structure is proposed and it is trained to segment gangue on the scraper conveyor in the complex environment of top coal caving face. Besides, the trained network is able to real-time segment gangue under special needs with an accuracy of 0.97 while takes very low computational costs and the content information of coal gangue is obtained through the calculation of gangue proportion. Finally, the experiment proves that our strategy of redesigning skip pathways and the introduction of DSDC, bridging residual learning framework and AFB are effective. Compared with other state-of-the-art methods for gangue segmentation, our lightweight HALNet achieves state-of-the-art performance, especially for segmentation of images with extremely similar gray levels between the components exposed by the scraper and the gangue when the content of coal gangue mixture is low. So the proposed algorithm has great potential to realize the gangue intelligent segmentation of top coal caving face in complex environment while takes little computational costs. However, the dataset in this study is mainly based on a series of image data collected from the underground top coal caving face. The training model in this study may not be able to directly obtain similar segmentation results for the test images collected in different scenes. However, the proposed segmentation model can obtain more satisfactory segmentation results by training more different scene data sets, which is also the future work of this research.

Footnotes

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant No. U1704242). The authors thank the Coal Industry Engineering Research Center of Top-coal Caving Mining for its help. The authors declare no conflicts of interest.

References

Zhang

Q.L.

, Yue

J.C.

, Liu

, Feng

and Li

H.M.

, Study of automated top-coal caving in extra-thick coal seams using the continuum-discontinuum element method, International Journal of Rock Mechanics and Mining Sciences 122 (2019).

Liu

, Li

, Wang

, Lu

, Zhao

and Yang

, Coal-Gangue Interface Detection Based on Ensemble Empirical Mode Decomposition Energy Entropy, IEEE Access 9 (2021), 54639–54648.

Wang

J.C.

, Pan

W.D.

, Zhang

G.Y.

, Yang

S.L.

, Yang

K.H.

and Li

L.H.

, Principles and applications of image-based recognition of withdrawn coal and intelligent control of drawing opening in longwall top coal caving face, Journal of China Coal Society (2021), 1–16.

Pang

H.D.

, Wang

S.B.

, Dou

X.J.

, Liu

H.G.

, Chen

, Yang

S.G.

, Wang

and Wang

S.Y.

, A Feature Extraction Method Using Auditory Nerve Response for Collapsing Coal-Gangue Recognition, Applied Sciences-Basel 10 (2020).

, Xu

, Huang

, Guo

, Li

, Wang

, Meng

, Pan

, Niu

, Xue

and Zhao

, Theory and its key technology framework of intelligentized fully-mechanized caving mining in extremely thick coal seam, J China Coal Soc 44 (2019), 42–53.

Zou

, Yu

X.H.

, Li

, Lei

and Yu

, Nondestructive Identification of Coal and Gangue via Near-Infrared Spectroscopy Based on Improved Broad Learning, IEEE Transactions on Instrumentation and Measurement 69 (2020), 8043–8052.

Ding

Z.H.

, Chen

G.D.

, Wang

, Chi

W.Z.

, Wang

Z.H.

and Fan

Y.L.

, A Real-Time Multilevel Fusion Recognition System for Coal and Gangue Based on Near-Infrared Sensing, IEEE Access 8 (2020), 178722–178732.

Eshaq

R.M.A.

, Hu

E.Y.

, Li

M.G.

and Alfarzaeai

M.S.

, Separation Between Coal and Gangue Based on Infrared Radiation and Visual Extraction of the YCbCr Color Space, IEEE Access 8 (2020), 55204–55220.

Yang

and Zeng

Q.L.

, Impact-slip experiments and systematic study of coal gangue “category” recognition technology Part I: Impact-slip experiments between coal gangue mixture and top coal caving hydraulic support and the study of coal gangue “category” recognition technology, Powder Technology 392 (2021), 224–240.

10.

Liu

, He

, Liu

C.Y.

, Gao

and Yan

Y.H.

, Coal-gangue interface detection based on Hilbert spectral analysis of vibrations due to rock impacts on a longwall mining machine, Proceedings of the Institution of Mechanical Engineers Part C-Journal of Mechanical Engineering Science 229 (2015), 1523–1531.

11.

Zhang

G.X.

, Wang

Z.C.

and Zhao

, Recognition of rock-coal interface in top coal caving through tail beam vibrations by using stacked sparse autoencoders, Journal of Vibroengineering 18 (2016), 4261–4275.

12.

Guo

Y.C.

, Wang

X.Q.

, Wang

, Hu

and Wang

W.S.

, Identification Method of Coal and Coal Gangue Based on Dielectric Characteristics, IEEE Access 9 (2021), 9845–9854.

13.

Wang

X.Q.

, Wang

, Guo

Y.C.

, Hu

and Wang

W.S.

, Recognition of coal and gangue based on dielectric characteristics and geometric constraints under multi factors, Energy Sources Part a-Recovery Utilization And Environmental Effects (2021).

14.

Zhang

N.B.

and Liu

C.Y.

, Radiation characteristics of natural gamma-ray from coal and gangue for recognition in top coal caving, Scientific Reports 8 (2018).

15.

Sun

, Xuan

, Song

, Li

and Jia

, A texture fused superpixel algorithm for coal mine waste rock image segmentation, International Journal of Coal Preparation and Utilization (2019), 1–12.

16.

Sun

, Lu

, Xuan

, Li

, Zhang

, Niu

and Jia

, Separation of gangue from coal based on supplementary texture by morphology, International Journal of Coal Preparation and Utilization (2019), 1–17.

17.

Sun

, Jia

, Hei

, Yang

, Cheng

, Li

, Wang

and Tang

, Application of concave point matching algorithm in segmenting overlapping coal particles in X-ray images, Minerals Engineering 171 (2021).

18.

, Duan

, He

and Yang

, Image positioning and identification method and system for coal and gangue sorting robot, International Journal of Coal Preparation and Utilization (2020), 1–19.

19.

Yang

, Peng

, Li

, Xie

, Wu

and Wang

, Gangue Localization and Volume Measurement Based on Adaptive Deep Feature Fusion and Surface Curvature Filter, IEEE Transactions on Instrumentation and Measurement 70 (2021), 1–13.

20.

Liu

, Li

and Gao

, Recognition Methods for Coal and Coal Gangue Based on Deep Learning, IEEE Access 9 (2021), 77599–77610.

21.

D.Y.

, Wang

G.F.

, Zhang

and Wang

, Coal gangue detection and recognition algorithm based on deformable convolution YOLOv3, IET Image Processing (2021).

22.

Gao

, Sun

, Li

, Pei

, Hu

and Xiao

, Automatic Coal and Gangue Segmentation Using U-Net Based Fully Convolutional Networks, Energies 13 (2020).

23.

, Apel

D.B.

, Szmigiel

and Chen

, Image Recognition of Coal and Coal Gangue Using a Convolutional Neural Network and Transfer Learning, Energies 12 (2019).

24.

LeCun

, Bengio

and Hinton

, Deep learning, Nature 521 (2015), 436–444.

25.

Chouhan

S.S.

, Kaul

and Singh

U.P.

, Image Segmentation Using Computational Intelligence Techniques: Review, Archives of Computational Methods in Engineering 26 (2018), 533–596.

26.

Ronneberger

, Fischer

and Brox

, U-Net: Convolutional Networks for Biomedical Image Segmentation, in 18th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Lecture Notes in Computer Science (2015), 234–241.

27.

Howard

A.G.

, Zhu

, Chen

, Kalenichenko

, Wang

, Weyand

, Andreetto

and Adam

, MobileNets: Efficient convolutional neural networks for mobile vision applications, in Computer Vision and Pattern Recognition, (Hawaii · United States, 2017).

28.

Shouxiang

, Xueliang

, Shuai

and Guoqing

, Intelligent precise control technology of fully mechanized top coal caving face, Journal of China Coal Society 45 (2020), 2008–2020.

29.

Wang

, Lv

and Lu

, Research onmethods to differentiate coal and gangue using image processing and a support vector machine, International Journal of Coal Preparation and Utilization (2018), 1–4.

30.

Wang

, Wang

, Guo

, Hu

and Wang

, Coal gangue image segmentationmethod based on edge detection theory of star algorithm, International Journal of Coal Preparation and Utilization (2021).

31.

Khanna

, Londhe

N.D.

, Gupta

and Semwal

, A deep Residual U-Net convolutional neural network for automated lung segmentation in computed tomography images, Biocybernetics And Biomedical Engineering 40 (2020), 1314–1327.

32.

Guo

C.L.

, Szemenyei

, Yi

Y.G.

, Wang

W.L.

, Chen

B.E.

, Fan

C.Q.

and Ieee Comp

S.O.C.

, SA-UNet: Spatial Attention U-Net for Retinal Vessel Segmentation, in 25th International Conference on Pattern Recognition (ICPR), International Conference on Pattern Recognition (2021), 1236–1242.

33.

Zhang

J.X.

, Lv

X.G.

, Zhang

H.B.

and Liu

, A ResU-Net: Attention Residual U-Net for Brain Tumor Segmentation, Symmetry-Basel 12 (2020).

34.

B.F.

and Chang

C.Y.

, Semantic Segmentation of High-Resolution Remote Sensing Images Using Multiscale Skip Connection Network, IEEE Sensors Journal 22 (2022), 3745–3755.

35.

C.C.

, Lu

F.L.

, Zhang

X.X.

and Zhang

G.Y.

, Joint Dedusting and Enhancement of Top-Coal Caving Face via Single-Channel Retinex-Based Method with Frequency Domain Prior Information, Symmetry-Basel 13 (2021).

36.

Yang

and Soatto

, FDA: Fourier domain adaptation for semantic segmentation, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2020), 4085–4095.

37.

, Zhang

, Ren

and Sun

, Deep residual learning for image recognition, in Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), 770–778.

38.

Heaven

, Deep Trouble for Deep Learning, Nature 574 (2019), 163–166.

39.

Wang

, Zhou

, Liu

, Xiong

, Gao

, Wu

and Latecki

L.J.

, Lednet: A lightweight encoder-decoder network for real-time semantic segmentation, in 2019 IEEE International Conference on Image Processing (ICIP), (IEEE, 2019), 1860–1864.

40.

Romera

, Alvarez

J.M.

, Bergasa

L.M.

and Arroyo

, Erfnet: Efficient residual factorized convnet for real-time semantic segmentation, IEEE Transactions on Intelligent Transportation Systems 19 (2017), 263–272.

41.

C.Q.

, Wang

J.B.

, Peng

, Gao

C.X.

, Yu

and Sang

, BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation, in 15th European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science (2018), 334–349.