Obstacle Detection Method of Underground Electric Locomotive Rail Based on Instance Segmentation

Abstract

Real-time and accurate obstacle detection is a vital technology for electric locomotives, especially as driverless vehicles are introduced. A method of obstacle detection for underground electric locomotive rail based on instance segmentation is developed to solve the problems of misdetection and missing detection, low detection accuracy, and slow detection speed of rail obstacles. The method of locating the track mask, demarcating the effective driving boundary, expanding the track mask, and forming the effective driving area is adopted to verify whether the target is an obstacle based on whether the target is located in the effective driving area, to avoid the problem of misdetection and missing detection of the target obstacle. The YOLACT++ (You Only Look At CoefficienTs) model is improved, and path augmentation and target classification loss function replacement strategies are adopted to enhance the model’s ability to detect target details and increase the accuracy of target segmentation. Compared with traditional image processing, this method can detect both straight rail and turnout. The mean average precision of boundary box mAP_0.5(box) and mask mAP_0.5(mask) of the improved YOLACT++ model reaches 98.52% and 98.55%, which is higher than that of the YOLACT++ model, and the detection frame rate reaches 21.9 frames per second.

Keywords

rail rail safety computer models system safety train

Coal is still an essential raw material and the main energy source in China, and has a critical significance in the nation’s economic development ( 1 – 3 ). Underground rail is an important part of the coal auxiliary transportation system, responsible for the transportation of coal, materials, and personnel in coal mines. In China most electric locomotives in coal mines are manually driven, so safety problems (e.g., pedestrian collision, rear-end collision, and derailment) are easily caused by narrow underground roadways and driver fatigue ( 4 , 5 ). Accordingly, the research and development of driverless technology for locomotives in coal mine auxiliary transportation is of great significance in the reduction of coal mine personnel and improved efficiency, as well as safer coal production ( 6 ).

Active obstacle detection is an essential function for a driverless electric locomotive. It is also an important safety function for human-driven vehicles. It mainly transmits instructions to the driver or control center when foreign objects invade the driving area of the locomotive, to prevent striking such objects, rear-end collision, and other safety problems. The detection of rail obstacles has primarily employed LiDAR, millimeter-wave radar, optical camera, and other sensors. For instance, Guo et al. ( 7 ) proposed a rail detection method using the geometric features of the rail in accordance with the transverse and longitudinal distance information of the rail scanned by LiDAR, which is capable of decreasing the probability of rail misdetection. Shen et al. ( 8 ) used laser radar method to collect data and considered the laser reflection intensity in the algorithm for detection of rail obstacles, which can increase the speed and accuracy of obstacle detection. Ye et al. adopted a prior target detection module and target detection module to form a target detection network. This network can automatically detect railway obstacles using a visual camera and a millimeter-wave radar continuously. However, targets that have no effect on the running of the rail in the adjacent rail are not removed ( 9 ). Wang et al. identified rail obstacles by integrating cameras and LiDAR data, and estimated their distance after segmentation and identification of rail and locomotive. Their method can effectively detect small obstacles, but it only detects targets in the rail region without considering whether there are obstacles close to the rail ( 10 ). He et al. used LiDAR and optical cameras to build a data acquisition system, improved the YOLOV4 model, and developed a mask segmentation method to increase the accuracy of the model in detecting obstacles. However, that model could easily cut targets during the delineation of the area of interest, resulting in missed target detection and error detection ( 11 ). He et al. proposed the ME-Mask R-CNN model to increase target accuracy. This model can be applied to various rail scenes using long and short focal cameras and LiDAR for data collection. However, this model has a slow detection speed, and it is not suitable for underground rail transit ( 12 ). The complex underground environment of the coal mine creates significant interference to LiDAR. Notably, the irregular sidewall of the mine roadway and numerous pipelines arranged along it significantly affected the construction of the radar point cloud image. Thus, the model cannot meet the requirements of active obstacle detection for underground locomotives. The process of detection of rail obstacles based on LiDAR is complex, and the equipment for detection is expensive. Compared with LiDAR, optical camera is characterized by high imaging speed, high accuracy, and low cost, and it has been extensively applied to systems for the detection of rail transit obstacles. Insufficient illumination conditions and complex lines in coal mines have been considered major problems in conventional rail area detection. With the rapid advance of machine vision and deep learning, these problems can be solved using an intelligent rail transit auxiliary driving system based on computer vision ( 13 – 21 ).

A method of obstacle detection based on instance segmentation is proposed in this study for obstacle detection and ranging of driverless electric locomotives in coal mines. Images are collected by explosion-proof cameras and other optical sensors, and the improved YOLACT++ (You Only Look At CoefficienTs) instance segmentation model is used to increase the accuracy of target detection, perform pixel-level segmentation during target detection, locate the rail mask where the electric locomotive is located, and delimit the effective driving area. According to the overlapping relationship between the target mask and the effective driving area, whether the target is an obstacle or not can be judged, and the distance to the obstacle can be calculated using the geometric relationship. Instructions to sound the horn, decelerate, and/or brake can then be issued for the electric locomotive. The main contributions of this paper are as follows.

As can be seen from the above overview, the conventional methods of track obstacle identification are unable to recognize an obstacle next to the current track, or unable to discern whether the obstacle belongs to the current track. We therefore propose an obstacle recognition method for rail locomotives based on instance segmentation network, which determines whether the target is an obstacle by delineating the effective detection area and performing distance measurement. This obstacle recognition method can improve the guarantee of safety for the use of driverless technology for rail locomotives.

The disadvantages of existing rail locomotive obstacle recognition models are analyzed, and the necessity of applying optical camera to collect images in underground coal mines is discussed. An improved instance segmentation model is proposed to enhance target detection by path enhancement strategy and replacement of target loss function strategy.

Electric Locomotive Rail Identification and Obstacle Detection

YOLACT++ Principle

Bolya et al. ( 22 ) proposed the YOLACT++ model in 2019, which has a higher detection accuracy and a faster detection speed than other instance segmentation networks. Compared with the YOLACT model, YOLACT++ makes the following improvements. (i) The deformable convolution Deformable Convolutional Network (DCN) is introduced into the trunk feature extraction network to increase the target locating accuracy. (ii) The prediction head branch is improved to increase the number of anchors three times. (iii) The branch of mask re-scoring is introduced to re-score each mask to increase the segmentation accuracy of the mask.

The YOLACT++ network architecture process is illustrated in Figure 1. Similar to other instance segmentation models, this model first uses a feature pyramid network (FPN) as backbone network and the characteristics of the pyramid network combining feature extraction. Different from the two-stage instance segmentation model, the model increases its efficiency through two parallel sub-tasks: prototype mask branches and target detection branches. The former is used to generate prototype masks, and the latter is used to predict mask coefficients. Subsequently, the linear combination of mask coefficient and prototype mask filtered by non-maximum suppression (NMS) can achieve the prediction result. Lastly, the mask and confidence of the respective target can be obtained after “crop and threshold” operations.

Electric Locomotive Rail Obstacle Detection Scheme

A roadmap of the obstacle detection technology for underground electric locomotive rail is presented in Figure 2.

First, the video information of the roadway ahead is acquired by the on-board camera of the electric locomotive, and the video is input into the YOLACT++ instance segmentation network model by OpenCV. The model outputs information (e.g., target mask, boundary box, category, and confidence). Subsequently, the effective driving boundary is delimited by locating the mask at the current rail of the electric locomotive, and the current rail mask is expanded to form an effective detection area. Whether the target is an obstacle is determined by whether the target mask overlaps with the mask of the effective detection area. Lastly, the target mask is displayed, and the distance of obstacles is calculated based on the geometric relationship, and the electric locomotive is given the action of sounding the horn, decelerating, braking, and other actions.

Effective Detection Area

Different from other means of transportation, electric locomotive only runs on a predetermined rail, and the on-board camera collects more image information, so it only needs to judge whether there are obstacles in the area inside and beside the rail (i.e., the effective detection area) ( 23 – 25 ).

Locating the Current Rail Mask

The on-board camera of electric locomotive should be calibrated during installation, so the image collected by the camera contains the information of the locomotive’s running area and its surrounding environment. Multiple rail masks are obtained by the instance segmentation network, and the mask corresponding to the current locomotive rail is in the middle region of the image. The rail mask perspective is transformed into an aerial view, the Sobel operator is employed to detect the edge and extract the rail lines, and the cubic polynomial is used to fit the rail lines to obtain the rail mask with smooth edges.

The traditional orbit positioning schematic is shown in Figure 3, Figure 3a shows the original image; the original image is obtained by Sobel edge detection operator in Figure 3b. The edge detection image is obtained by perspective transformation in Figure 3c. The histogram is used to count the number of points in each column in Figure 3c, and the column with the most points is taken as the initial sliding window position. The horizontal coordinates of each point in the sliding window are calculated, and the average value is counted as the next sliding window center position. The center position of the next sliding window is calculated, and all sliding windows corresponding to the orbit line are obtained by cycling through this step, as in Figure 3d. The sliding windows corresponding to the left and right orbit lines are extracted, and the pixel positions of the left and right orbit lines are obtained, and the orbit line function is fitted by fitting the pixel points with a cubic polynomial, as in Figure 3e. The orbit area is filled according to the orbit line fitting function, as in Figure 3f. The orbit area is transformed by the inverse perspective to obtain Figure 3g. We fuse Figure 3g with the original image to obtain the positioned orbit and display the image of the orbit region, as in Figure 3h.

For the track with turnouts, the schematic diagram of traditional track positioning is shown in Figure 4. As can be seen from Figure 4c, when the traditional track positioning method identifies track with turnouts, the sliding window positioning is not able to distinguish the turnouts, resulting in incorrect track line fitting and tendency to false detection, and so forth, further causing inaccurate track line detection. As shown in Figure 4f1, the traditional track line positioning method locates the left-turn track as the straight track, and in Figure 4f2, the traditional track line positioning method locates the right-turn track as the straight track as well, which further indicates that the traditional track positioning method is not applicable to the track with turnouts.

Several track masks are obtained by the example segmentation network, and the mask corresponding to the track where the locomotive is located should be located in the middle region of the image; its positioning schematic diagram is shown in Figure 5. In this paper, the track positioning box is set at the bottom of the image, and the track mask corresponding to the track mask where the locomotive is located is further determined by judging whether the center bottom point of the track mask is located in the track positioning box. It can be seen from Figure 5d1 that the corresponding track is the track where the locomotive is located.

The fitting schematic of track mask positioning is shown in Figure 6. Figure 6a shows the original image. The original image obtains the track mask by the example segmentation network model, and the corresponding mask of the track where the locomotive is located is obtained by track mask positioning, as in Figure 6b. The track mask image is obtained after perspective transformation, as in Figure 6c. The edge image of the track mask is obtained by using the Sobel edge detection operator, as in Figure 6d. We use a histogram to count the number of points in each column shown in Figure 6d. Take the column with the most points as the initial sliding window position, calculate the horizontal coordinates of each point in the sliding window, and count its average value as the next sliding window center position. Loop this step to obtain all sliding windows corresponding to the track line. Extract the sliding windows corresponding to the left and right track lines, obtain the pixel positions of the left and right track lines, and after the third polynomial fitting of the pixel points, we obtain the fitted track line function, as in Figure 6e. The track area is filled according to the track line fitting function, as in Figure 6f. The track area is transformed by the inverse perspective, as in Figure 6g. The image of the track area is positioned and displayed by fusing Figure 6g with the original image, as in Figure 6h.

For track with turnouts, the track localization fitting schematic proposed in this paper is shown in Figure 7. From Figure 7, b1 and b2 , it can be seen that the track mask localization method can determine the track mask where the locomotive is currently located directly on the track mask obtained from the instance partition network model, which further enhances the reliability of track mask localization. In Figure 7, a1 and a2 , the track where the locomotive is located corresponds to the left-turn and right-turn regions respectively. In Figure 7, b1 and b2 , the current locomotive mask position is judged and perspective transformation is performed, so that the current locomotive is in the left-turn and right-turn tracks. After edge detection, track line extraction, fitting track line, and fitting track area filling, a smooth and flat track mask can be obtained, as in Figure 7, e1 and e2 . From Figure 7, f1, f2 , a1 , and a2 , it can be seen that the locomotive driving track is consistent with the corresponding track area in the original image after the track mask positioning fitting in this paper, which further shows that the method proposed in this paper is more reliable than the traditional track positioning method when there is a turnout track.

Delineation of Effective Travel Boundaries and Expansion of Track Masks

Obstacles affecting the normal running of the locomotive exist not only on the inner side of the track, but in some areas on both sides of the track, where obstacles can also bring adverse factors to the safe operation of the locomotive. To detect whether there are obstacles on both sides of the track, this paper delineates the effective driving boundary of the locomotive with one-quarter track width on both sides of the track mask generated by the fitting, and when the target is located within the effective driving boundary, it is also recognized as an obstacle to the normal driving of the locomotive.

This paper delineates the effective driving boundary schematic diagram as shown in Figure 8. Figure 8, a1, b1 , and c1 shows the table track positioning fitting mask combined with the original image. Figure 8, a2, b2 , and c2 , shows the table track positioning fitting mask. Figure 8, a3, b3 , and c3 , shows the table track positioning fitting mask respectively to both sides for expansion. Figure 8, a4, b4 , and c4 , shows the table expanded delineated effective driving boundary mask. Figure 8, a5, b5 , and c5 , The effective driving boundary mask is combined with the original image, and the effective driving boundary mask is the effective driving area mentioned in this paper.

Obstacle Detection and Ranging

There are various targets in the image of electric locomotive, and only the targets located in its effective detection area will hinder the normal running of the locomotive. Figure 9 is a schematic diagram of the detection and ranging of obstacles. When the target is located in the effective detection area, the system determines it as an obstacle. Based on the pixel value of the rail interval at the X-axis direction where the obstacle is located, the distance between the obstacle and the locomotive can be calculated using geometric relations. The specific calculation formula is expressed in Equation 1,

D = \frac{L}{l} \cdot d

(1)

where L denotes the actual rail width; l represents the pixel value of the rail interval at the X-axis direction of the obstacle; d is the pixel value in the image between the obstacle and the locomotive; D is the actual distance between the obstacle and the electric locomotive.

The system issues different instructions to the electric locomotives according to the types and distances of obstacles, as listed in Table 1.

Table 1.

Relationship of Obstacle Category Distance and Instruction

Obstacle	Instructions at different distances
Obstacle	>10 m	5–10 m	<5 m
Person	Sound horn	Sound horn and decelerate	Brake
Electric locomotive	Decelerate	Decelerate	Brake
Stone	Decelerate	Decelerate	Brake

Improve YOLACT++

Path Augmentation Policy

YOLACT++ feature extraction is achieved through the combination of trunk network (ResNet) and FPN. The trunk network generates feature maps of different dimensions and resolutions from bottom to top. Moreover, the feature pyramid grid changes the resolution from top to bottom, and it performs feature fusion with the feature maps generated by the trunk network ( 26 , 27 ). In detecting images of coal mine electric locomotives operating in multi-condition scenarios, the underlying features of the image include target details (e.g., edge shape), which more significantly facilitates target location. High-level features of the image consist of semantic information (e.g., category features), which are more conducive to object classification. As shown in Figure 1, when the underlying feature information of the image is transferred to P7 through the trunk network, the long path increases the difficulty of location information transmission, and multi-layer convolutional pooling easily results in the loss of the underlying detail feature information of the image.

To ensure accurate transmission of image information and to take full advantage of low-level detail feature information, this study introduces bottom-up path augmentation (BPA) in PANET to enhance the feature pyramid structure and decrease the transmission distance of low-level feature information ( 28 ). To ensure the detection speed, P6 and P7 at the input ends of the feature pyramid are cancelled in this study. The improved YOLACT++ network architecture diagram is illustrated in Figure 10. The [P3, P4, P5] feature layers of different scales are the input end of path augmentation, and the [N3, N4, N5, N6, N7] feature layers are obtained after path augmentation. [N3, N4, N5, N6, N7] is adopted to generate the prediction mask coefficient, where N3 is the input of the prototype mask branch to generate different prototype masks.

Figure 11 shows the connection structure of FPN and BPA. P3 feature layer is directly copied to N3 feature layer without any processing. The feature layer of N3 has the convolution operation with a convolution kernel size of 3 × 3 and a step size of two, and the size of the feature map is halved. Subsequently, the feature layer is added to the feature layer of P4. Next, the convolution operation with convolution kernel size of 3 × 3 and step size of one is achieved for the feature layer of N4, and Relu activation function is added after each convolution operation. N5 feature layer can be obtained by the same steps, and N6 and N7 feature layers can be obtained by up-sampling N5 feature layer twice.

Focal Loss Function

Affected by the imbalance of data sample complexity in the process of image acquisition and data expansion, the imbalance of data sample complexity in the single-stage instance segmentation model is more prominent than that in two-stage instance segmentation model. YOLACT++, a single-stage instance segmentation model, adopts cross entropy loss function for target classification, targets of different categories are treated equally, and the training degree of difficult samples is relatively low. The calculation formula of cross entropy is expressed in Equation 2.

CE (p_{t}) = - \log (p_{t})

(2)

Among

p_{t} = {\begin{matrix} p \\ 1 - p \end{matrix} \begin{matrix} if y = 1 \\ otherwise \end{matrix}

(3)

where p denotes sample prediction probability and y is the actual label.

The operation image of multi-condition scenarios of the coal mine electric locomotive contains multiple types of targets, some of which are simple and easy to distinguish, while others are complex and difficult to distinguish. Compared with the background information, part of the target information accounts for a small proportion, resulting in the loss value calculated by the classification loss function containing most of the negative sample background loss. To balance the effect of unbalanced data samples on model loss, the focal loss (FL) function is introduced in this study to replace the original classification loss function, reduce the loss weight of simple samples, reduce the loss proportion of negative samples, and control the model learning ability caused by sample difference ( 29 ).

The FL function is shown in Equation 4,

FL (p_{t}) = - α_{t} (1 - p_{t})^{γ} \log (p_{t})

(4)

Among

α_{t} = {\begin{matrix} α \\ 1 - α \end{matrix} \begin{matrix} if y = 1 \\ otherwise \end{matrix}

(5)

where α_t is the balance factor, which is used to control the weight of positive and negative samples in the loss. (1-p_t)^γ is the modulation factor, which is used to control the weight of difficult samples in the loss. In this study, α = 0.25 and γ = 2 are taken as equilibrium parameters.

The formula for calculation of the YOLACT++ loss function is expressed in Equation 6,

L_{T} = L_{B} + L_{C} + L_{M} + L_{S} + L_{I}

(6)

where L_T is the total loss, L_B is the locating loss of boundary frame, L_C is classified loss, L_M is mask loss, L_S is semantic segmentation loss, and L_I is the mask intersection ratio loss.

Experiment and Result Analysis

Experimental Environment

The hardware environment of the model experiment is listed in Table 2. The main software environment is Python3.8, PyTorch1.8, and Torchvision0.9. The experimental category is set to 5 (including locomotive, pathway, person, stone, and background). The initial learning rate is set to 0.0005, the learning rate attenuation coefficient is set to 0.1, the momentum is set to 0.9, and the weight attenuation coefficient is set to 0.005.

Table 2.

Experiment Hardware

Hardware	Information
System	Ubuntu18.04.5
CPU	Intel Core i7-8700 @3.02GHz 6 Nuclear
GPU	Nvidia GeForce GTX1080(8G)
Memory	16GB (DDR3 1600MHz)

Note: CPU = Central Processing Unit; GPU = Graphics Processing Unit.

Data Acquisition and Pretreatment

The image dataset in this study consists of multiple operation scenes of a coal mine electric locomotive collected by explosion-proof cameras and other equipment, with a total of 360 images with a size of 1,080 × 1,920. Some images of the dataset are presented in Figure 12. To ensure the reliability of the dataset images and enhance the accuracy of network training, the data samples are expanded by changing the brightness and chroma. After the expansion, there are 1,600 images in the dataset, and 1,120 images in the training set and 480 images in the verification set are obtained by dividing the training set and verification set at a ratio of 7:3. The VIA image annotation tool is used to annotate the instance targets in the dataset, create annotation areas to obtain corresponding JSON files, and convert the JSON files into COCO dataset format.

Evaluation Indicators

To comprehensively evaluate the target detection and segmentation performance of the improved YOLACT++ model and verify the feasibility of the model, average precision (AP) and mean AP (mAP) of boundary boxes and masks were adopted in this study and frames per second (FPS) were used as evaluation indicators. In this study, the target categories consist of electric locomotive, rail, signal lamp, pedestrian, and stone. AP is the average precision of single-class target prediction, equal to the precision–recall curve integral, that is, the integration of the precision–recall curve. Precision is the proportion of a certain type of target correctly identified among the number of positive samples classified by the model. Recall is the proportion of the correct samples of a certain target in the model classification to all positive samples. mAP is equal to the average of all categories of AP. The calculation formula for precision, recall, AP, and mAP is expressed below ( 30 , 31 ):

\Pr ecision = \frac{TP}{TP + FP}

(7)

Re call = \frac{TP}{TP + FN}

(8)

AP = \int_{0}^{1} P (R) dR

(9)

mAP = \frac{AP}{n}

(10)

where TP denotes the number of positive samples correctly identified as targets; FP represents the number of negative samples incorrectly identified as targets; FN expresses the number of samples whose target is not correctly identified. The mAP of target boundary box AP_box takes the intersection ratio Intersection over Union (IOU) between target prediction boundary box and target real boundary box as the judgment threshold, which is determined as positive and negative samples. The mAP of target mask AP_mask takes the IOU ratio of target prediction mask and target real mask as the judgment threshold, which is determined as positive and negative samples.

Verifying Model Performance

To verify the feasibility of the improved YOLACT++ model, this study trained the expanded dataset in the experimental environment of Experiment and Result Analysis. The maximum iteration step max_iter was set as 25,000, and the learning rate interval was set as (12,000, 18,000, 22,000). The number of samples captured in one training session was set to a batch size of four. The loss curve of the improved YOLACT++ model is presented in Figure 13. As depicted in Figure 13, the model loss gradually converges and tends to be stable after 50,000 iterations. The comparative analysis results of the YOLACT++ model before and after improvement are listed in Table 3.

Table 3.

Results of Model Comparison Before and After YOLACT++ Improvement

Model	AP_0.5 (box/mask, %)				mAP_0.5 (box/mask, %)	mAP_all (box/mask, %)	Frames per second
Model	Electric locomotive	Pathway	Person	Stone	mAP_0.5 (box/mask, %)	mAP_all (box/mask, %)	Frames per second
YOLACT++	98.92/98.96	96.89/96.87	98.02/98.02	97.20/97.20	97.76/97.76	85.91/81.79	22.2
YOLACT ++ (BPA)	99.84/99.56	95.93/96.91	98.00/98.01	97.41/97.08	97.80/97.99	85.28/82.48	21.9
YOLACT ++ (FL)	99.95/99.96	97.75/97.85	98.02/98.02	97.38/97.34	98.27/98.29	87.33/82.72	22.2
YOLACT ++ (BPA +FL)	99.95/99.97	97.81/97.89	99.00/99.00	97.32/97.34	98.52/98.55	87.56/82.42	21.9

Note: YOLACT++ = You Only Look At CoefficienTs; AP = average precision; mAP = mean average precision; BPA = bottom-up path augmentation; FL = focal loss.

In Table 3, YOLACT++ represents the original model; YOLACT++ (BPA) represents the YOLACT++ model that only uses the BPA strategy; YOLACT++ (FL) represents the YOLACT++ model using only the FL strategy; YOLACT++ (BPA+FL) denotes a YOLACT++ model that uses both BPA and FL strategies. As depicted in Table 3, the AP of boundary box AP_box and mask AP_mask of the model are improved compared with before improvement after using the BPA and FL strategy. Compared with the model before the improvement, at the IOU threshold value of 0.5, the mAP of boundary box mAP_0.5(box) increases by 0.76% to 98.52%, and the mAP of mask mAP_0.5(mask) increases by 0.79% to 98.55%. Compared with the model before improvement, at the full threshold of IOU, the mAP of boundary box mAP_all(box) increases by 1.65%, reaching 87.56%, and the mAP of mask mAP_all(mask) increases by 0.63%, reaching 82.42%. The detection speed of the improved model is only 0.3 FPS less than before the improvement, reaching 21.9 FPS, but it is also very close to the real-time target detection and segmentation speed requirement.

The model detection results before and after improvement are shown in Figure 14.

As depicted in Figure 14b1, the YOLACT++ model fails to detect the rock targets. As depicted in Figure 14, b2 and c2 , the YOLACT++ and YOLACT++(BPA) models fail to detect the right short-rail targets. As depicted in Figure 14, b3 and c3 , the YOLACT++ and YOLACT++(BPA) models fail to detect pedestrian targets. As depicted in Figure 14, b4 to d4 , the YOLACT++, YOLACT++(BPA), and YOLACT++ (FL) models fail to detect pedestrian targets on the left. As shown in Figure 14, e1 to e4 , and Table 2, the YOLACT++(BPA+FL) model can effectively detect downhole rocks, short rails, pedestrians, and other targets, and increase the mean accuracy of boundary box and mask of the YOLACT++ model. Therefore, the YOLACT++ (BPA+FL) model serves as an instance segmentation network model in underground electric locomotive rail identification and obstacle detection.

The YOLACT++(BPA+FL) model is applied to the underground electric locomotive rail obstacle detection technology method, and the visual results of obstacle detection and ranging are presented in Figure 15.

As depicted in Figure 15, the target (i.e., obstacle) located in the effective detection area can be displayed, and its distance can be calculated and displayed on the image by Equation 1. As depicted in Figure 15, d1, d2 , and d3 , the distance to the detected obstacle is less than 5 m. Thus, the control terminal should issue the instruction to the electric locomotive to brake, and the electric locomotive should perform a braking action. As depicted in Figure 15d4, the detected obstacle distance is greater than 5 m and less than 10 m. Accordingly, the control terminal should issue instructions to the electric locomotive to sound the horn and decelerate, and the electric locomotive performs those actions.

Discussion and Conclusion

An obstacle detection method based on instance segmentation is proposed in this study, which can quickly identify a target obstacle and calculate its distance from the approaching electric locomotive. The system issues different instructions to the electric locomotive in accordance with the different types and distances of obstacles. The proposed method can provide a technical guarantee for the use of driverless underground electric locomotives, which mainly consists of two parts.

The first part is proposing the effective driving area by locating the rail mask, demarcating the effective driving boundary, expanding the rail mask, and finally establishing the effective driving area. According to whether the target mask and effective driving area overlap, whether the target belongs to the obstacle can be determined, and the obstacle distance can be further calculated. The demarcation of effective driving area can detect obstacles on the rails and provide a method of detecting obstacles beside the rails. Different from the conventional rail line identification, the rail mask based on instance segmentation adopted in this study can distinguish rail turnouts more easily and increase the reliability of effective driving area demarcation.

The second part is improving the instance segmentation model. Path augmentation strategy and target classification loss function replacement strategy are adopted to decrease the transmission distance of the underlying feature information, effectively control the model learning ability caused by the sample difference, further enhance the ability to detect model target details, and increase the target segmentation accuracy. The average accuracy of the bounding box reaches 98.52% and the average accuracy of the mask reaches 98.55% when the improved YOLACT++ model is at the IOU threshold of 0.5. The improved YOLACT++ model achieves an average accuracy of 87.56% for the bounding box and 82.42% for the average accuracy mask at the full IOU threshold. The detection speed of the improved model reaches 21.9 FPS, close to the requirement of real-time target detection and segmentation speed.

Discussion of these results shows that the improved YOLACT++ algorithm can be used to detect the target detection front-end in the obstacle identification algorithm to provide a reliable mask for the delineation of the effective obstacle detection area. The obstacle recognition method proposed in this paper can provide the technology for driverless motor vehicles. A limitation of this research is that in actual working conditions, when collecting the dataset of the locomotive driving in multiple mine workings, other harsh environments such as motion blur, humidity, darkness, dust interference, and so forth that affect the image acquisition should be considered, and other image pre-processing algorithms can be studied later to realize multi-target detection and obstacle recognition in the harsh environment of an underground coal mine by combining the detection model with high detection accuracy and speed.

Footnotes

Author Contributions

The authors confirm contribution to the paper as follows: study conception and design: J. Tong, S. Wang; data collection: W. Wang, S. Zong; analysis and interpretation of results: S. Wang, Y. Guo; draft manuscript preparation: J. Tong, T. Yang. All authors reviewed the results and approved the final version of the manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work of Shuang Wang was supported by the National Natural Science Foundation of China under Grant No. 51904007, Anhui Provincial University System Innovation Project under Grant No. GXXT-2020-60, and Open Fund of State Key Laboratory of Mining Response and Disaster Prevention and Control in Deep Coal Mine under Grant No. SKLMRDPC20KF10. The work of Yongcun Guo was supported by the Major Science and Technology Project of Anhui Province under Grant No. 202003a05020021.

ORCID iD

Jiale Tong

References

Wang

Yuan

Sun

Liu

Carbon Emissions of Coal Supply Chain: An Innovative Perspective from Physical to Economic. Journal of Cleaner Production, Vol. 295, 2021, p. 126377.

Wang

Ren

Intelligent and Ecological Coal Mining as well as Clean Utilization Technology in China: Review and Prospects. International Journal of Mining Science and Technology, Vol. 29, No. 2, 2019, pp. 161–169.

Jia

Lin

How to Achieve the First Step of the Carbon-Neutrality 2060 Target in China: The Coal Substitution Perspective. Energy, Vol. 233, 2021, p. 121179.

Han

Wei

Cheng

Driverless Technology of Underground Locomotive in Coal Mine. Journal of China Coal Society, Vol. 45, No. 6, 2020, pp. 2104–2115.

Wang

Guo

Zhao

Obstacle Detection Method of Unmanned Electric Locomotive in Coal Mine Based on YOLOv3-4L. Journal of Electronic Imaging, Vol. 31, No. 2, 2022, p. 023032.

Jiang

Yang

Zhang

Recent Progress on Smart Mining in China: Unmanned Electric Locomotive. Advances in Mechanical Engineering, Vol. 9, No. 3, 2017, pp. 1–10.

Guo

Cai

Jiang

A Railway Rail Detection Method Using LiDAR. Journal of Central South University (Science and Technology), Vol. 51, No. 2, 2020, pp. 560−566.

Shen

Qian

Xie

Yuan

Zeng

Zhang

Obstacle Detection Algorithm of Fully Automatic Train Considering Reflection Intensity. Journal of Tongji University (Natural Science), Vol. 50, No. 1, 2022, pp. 6–12.

Zhang

Liu

Railway Traffic Object Detection Using Differential Feature Fusion Convolution Neural Network. IEEE Transactions on Intelligent Transportation Systems, Vol. 22, No. 3, 2021, pp. 1375–1387.

10.

Wang

A Camera and LiDAR Data Fusion Method for Railway Object Detection. IEEE Sensors Journal, Vol. 21, No. 12, 2021, pp. 13442–13454.

11.

Zhou

Chen

Liu

Yao

Shan

Obstacle Detection of Rail Transit Based on Deep Learning. Measurement, Vol. 176, 2021, p. 109241.

12.

Qiu

Miao

Zou

Ren

Shen

Improved Mask R-CNN for Obstacle Detection of Rail Transit. Measurement, Vol. 190, 2022, p. 110728.

13.

Chen

Wang

Yang

Yao

Liu

Wang

Deep Learning for the Detection and Recognition of Rail Defects in Ultrasound B-Scan Images. Transportation Research Record: Journal of the Transportation Research Board, 2021. 2675: 888–901.

14.

Zhang

Zhao

Zhou

Foreign Body Detection in Rail Transit Based on a Multi-Mode Feature-Enhanced Convolutional Neural Network. IEEE Transactions on Intelligent Transportation Systems, Vol. 23, No. 10, 2022, pp. 18051–18063.

15.

Cao

Qin

Xie

Liu

Zhang

An Effective Railway Intrusion Detection Method Using Dynamic Intrusion Region and Lightweight Neural Network. Measurement, Vol. 191, 2022, p. 110564.

16.

Caliwag

Han

Park

Lim

Deep-Learning-Based Fault Occurrence Prediction of Public Trains in South Korea. Transportation Research Record: Journal of the Transportation Research Board, 2022. 2676: 710–718.

17.

Guan

Jia

Xie

Yin

A Lightweight Framework for Obstacle Detection in the Railway Image Based on Fast Region Proposal and Improved YOLO-Tiny Network. IEEE Transactions on Instrumentation and Measurement, Vol. 71, 2022, p. 5009116.

18.

Zhao

Wang

Zhou

Gao

A Stable Lightweight and Adaptive Feature Enhanced Convolution Neural Network for Efficient Railway Transit Object Detection. IEEE Transactions on Intelligent Transportation Systems, Vol. 23, No. 10, 2022, pp. 17952–17965.

19.

Chen

Zhu

Ali

Cao

A Survey on Theories and Applications for Self-Driving Cars Based on Deep Learning Methods. Applied Sciences, Vol. 10, 2020, p. 2749.

20.

Ren

Zhang

Zhai

Wang

Application of Lightweight Railway Transit Object Detector. IEEE Transactions on Industrial Electronics, Vol. 68, No. 10, 2021, pp. 10269–10280.

21.

Wang

Chen

Zhou

Yang

FarNet: An Attention-Aggregation Network for Long-Range Rail Track Point Cloud Segmentation. IEEE Transactions on Intelligent Transportation Systems, Vol. 23, No. 8, 2021, pp. 13118–13126.

22.

Bolya

Zhou

Xiao

Lee

YOLACT++ Better Real-Time Instance Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 44, 2022, pp. 1108–1121.

23.

Ding

Zhang

Xiao

Shu

A Lane Detection Method Based on Semantic Segmentation. Computer Modeling in Engineering & Sciences, Vol. 122, 2020, pp. 1039–1053.

24.

Gong

Chen

Zhong

Chen

Enhanced Few-Shot Learning for Intrusion Detection in Railway Video Surveillance. IEEE Transactions on Intelligent Transportation Systems, Vol. 23, No. 8, 2021, pp. 11301–11313.

25.

Yang

Wang

Zhou

Chen

Zhang

A Topology Guided Method for Rail-Track Detection. IEEE Transactions on Vehicular Technology, Vol. 71, 2022, pp. 1426–1438.

26.

Zhang

Ren

Sun

Deep Residual Learning for Image Recognition. Proc., IEEE Conference on Computer Vision and Pattern Recognition, Computer Vision Foundation. IEEE Xplore, 2016, pp. 770–778.

27.

Gkioxari

Dollar

Girshick

Mask R-CNN. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 42, 2020, pp. 386–397.

28.

Liu

Qin

Shi

Jia

Path Aggregation Network for Instance Segmentation. Proc., IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018.

29.

Lin

Goyal

Girshick

Dollar

Focal Loss for Dense Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 42, 2018, pp. 318–327.

30.

Guo

Qian

Leng

Automatic Railroad Track Components Inspection Using Real-Time Instance Segmentation. Computer-Aided Civil and Infrastructure Engineering, Vol. 36, 2021, pp. 362–337.

31.

Zhong

Liu

Wang

Liu

Yang

Han

Nunez

A Looseness Detection Method for Railway Catenary Fasteners Based on Reinforcement Learning Refined Localization. IEEE Transactions on Instrumentation and Measurement, Vol. 70, 2021, p. 3518913.