Passenger Flow Detection in Subway Stations Based on Improved You Only Look Once Algorithm

Abstract

Passenger flow detection plays an important role in guaranteeing passenger safety. It contributes to the efficiency of passenger flow control at stations. However, passenger flow detection in subway stations has the problems of target size inconsistency and poor detection effects of small targets. To solve this problem, an improved You Only Look Once algorithm (Improved-YOLO) is proposed based on the YOLOv4 for passenger flow detection in subway stations. The repeatable bidirectional feature fusion (BiFF) module was designed to combine with the adaptively spatial feature fusion (ASFF) module to replace the feature fusion network of the YOLOv4. To verify the effectiveness of the Improved-YOLO, the passenger flow dataset of Nanning Metro Line 1 was used for the experiment. Augmentation and transfer learning were then used to improve the performance of the model. Compared with the YOLOv4, the results showed that the mean average accuracy (mAP) of the Improved-YOLO increased from 89.63% to 92.96%, and the detection time of a single frame image reduced from 61 ms to 52 ms. Compared with other classical models, the Improved-YOLO showed satisfactory performance in passenger flow detection in subway stations. These experimental results can provide reference and theoretical support for real-time passenger flow detection and passenger behavior recognition in subways.

Keywords

data and data science artificial intelligence and advanced computing applications machine vision public transportation rail transit systems subway rail passenger rail transportation passengers

With the development of the world’s economy and science, the construction of urban rail transit has also increased year by year ( 1 – 3 ). The subway has become the preferred mode of transportation for people traveling in cities. Therefore, subway traffic management is crucial. With the increasing number of passengers, subway station passenger flow control has faced great challenges in recent years ( 4 ). Passenger flow detection is an important part of an intelligent traffic system. It is used to solve this problem and ensure the normal operation of subway stations. Nowadays, the image-based passenger flow detection method is the most commonly used detection method in subway stations ( 5 ). It relies on surveillance cameras to collect, process, and analyze passenger flow information. Since this method has good accuracy and timeliness, it is favored by most researchers. Image-based passenger flow object detection methods are divided into two types. One is the traditional image detection method based on the optical flow method ( 6 ) and the background subtraction method ( 7 ). The other is the object detection method based on deep learning. Sengar et al. ( 8 ) detected moving objects by detecting the histogram threshold of the optical flow field and the optical flow field of the candidate flow value. Chen et al. ( 9 ) proposed a new method for crowd motion analysis of video sequences based on integrated optical flow. In addition, pixel motion intensity, pixel number, and motion direction were used to describe the crowd. Costache et al. ( 10 ) used background subtraction to extract moving objects, and support vector machine was applied for classification. However, the traditional image detection methods are easily affected by the environment. The detection accuracy will be seriously reduced when the pedestrian flow is large.

Since 2014, He et al. ( 11 , 12 ) have been working on the application of convolutional neural networks (CNN) for object detection. Under conditions of illumination and deformation, CNN has the advantage of keeping the characteristics unchanged. Besides, it can not only generate feature models during training but also have good applicability and flexibility. Therefore, traditional image detection methods are being gradually replaced by deep learning methods. Object detection algorithms based on deep learning are mainly divided into two categories. One is the two-stage detection algorithm based on candidate regions. The region-based convolutional neural network (R-CNN) ( 13 ) is the most representative two-stage detection algorithm. Subsequently, the Fast R-CNN ( 14 ), Faster R-CNN ( 15 ), and Mask R-CNN ( 16 ) object detection algorithms were improved by the researchers. Another is the end-to-end detection algorithm with a single stage, for example, the single-shot multibox detector (SSD) ( 17 ) and You Only Look Once (YOLO) ( 18 – 21 ). They generate anchor boxes directly without generating candidate regions.

With the development of deep learning, surveillance video is widely used for tasks such as crowd analysis ( 22 , 23 ), face detection ( 24 , 25 ), industrial robots ( 26 ), detection of objects ( 27 – 29 ), rail defect detection ( 30 ), and detection of brain tumors ( 31 , 32 ). In recent years, deep learning models have been applied by researchers to detection of pedestrian objects. Li et al. ( 33 ) used the Faster R-CNN network to detect the human head and shoulders, and the experiment was able to achieve better results. Kajabad et al. ( 34 ) applied the YOLOv3 network to detect people in a museum, which had a fast detection speed. However, when multiple objects appear in the same grid cell, its detection accuracy will be greatly reduced. Dagli et al. ( 35 ) used the SSD model to detect people in a specified area and accordingly detect the number of people in a queue. Meanwhile, they combined the SSD model with the OpenVINO toolkit to deploy a smart queuing system. Ahmed et al. ( 36 ) adopted the SSD network to detect people from the top view to avoid occlusion between people. This is the first time a deep learning method has been used to detect people on an overlooking map. Similarly, Ahmed et al. ( 37 ) used the YOLOv3 network to detect people from the top view. They used transfer learning to improve the overall performance of the model, and the Deep Simple Online and Realtime Tracking (Deep SORT) was combined to track people. As mentioned in the above discussion, most of the existing studies concentrated on pedestrian detection from the top view. However, the cameras in subway stations are deployed laterally, which has the problem of occlusion and target size inconsistency. Therefore, pedestrian detection still has the potential for improvement when used in subway scenes.

The large amount of calculation, slow detection speed, and high hardware requirements of the two-stage detection algorithm mean that it is not conducive to the actual deployment. Therefore, a one-stage detection algorithm is selected for passenger flow detection.

A one-stage detection algorithm was proposed based on YOLOv4. We named it the Improved-YOLO. The Improved-YOLO has the characteristics of few parameters and high accuracy. The algorithm is mainly composed of a feature extraction network and a feature fusion network. The feature extraction network of the Improved-YOLO adopts the CSPDarknet53. The repeatable BiFF module was designed for the feature fusion network of Improved-YOLO. It can not only fuse more features with less computation, but also strengthen the connection between the front and back layers. The ASFF module was used after the BiFF module. It enables the network to spatially filter the features of other layers and retain effective information for feature fusion.

To summarize, our major contributions are as follows:

A passenger flow object detection method based on YOLOv4 was proposed. To improve the detection performance of the YOLOv4 and reduce the training parameters and training time, we designed a new feature fusion network while maintaining accuracy.

We used the passenger flow images of Nanning Metro Line 1 to create a dataset in PASCAL VOC format, and data augmentation methods were used to improve the generalization ability of the dataset. The passenger flow images come from different scenes and have different passenger flow intensities.

To verify the performance of Improved-YOLO, the experiment was carried out with image data of 416 × 416 pixels. Experiments showed that the mAP of Improved-YOLO reaches 92.96%, and the Improved-YOLO was stronger than other classical object detection networks.

The sections of this paper are arranged as follows. The YOLO algorithm is introduced, and the structural improvements of the YOLOv4 network are detailed in the next section. The relevant experimental methods are introduced in the third section. The results of experiments and discussions are shown to demonstrate the effectiveness of the proposed method in the fourth section. The conclusion is given, and the prospects for future research are proposed in the final section.

Proposed Method

YOLO

The YOLO model is a one-stage detection algorithm, which regresses the position and category of the boundary box in the output layer. Since it transforms the object detection problem into a regression problem, the detection speed is accelerated. The image is divided into S × S grids in the YOLO model, and the object is detected at the center point within each grid. Each grid will predict B bounding boxes and the confidence of the bounding box. The size and position of the bounding box can be characterized by four values: (x, y, w, h), where (x, y) denotes the center coordinate of the bounding box, w denotes the width of the bounding box, and h denotes the height of the bounding box. The Confidence is defined as follows:

Confidence = P_{r} (Object) \times {IOU}_{pred}^{truth}

(1)

If there is no object in the bounding box, P_r (Object) should be zero. Otherwise, P_r (Object) should be one. ${IOU}_{pred}^{truth}$ denotes the intersection over union between the prediction box and the real box.

Feature Extraction Network of Improved-YOLO

CSPDarknet53 ( 21 ) is used as the feature extraction network of the Improved-YOLO. It is used to extract feature maps from images. The CSPDarknet53 mainly consists of the Resblock module. The structure of the Resblock module is shown in Figure 1. It consists of convolutional layers and X Resunits concatenated. The Resunit refers to the residual structure in the Resnet that can build deeper networks. Feature extraction is performed with a 3 × 3 convolution kernel with a stride of two before the Resblock module. In the process of calculation, the repetition of gradient information increases the amount of calculation. The feature mapping of the basic layer is divided into two parts by the Resblock module, and then the two parts are concatenated by a cross-stage hierarchy. Therefore, the Resblock module can reduce the amount of computation while ensuring model accuracy.

Feature Fusion Network of Improved-YOLO

The feature pyramid network (FPN) is a multi-scale feature fusion method with a top-down combination. It can accurately combine low-level features with high-level features to obtain multi-layer fusion information ( 38 ). An additional bottom-up pathway is added to the FPN to construct the path aggregation network (PANet) ( 39 ). It improves the utilization rate and the transmission efficiency of low-level features.

The size of passengers as captured by the surveillance camera is inconsistent because they appear at different distances from the camera. When objects of different sizes appear in the image at the same time, the mutual interference between different feature layers reduces the effectiveness of the PANet. To solve this problem, a new feature fusion network is proposed, which takes the BiFF module and the ASFF module as its core. The cross-channel information is captured by the BiFF module, which enables the model to accurately locate the object area.The problem of low accuracy caused by inconsistent object size is solved by the ASFF module. Therefore, the BiFF module is combined with the ASFF module to further improve the accuracy of the model.

The feature fusion network core of Improved-YOLO is shown in Figure 2, where + denotes add. First, the BiFF module is used for feature fusion. Then, three different effective features are obtained. Finally, the ASFF module is used to build the feature fusion network core of Improved-YOLO.

Spatial pyramid pooling (SPP) networks

The pooling layer is usually used to reduce the feature dimension, realize the invariance of image transformation, and increase the receptive field. The SPP ( 11 ) module is a pooling method that performs better than other pooling strategies. It has the characteristic that any size feature map can be transformed into a fixed-size feature vector. It is composed of four maximum pooling layers with kernel sizes of 13 × 13, 9 × 9, 5 × 5, 1 × 1 as well as the concatenation. The kernel size of the pooling is close to or equal to the size of the feature map, so the feature map is fused with local and global features, which not only enriches the expressive ability of the feature map but also greatly expands the receptive field. Since the last DownSampling layer of the backbone network has the richest semantic features, the SPP module is set here. In addition, the SPP module is set to retain the feature information of small objects and reduce the loss of feature information.

BiFF networks

Compared with the PANet of YOLOv4, the BiFF module has the characteristics of reusing and improving the efficiency of fusion. The structure comparison of the BiFF module and the PANet module is shown in Figure 3, where c denotes concatenate.

The last three feature layers of the feature extraction network are P3, P4, and P5, respectively, which continue to be fed into the BiFF module as input. Since the number of channels of P3, P4, and P5 is inconsistent, it is necessary to adjust the number of channels. The feature layers are renamed to P3_in, P4_in, and P5_in after being adjusted. First, to avoid frequent changes in dimension, the number of channels of P3_in, P4_in, and P5_in are adjusted to 256 in this work. Second, to fuse more features without adding too much cost, we added skip connections from input nodes to output nodes on the same level. Finally, the max pooling method is used for DownSampling to reduce the parameters of the model. Meanwhile, for the network to learn key features in a certain path, the fast normalized fusion method is used to add additional weight to each path. The output after weight accumulation is defined as follows:

O = \sum_{i} \frac{w_{i}}{ε + \sum_{j} w_{j}} \cdot I_{i}

(2)

where w_i denotes the learnable weight, and w_i ≥ 0 is ensured by applying the Relu function after each w_i. To avoid numerical instability, ε = 0.0001 is a small value.

The BiFF module combines bidirectional cross-scale connection and weighted feature fusion. As a specific example, two fusion features of the BiFF module shown in Figure 3a are described at level 4:

P_{4}^{td} = Conv (\frac{w_{1} \cdot P_{3}^{in} + w_{2} \cdot Upsample (P_{5}^{td})}{w_{1} + w_{2} + ε})

(3)

P_{4}^{out} = Conv (\frac{w_{1}^{'} \cdot P_{4}^{in} + w_{2}^{'} \cdot P_{4}^{td} + w_{3}^{'} \cdot Maxpool (P_{3}^{out})}{w_{1}^{'} + w_{2}^{'} + w_{3}^{'} + ε})

(4)

where $P_{4}^{td}$ is the intermediate feature at level 4 on the top-down pathway, and $P_{4}^{out}$ is the output feature at level 4 on the bottom-up pathway. All other features are constructed in a similar way.

ASFF networks

To make it easier for the network to learn in back propagation, the output feature layer (P3_out, P4_out, and P5_out) of the BiFF module performs adaptive spatial feature fusion. The fusion feature formula is as follows:

y_{ij}^{l} = α_{ij}^{l} \cdot x_{ij}^{5 \to l} + β_{ij}^{l} \cdot x_{ij}^{4 \to l} + γ_{ij}^{l} \cdot x_{ij}^{3 \to l}

(5)

where $x_{ij}^{n \to l}$ denotes the feature vector at position (€, j) on the feature maps adjusted from level n to level l, $y_{ij}^{l}$ denotes the (i, j)-th vector of the output feature maps y^l among channels. $α_{ij}^{l}$ , $β_{ij}^{l}$ , $γ_{ij}^{l}$ are the spatial weights from three different feature layers to the layer l. We force $α_{ij}^{l} + β_{ij}^{l} + γ_{ij}^{l} = 1$ , and $α_{ij}^{l}$ , $β_{ij}^{l}$ , $γ_{ij}^{l}$ belong between zero and one, respectively. Taking the $α_{ij}^{l}$ as an example, the $α_{ij}^{l}$ is defined as follows:

α_{ij}^{l} = \frac{e^{λ_{α_{ij}}^{l}}}{e^{λ_{α_{ij}}^{l}} + e^{λ_{β_{ij}}^{l}} + e^{λ_{γ_{ij}}^{l}}}

(6)

where $α_{ij}^{l}$ , $β_{ij}^{l}$ , and $γ_{ij}^{l}$ are defined by using the softmax function with $λ_{α_{ij}}^{l}$ , $λ_{β_{ij}}^{l}$ , and $λ_{γ_{ij}}^{l}$ as control parameters, respectively. Finally, the 1 × 1 convolution layer is used to calculate the weight scalar maps from $x^{5 \to l}$ , $x^{4 \to l}$ , and $x^{3 \to l}$ , respectively. All other spatial weights are calculated in a similar way. With this method, we will gain the outputs {y⁵, y⁴, y³} that are used for object detection.

Improved-YOLO

The Improved-YOLO network is shown in Figure 4. It consists of a feature extraction network, a feature fusion network, and a prediction layer. The feature extraction network CSPDarknet53 is composed of five Resblock modules. With the feature extraction network, the results of the last three feature layers are fed into the feature fusion network.

To reduce the loss of feature information in the feature fusion network, the SPP module was added after the last feature layers of the feature extraction network. Before the BiFF module performs feature fusion operations, the convolution layer of 1 × 1 is used to adjust the channel number of the three feature layers to 256. After the BiFF module, the outputs of the BiFF module are used as inputs for the ASFF module to perform feature fusion operations. Finally, the 1 × 1 convolution layer is used to change the number of channels output by the ASFF module. So far, feature fusion has been completed, and the results are being passed to the prediction layer.

The size of the output feature map from the feature fusion network is 52 × 52 × 128, 26 × 26 × 256 and 13 × 13 × 512, respectively. The Improved-YOLO used nine bounding boxes with a fixed width and fixed height for prediction. The selection of bounding boxes before detection will directly affect the accuracy of detection. To obtain suitable bounding boxes, the K-means clustering algorithm is used to calculate the bounding boxes suitable for this work. In this work, the nine bounding box sizes obtained are: 7 × 27, 8 × 42, 9 × 49, 12 × 55, 14 × 69, 18 × 81, 21 × 107, 29 × 138, 46 × 183.

Experiment

Training

The training and testing of the model were carried out on the computer in the laboratory. The CPU used was an Inter® Core^TM i3-9100F and the GPU was an NVIDIA GeForce GTX1060 (6GB). The software environment was as follows: Ubuntu 18.04 + Pytorch-1.4.0 + Cuda-10.1 + Cudnn-7.6.3 + Python 3.7.0 + OpenCV-4.0.0 ect. The parameters set in the experiment are shown in Table 1.

Table 1.

Training Parameters of the Improved-YOLO

Stage	Parameter
Stage	Input size	Batch size	Epochs	l₀	T_n	l_min	Decay
First stage	416 × 416	8	25	0.001	5	0.0001	0.0005
Second stage	416 × 416	4	75	0.0005	4	0.00005	0.0005

Note: l₀ = the initial learning rate; l_min = the minimum learning rate; T_n = the half period of the cosine function; YOLO = You Only Look Once.

The input image was adjusted to a resolution of 416 × 416. The whole training process was divided into two stages. The training results of the first stage were used as a prerequisite for the second stage. This process produces a regularization effect to enhance generalization and prevent overfitting ( 40 ). In the first stage, the pre-training parameters of the feature extraction network were frozen in the training process. In the second stage, all parameters were used for training. Since the feature map is generic during freezing training, the training efficiency and convergence speed of the model are improved in the first stage ( 41 ). During training, the cosine annealing algorithm was used to adjust the learning rate of each epoch. The formula is defined as follows:

l_{r} = l_{\min} + \frac{1}{2} (l_{0} - l_{\min}) (1 + \cos (\frac{T_{i}}{T_{n}} π))

(7)

where l₀ denotes the initial learning rate, l_min denotes the minimum learning rate, T_n denotes the half period of the cosine function, and T_i denotes the cycle of execution to the i-th epoch. In the first stage, let l₀ = 0.001, l_min = 0.0001, T_n = 5, and the batch size set is to eight. In the second stage, l₀ = 0.0005, l_min = 0.00005, T_n = 4, and the batch size is set to four. The number of iteration epochs in the first stage and the second stage is 25 and 75, respectively. The Adam gradient descent algorithm was used as the optimizer, and the weight decay is 0.0005.

The loss function of the Improved-YOLO is defined as follows:

Loss = Los s_{CIoU} + Los s_{conf} + Los s_{cls}

(8)

where Loss_cIoU denotes the error of predicting the bounding box, Loss_conf represents the error of confidence error, and Loss_cls is the classification error.

Data Processing

The data for the experiment comes from Nanning Metro Line 1. The monitoring camera of the subway station was used to capture the passenger flow data in real time, and the original image of 1,920 × 1,080 pixels was obtained by clipping the video data. The dataset was produced through techniques of manual screening, data augmentation, and image annotation. Finally, we had data from 6,733 images.

Each image was labeled accordingly using professional software, and the labels were made in the PASCAL VOC format. The distribution ratios for the training set and testing set are 70% and 30%, respectively, and the IOU is set to 0.5. The IOU (Intersection over union) represents the intersection and combination ratio of prediction frame and real frame.

Performance Evaluation

To compare the performance of Improved-YOLO with other classical models, the following evaluation metrics were used:

Precision, recall, and F1 score.

In image accuracy evaluation, the confusion matrix represents the accuracy of the classification results, and it is usually used to compare the classification results with the actual measured values. The results can be divided into four types: true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). The classification results of the confusion matrix are shown in Table 2.

Table 2.

Confusion Matrix for the Inspection Results

		Actual
Confusion matrix		Positive	Negative
Predicted	Positive	True positive	False positive
Predicted	Negative	False negative	True negative

Precision and recall are defined as follows:

\Pr ecision = \frac{TP}{TP + FP}

(9)

Recall = \frac{TP}{TP + FN}

(10)

Drawing the precision-recall (P-R) curve with the recall rate as the horizontal axis and the precision as the vertical axis. The P-R curve is the best indicator for evaluating the object detection algorithm. The F1 score is the harmonic mean of precision and recall. It is defined as follows:

F 1 = \frac{2 \times P \times R}{P + R}

(11)

where P denotes precision and R denotes recall.

2. Mean average precision (mAP).

The average precision (AP) is the area under the P-R curve, and it is calculated using the 11-point interpolated average precision calculation method. The precision is averaged over 11 spaced recall levels [0, 0.1,…, 1]. AP is defined as follows:

AP = \frac{1}{11} \sum_{r \in {0, 0.1, . . ., 1.0}} ρ_{interp} (r)

(12)

ρ_{interp} (r) = max_{\tilde{r} : \tilde{r} \geq r} p (\tilde{r})

(13)

AP denotes the performance of the detection model in detecting a certain class, while mAP denotes the performance of the detection model in detecting all classes, that is, the average AP of all classes. The passengers were only detected in this work, so AP is equal to mAP.

3. Average detection time.

The average detection time can well reflect the real-time performance of the network model. The average detection times of different object detection algorithms were compared in this work.

Results and Discussion

Training Loss Comparison

The loss represents the difference between the predicted value and the true value. The smaller the training loss value, the better the model is constructed.

The training loss curves of Improved-YOLO, YOLOv4 and YOLOv4-Tiny are shown in Figure 5. The Improved-YOLO is superior to the other two models in convergence speed and final loss value. The final loss value of the YOLOv4-Tiny is 12.75 and that of the YOLOv4 is 9.34. The final loss value of the Improved-YOLO is 8.74, which is approximately 0.6 lower than that of the YOLOv4. This result shows that the detection capability of Improved-YOLO has been enhanced. The distributed training method used in the training process caused the loss values of all three models to increase significantly after the 25^th epoch. When the models are trained to the 25^th epoch, the first stage of training ends and the second stage of training begins.

Comparison of Experimental Results

Figure 6 shows partial detection results of Improved-YOLO in different scenes and different occlusion degrees. These scenes include the platform, station hall, escalator entrance, and exit. It can be seen from Figure 6 that the sizes of the passengers vary greatly depending on the camera arrangement.

To determine the best algorithm for detecting passenger flow objects, the Improved-YOLO is compared with other commonly used object detection algorithms (SSD, Faster R-CNN, YOLOv4 and YOLOv4-Tiny). The same training set, validation set, testing set, and training parameters are used in the experiment. The experimental results are shown in Table 3. Figure 7 shows the P-R curves of different models. The testing set consisted of 1,498 images, in which 6,556 people were labeled. TP indicates the number of people correctly detected in the testing set, and it reflects the statistical performance of the model.

Table 3.

Experimental Results of Different Methods

Model	Mean average accuracy (%)	Precision (%)	Recall (%)	F1 scores	Inspection time	True positives
SSD	82.77	96.55	42.31	0.59	43 ms	5,896
YOLOv4-Tiny	85.22	94.58	76.6	0.83	22 ms	5,933
Faster R-CNN	89.16	71.83	91.28	0.80	174 ms	6,262
YOLOv4	89.63	95.96	80.43	0.87	61 ms	5,979
Improved-YOLO	92.96	95.14	87.19	0.91	52 ms	6,361

Note: YOLO = You Only Look Once; SSD = single-shot multibox detector; R-CNN = region-based convolutional neural network.

From Table 3, the mAP, precision, recall, F1 scores and TP of Improved-YOLO were 92.96%, 95.14%, 87.19%, 0.91, and 6,361 respectively. Compared with other models, the mAP and TP of Improved-YOLO showed the best results. Therefore, Improved-YOLO has the strongest performance in detection and statistics. Compared with the PANet module of YOLOv4, the BiFF module of Improved-YOLO adds skip connections from the input node to the output node on the same level path, thereby reducing the network parameters of the entire model and improving the detection speed. Therefore, the detection time of Improved-YOLO is 52 ms, which is 9 ms faster than YOLOv4. The SSD is a typical one-stage target detection algorithm. Its detection speed is second only to YOLOv4-Tiny, and the single frame image detection time is 43 ms. Since the acceptable input image for the SSD is 300 × 300, it is difficult for the SSD to retain effective information when extracting image features. In particular, the subway is a scene with serious passenger flow occlusion. Therefore, the SSD has serious missed detection, and its mAP is the lowest. The YOLOv4-Tiny is a simplified version of the YOLOv4, which uses two feature layers for classification and regression prediction. Therefore, it has a very fast detection speed, and its detection time is 22 ms. The Faster R-CNN is a two-stage model with the mAP of 89.16%. Since Faster R-CNN generates candidate regions, its detection speed is greatly reduced. The detection time of Faster R-CNN is 174 ms, which makes it difficult to meet the real-time requirements.

Figure 8 shows the detection results of each model in the abnormal passenger flow (high passenger flow), and the statistical results are given in the upper left corner of the picture. As shown in Figure 8, the Improved-YOLO has satisfactory detection performance in the case of inconsistent passenger object size and abnormal passenger flow. Therefore, the Improved-YOLO is more suitable for passenger flow detection in oblique camera placement scenarios. The comparison of partial detection results between the Improved-YOLO and the YOLOv4 at the escalator entrance is shown in Figure 9. It can be seen from Figure 9b that YOLOv4 has missed detection. This result demonstrates that the Improved-YOLO effectively resolves the missed detection issues of YOLOv4.

In summary, compared with other models, the Improved-YOLO has higher accuracy and faster detection speed, which can achieve a satisfactory balance between detection accuracy and detection speed. In addition, it has satisfactory statistical ability. Therefore, the Improved-YOLO meets the requirements of subway station passenger flow detection.

Ablation Experiments

Ablation experiments were conducted to prove the validity of the SPP module, the BiFF module, and the ASFF module. This experiment was divided into five groups, and the feature extraction networks of the five groups are CSPDarknet53. We verified the effectiveness of the SPP module, the BiFF module, and the ASFF module by changing their usage count. The feature fusion network of group 1 uses one BiFF module. The feature fusion network of group 2 adds one SPP module to the last layer of the feature extraction network. By comparing the experiments of group 1 and group 2, the effectiveness of the SPP module can be verified. Group 3 and group 4 experiments are constructed by changing the number of usage counts of the BiFF module. By comparing the results of groups 2, 3, and 4, it is proved that the repeated use of the BiFF module can effectively improve the mAP of the model. Finally, group 5 adds the ASFF module based on group 4. By comparing the experiments of group 4 and group 5, the effectiveness of the ASFF module can be verified. To better compare the performance of different combinations, the same dataset is used for both training and testing. The result of the ablation experiment is shown in Table 4. Figure 10 shows the P-R curves of the ablation experiment.

Table 4.

Results of the Ablation Experiment

Group	Model	Mean average accuracy (%)
1	CSPDarknet53 + 1BiFF	85.45
2	CSPDarknet53 + SPP + 1BiFF	89.58
3	CSPDarknet53 + SPP + 2BiFF	91.77
4	CSPDarknet53 + SPP + 3BiFF	92.29
5	CSPDarknet53 + SPP + 3BiFF + ASFF (Improved-YOLO)	92.96

Note: YOLO = You Only Look Once; BiFF = bidirectional feature fusion; SPP = spatial pyramid pooling; ASFF = adaptively spatial feature fusion.

From Table 4, since the SPP module can extract multi-level features and improve the robustness of the detection model, the mAP of group 2 is 5.13% higher than that of group 1. By comparing with the experiments of groups 2, 3, and 4, the results show that the mAP of the model can be improved by repeatedly using the BiFF module in the feature fusion network. Since the ASFF module enables the model to learn how to spatially filter the effective features of other feature layers and retain the effective features for fusion, the map of group 5 is 0.67% higher than that of group 4. In summary, the SPP module, BiFF module, and ASFF module can effectively improve the accuracy of the model.

Conclusion

Traditional methods of pedestrian detection have been gradually replaced by deep learning methods. High passenger flow detection accuracy and fast detection speed play an important role in maintaining the normal operation of subway stations. In this paper, the Improved-YOLO object detection method was proposed for passenger flow detection in subway stations. The CSPDarknet53 was used to extract features as the feature extraction network of the Improved-YOLO. To improve the detection accuracy of small targets, the SPP module was added to the feature fusion network to increase the receptive field. To improve the fusion efficiency, the BiFF module was designed for a feature fusion network. To reduce the influence of target size inconsistency on detection, the ASFF module was added to the feature fusion network to filter the interference features of other feature layers. The data set used in the experiment was from Nanning Metro Line 1. First, we compared the loss values of the YOLOv4-Tiny, the YOLOv4, and the Improved-YOLO. The experimental results show that the Improved-YOLO has the lowest loss value, and the final loss value is 8.81. Second, we compared the experimental results of the Improved-YOLO with SSD, Faster R-CNN, YOLOv4-Tiny, and YOLOv4. The experimental results show that the Improved-YOLO not only has the highest mAP and statistical performance but also has a faster detection speed. Finally, the effectiveness of the SPP module, the BiFF module, and the ASFF module is verified by ablation experiments. The experimental results show that the mAP of the model can be improved by using the SPP module, the BiFF module, and the ASFF module.

Further work will focus on the following aspects:

Explore the method of improving the feature extraction network, which can better extract the feature information of small targets, and further improve the detection accuracy while ensuring the detection speed.

Explore the use of existing unsupervised learning methods to improve the problem of mutual occlusion between passengers in scenes with large passenger flow, to reduce the occurrence of missed detection.

Increase the amount of data, improve generalization ability of the model, and evaluate the model using other large public data sets.

Footnotes

Author Contributions

The authors confirm contribution to the paper as follows: study conception and design: D. He; data collection: Y. Chen, B. Liu; analysis and interpretation of results: X. Li, X. Teng; draft manuscript preparation: Y. Zhang. All authors reviewed the results and approved the submission of the manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research was supported by the National Natural Science Foundation of China (Grant No. 51765006), Major Project of Science and Technology of Guangxi Province of China (Grant No. Guike AA20302010), Interdisciplinary Scientific Research Foundation of Guangxi University Grant numbers (Grant No. 2022JCA003), and the Innovation Project of Guangxi Graduate Education (Grant No. YCSW2022102).

References

Zhu

Jiao

Zhang

Huo

Sustainable Development of Urban Metro System: Perspective of Coordination between Supply and Demand. International Journal of Environmental Research and Public Health, Vol. 18, 2021, p. 10349.

Liu

Jin

Chen

Shan

Fault Diagnosis of Flywheel Bearing Based on Parameter Optimization Variational Mode Decomposition Energy Entropy and Deep Learning. Energy, Vol. 239, 2022, p. 122108.

Shen

Shao

Chen

Hybrid Approach Combining Modified Gravity Model and Deep Learning for Short-Term Forecasting of Metro Transit Passenger Flows. Transportation Research Record, 2020. 2675: 25–38.

Wei

Zhang

Wei

Tong

Huang

Cao

Metro Passenger-Flow Representation via Dynamic Mode Decomposition and its Application. IEEE Transactions on Neural Networks and Learning Systems, Vol. 176, 2021, 109241.

Lyu

Research on Subway Pedestrian Detection Algorithm Based on Big Data Cleaning Technology. Wireless Communications & Mobile Computing, 2021, pp. 1–10.

Denman

Fookes

Sridharan

Improved Simultaneous Computation of Motion Detection and Optical Flow for Object Tracking. 2009 Digital Image Computing: Techniques and Applications, Melbourne, Victoria, Australia, December 1–3, 2009, IEEE, New York, pp. 175–182.

Zivkovic

Heijden

Efficient Adaptive Density Estimation Per Image Pixel for the Task of Background Subtraction. Pattern Recognition Letters, Vol. 27, 2006, pp. 773–780.

Sengar

S. S.

Mukhopadhyay

Motion Detection Using Block Based Bi-Directional Optical Flow Method. Journal of Visual Communication and Image Representation, Vol. 49, 2017, pp. 89–103.

Chen

Nedzvedz

O. V.

Ablameyko

S. V.

Application of Integral Optical Flow for Determining Crowd Movement From Video Images Obtained Using Video Surveillance Systems. Journal of Applied Spectroscopy, Vol. 85, 2018, pp. 126–133.

10.

Costache

Popescu

Popa

Mocanu

Moving Person Detection Using Blob Descriptors and Support Vector Machine. 25th International Conference on Systems, Signals and Image Processing (IWSSIP), Maribor, Slovenia, 2018, pp. 1–5.

11.

Zhang

Ren

Sun

Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 37, 2015, pp. 1904–1916.

12.

Zhang

Zou

Sun

Accelerating Very Deep Convolutional Networks for Classification and Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 38, 2015, pp. 1943–1955.

13.

Girshick

Donahue

Darrell

Malik

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. arxiv Preprint arxiv: 1311.2524, 2014.

14.

Girshick

Fast R-CNN. 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1440–1448.

15.

Ren

Girshick

Sun

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis & Machine Intelligence, Vol. 39, 2017, pp. 1137–1149.

16.

Gkioxari

Dollár

Girshick

Mask R-CNN. 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2980–2988.

17.

Liu

Anguelov

Erhan

Szegedy

Reed

C.-Y.

Berg

A. C.

SSD: Single Shot MultiBox Detector. European Conference on Computer Vision, Vol. 9905, 2016, pp. 21–37.

18.

Redmon

Divvala

Girshick

Farhadi

You Only Look Once: Unified, Real-Time Object Detection. arxiv Preprint arxiv: 1506.02640, 2016.

19.

Redmon

Farhadi

YOLO9000: Better, Faster, Stronger. Proc., IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, 2017, pp. 7263–7271.

20.

Redmon

Farhadi

YOLOv3: An Incremental Improvement. arxiv Preprint arxiv: 1804.02767, 2018.

21.

Bochkovskiy

Wang

Liao

H. Y. M.

YOLOv4: Optimal Speed and Accuracy of Object Detection. arxiv Preprint arxiv: 2004.10934, 2020.

22.

Sreenu

Durai

M. A. S.

Intelligent Video Surveillance: A Review Through Deep Learning Techniques for Crowd Analysis. Journal of Big Data, Vol. 6, 2019, pp. 1–27.

23.

Elbishlawi

Abdelpakey

M. H.

Eltantawy

Shehata

M. S.

Mohamed

M. M.

Deep Learning-Based Crowd Scene Analysis Survey. Journal of Imaging, Vol. 6, 2020, p. 95.

24.

Al-Allaf

O. N. A.

Review of Face Detection Systems Based Artificial Neural Networks Algorithms. arxiv Preprint arxiv: 1404.1292, 2014.

25.

Qezavati

Majidi

Manzuri

M. T.

Partially Covered Face Detection in Presence of Headscarf for Surveillance Applications. 4th International Conference on Pattern Recognition and Image Analysis (IPRIA), Tehran, Iran, 2019, pp. 195–199.

26.

Ling

Liu

Location Recognition Algorithm for Vision-Based Industrial Sorting Robot via Deep Learning. International Journal of Pattern Recognition and Artificial Intelligence, Vol. 33, No. 7, 2019, p. 1955009.

27.

Chen

Miao

Shan

Ren

Obstacle Detection in Dangerous Railway Track Areas by a Convolutional Neural Network. Measurement Science & Technology, Vol. 32, No. 10, 2021, p. 105401.

28.

Zou

Chen

Liu

Miao

Rail Transit Obstacle Detection Based on Improved CNN. IEEE Transactions on Instrumentation and Measurement, Vol. 70, 2021, pp. 1–14.

29.

Qiu

Miao

Zou

Ren

Improved Mask R-CNN for Obstacle Detection of Rail Transit. Measurement, Vol. 190, 2022, p. 110728.

30.

Chen

Wang

Yang

Yao

Liu

Wang

Deep Learning for the Detection and Recognition of Rail Defects in Ultrasound B-Scan Images. Transportation Research Record, 2021. 2675: 888–901.

31.

Saba

Mohamed

A. S.

El-Affendi

Amin

Sharif

Brain Tumor Detection Using Fusion of Hand Crafted and Deep Learning Features. Cognitive Systems Research, Vol. 59, 2020, pp. 221–230.

32.

Ari

Hanbay

Deep Learning Based Brain Tumor Classification and Detection System. Turkish Journal of Electrical Engineering & Computer Sciences, Vol. 26, 2018, pp. 2275–2286. DOI: 10.3906/elk-1801-8.

33.

Zhang

Fang

Wang

Yin

Deep People Counting with Faster R-CNN and Correlation Tracking. Proc. International Conference on Internet Multimedia Computing and Service, 2016, pp. 57–60.

34.

Kajabad

E. N.

Ivanov

S. V.

People Detection and Finding Attractive Areas by the Use of Movement Detection Analysis and Deep Learning Approach. Procedia Computer Science, Vol. 156, 2019, pp. 327–337.

35.

Dagli

Eken

Deploying a Smart Queuing System on Edge With Intel OpenVINO Toolkit. Soft Computing, Vol. 25, 2021, pp. 10103–10115.

36.

Ahmad

Ahmed

Ullah

Ahmed

A Deep Neural Network Approach for Top View People Detection and Counting. 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), New York, NY, 10–12 October 2019, IEEE, New York, pp. 1082–1088.

37.

Ahmed

Ahmad

Jeon

Top View Multiple People Tracking by Detection Using Deep SORT and YOLOv3 With Transfer Learning: Within 5G Infrastructure. International Journal of Machine Learning and Cybernetics, Vol. 12, 2021, pp. 3053–3067.

38.

Lin

T. Y.

Dollár

Girshick

Hariharan

Belongie

Feature Pyramid Network for Object Detection. arxiv Preprint arxiv: 1612.03144, 2017.

39.

Liu

Qin

Shi

Jia

Path Aggregation Network for Instance Segmentation. arxiv Preprint arxiv: 1803.01534, 2018.

40.

Barshan

Fieguth

Stage-Wise Training: An Improved Feature Learning Strategy for Deep Models. In Feature Extraction: Modern Questions and Challenges, PMLR, Canada, Montreal, 2015, pp. 49–59.

41.

Yang

Yan

Yuan

Jin

Why Does Stagewise Training Accelerate Convergence of Testing Error Over SGD. arXiv preprint arXiv:1812.03934, 2018.