Multi-Pedestrian Tracking Based on Improved YOLOv8 and OC-SORT

Abstract

Multi-pedestrian tracking is an important task for the environment perception systems of autonomous vehicles. In the multi-pedestrian tracking task, mutual occlusion, posture changes, small size, and poor lighting conditions usually pose challenges. To overcome these problems, we propose a detection-based multi-pedestrian tracking method, that is, combining the improved You Only Look Once (YOLO) v8 object detection algorithm with the improved observation-centric simple online and real-time tracking (OC-SORT) algorithm. Specifically, first, we improve the YOLOv8 pedestrian detector by constructing a C2f-Clo block, introducing an explicit visual center block, and designing a lightweight shared convolutional detection head. Second, we improve the OC-SORT tracker using a height-modified intersection over union. Results of experiments on the MOT17 and MOT20 pedestrian tracking datasets show that our method achieves 7.1% and 6.5% HOTA boosts, 8.5% and 8.8% MOTA improvements, 5.5% and 6.1% MOTP increases, 5.2% and 6.7% IDF1 boosts, and 648 and 692 IDSW decreases, respectively, compared with the baseline.

Keywords

multi-pedestrian tracking autonomous vehicle artificial intelligence

Introduction

The explosive development of AI technology has made autonomous driving possible, and multi-pedestrian tracking, which aims to detect and track multiple pedestrians continuously in a video sequence, is a fundamental and important key technology in the environment perception task of autonomous driving; the key lies in accurately identifying the pedestrians in the video and maintaining their identity consistency among video frames, that is, making sure that each pedestrian has a unique and continuous identification. (This technology is also widely used in surveillance, robot navigation, and so forth) ( 1 ).

In recent years, thanks to powerful computing devices and continuous innovation of algorithms, deep-learning-based object detection algorithms have made impressive achievements, and detection-based multi-pedestrian tracking methods have gradually become mainstream. These methods usually consist of two main components: the former is a pedestrian detector for localizing the position of pedestrians, and the latter is a data association module for matching detection results in current frames with predicted trajectories in previous frames based on pedestrians’ re-recognition features ( 2 ).

Despite the impressive achievements of existing multi-pedestrian tracking methods, they still do not perform well in some challenging situations and, in general, we believe that several main challenges remain: 1) The pedestrian’s size on the image shows obvious differences with distance and, usually, pedestrians at long distances occupy very few pixels in the image, which can easily lead to missed tracking and ID switching; 2) In crowded scenarios, mutual occlusion is unavoidable, which often leads to problems; 3) Poor lighting, posture changes, and so forth, lead to possible changes in the appearance of pedestrians, which requires the algorithm to have a high robustness; 4) For applications that require real-time response, such as autonomous driving, the delay of the algorithm should be minimized.

In this paper, we propose a new detection-based multi-pedestrian tracking algorithm. We combine the improved You Only Look Once (YOLO) v8 object detection algorithm and the improved observation-centric simple online and real-time tracking (OC-SORT) algorithm. The improved YOLOv8 algorithm is the pedestrian detector, which delivers the pedestrian’s bounding box to the downstream OC-SORT tracker for data association, and the two are combined to ultimately output the pedestrian’s bounding box, ID number, and historical trajectory.

Considering that the performance of the pedestrian detector will directly affect the performance of our method, we make a series of improvements to YOLOv8. Specifically, first, we embed the Clo-block proposed by CloFormer into the bottleneck in C2f, thus constructing the novel C2f-Clo block, and then we replace all the C2f-blocks in the backbone network with C2f-Clo blocks ( 3 ). Second, we introduce an explicit visual center (EVC) mechanism at the back-end of the backbone network. The EVC detector is used to improve the algorithm’s detection performance for small-size dense pedestrians. Third, we design a lightweight shared convolutional detection head—thanks to the shared convolution, the number of parameters is drastically reduced, which makes the model lighter yet outputs more refined predictions. To further enhance the robustness of the pedestrian tracker, we optimize OC-SORT as well. Specifically, we introduce a height-modified IoU (HIoU), that is, introducing a height state to modify the IoU to enhance the tracker’s ability to associate data when pedestrians occlude and overlap each other.

We conduct a series of ablation and comparison experiments on the MOT17 and MOT20 pedestrian tracking datasets, and the experiment results show that our method has some degree of improvement in HOTA, MOTA, MOTP, and IDF1 (although IDSW has a marked decrease) which suggests that all the above improvements are fruitful.

In summary, the key contributions of our proposed method are as follows:

1) We construct a novel C2f-Clo block by borrowing the Clo-block proposed by CloFormer, which significantly improves the performance of the pedestrian detector without significantly increasing the computational burden.

2) We introduce an EVC mechanism at the back-end of the backbone network, which improves the algorithm’s detection performance for small-size dense pedestrians.

3) We design a lightweight shared convolutional detection head, which is lighter but produces higher quality predictions.

4) We introduce an HIoU, which enhances the robustness of the pedestrian tracker.

Related Work

Our proposed multi-pedestrian tracking approach utilizes a detection-based tracking paradigm; thus, we present recent related research in the Object Detection and Multi-Object Tracking sections.

Object Detection

In the past decade, thanks to the boost in performance of computing devices and the continuous innovation of algorithms, deep-learning-based object detection algorithms have experienced rapid development, which can be generally categorized into two-stage and one-stage methods. Two-stage methods prioritize detection accuracy, and example models include Faster region-based convolutional neural network (R-CNN), Mask R-CNN, and Cascade R-CNN. The R-CNN algorithm is a pioneering work in two-stage object detection algorithms and, to achieve full end-to-end object detection, researchers continue to develop the Fast R-CNN architecture that employs a fully connected (FC) layer to perform the tasks of category classification and bounding box regression simultaneously ( 4 , 5 ). Based on the above research, Ren et al. proposed a more efficient Faster R-CNN algorithm by replacing the selective search algorithm with a region proposal network ( 6 ). Mask R-CNN extends Faster R-CNN by adding a branch to predict the binary mask for each instance along with the standard bounding box regression branch ( 7 ). Cascade R-CNN progressively improves detection results by cascading detectors, with each detection stage optimized based on the output of the previous stage, gradually improving detection accuracy ( 8 ). One-stage methods prioritize inference speed, and example models include the YOLO series and the Single Shot MultiBox Detector (SSD). YOLOv5 is the masterpiece of the YOLO series, and it has a series of optimizations based on the previous version, including using adaptive anchor and image scaling, in addition to optimizing the loss function and activation function. Although the early YOLO algorithm has achieved remarkable results, it still suffers from problems such as insufficient feature characterization capability. For this reason Liu et al. proposed SSD to improve the detection performance for various sized objects by combining multi-scale feature fusion ( 9 ). Subsequent researchers have improved the shortcomings of SSD in specific scenarios. Yi et al. proposed the Attentive Single Shot MultiBox Detector (ASSD), which enables the network to better focus on the object region through an attention mechanism ( 10 ). Zhai et al. proposed a DenseNet and Feature Fusion-based Single Shot MultiBox Detector (DFSSD), which takes the visual perception information into account and improves the detection performance for small objects in dense scenarios ( 11 ). Meanwhile, newer versions of algorithms have been proposed, such as YOLOv8 which introduces a more advanced feature extraction network based on YOLOv5, and adopts the anchor-free detection head to achieve more accurate object detection. Although two-stage object detection algorithms usually outperform one-stage object detection algorithms in detection accuracy, considering the higher real-time requirements of multi-pedestrian tracking systems, we favor YOLOv8, which is faster in its inference and more flexible, and we improve it effectively to make up for the lack of detection accuracy.

Multi-Object Tracking

Multi-object tracking algorithms can be broadly categorized into detection-based (tracking by detection [TBD]) paradigms and joint detection and tracking (JDT) paradigms. Simple online and real-time tracking (SORT) is one of the earliest multi-object tracking algorithms, which integrates the Kalman filter with the Hungarian matching algorithm to realize real-time online tracking, but it does not consider the appearance features and thus lacks robustness ( 12 ). DeepSORT introduces appearance feature re-identification and cascade matching strategies based on SORT to reduce ID switching and miss tracking ( 13 ). Cai et al. proposed Multi-Object Tracking with Memory (MeMOT), which contains a long-term spatiotemporal memory mechanism that stores the features of the tracked object in all past frames and encodes the stored information as tracking cues, which achieves simultaneous detection and tracking ( 14 ). Cao et al. proposed OC-SORT, which quickly matches and tracks objects through a sorting algorithm, and is especially good at handling occlusion and overlapping situations ( 15 ). All above algorithms belong to the TBD paradigm algorithms. Zhang et al. proposed FairMOT, a JDT paradigm algorithm which employs CenterNet as a detector, uses a shared convolutional neural network to simultaneously handle two tasks of object detection and feature extraction, and integrates low-dimensional information to represent re-identified features ( 16 , 17 ). Although its tracking accuracy and real-time performance are greatly improved, CenterNet is difficult to use in crowded scenes to detect pedestrians with high overlap. Other than that, Kernelized Correlation Filters (KCF), Tracktor++, and CenterTrack are also representative JDT algorithms (18 –20). Although the JDT paradigm has been popular in recent years, the flexibility and increasing performance of detection algorithms make the TBD paradigm still the dominant research trend at present, so we chose the OC-SORT algorithm and optimize it to further improve the tracking performance.

Method Design and Implementation

We briefly describe our pedestrian detector in the Our Improved YOLOv8 section, and detail a series of network improvements in the C2f-Clo Block Construction, EVC Block Introduction, and Lightweight Shared Convolution Detection Head sections. Finally, we present the enhancements to OC-SORT in the HIoU Introduction section.

Our Improved YOLOv8

YOLOv8 provides a variety of scale models, such as YOLOv8n, YOLOv8s, and YOLOv8l, to meet the needs of different hardware platforms and application scenarios. Compared with previous versions, YOLOv8 has made significant improvements and innovations in several aspects. First, in the backbone network, YOLOv8 adopts C2f-blocks with richer gradient flow. Second, YOLOv8 incorporates a variety of attention mechanisms and introduces deformable convolution and dynamic convolution techniques, among others, which help the model capture detailed information more accurately. In addition, YOLOv8 introduces new feature fusion modules such as the Bi-directional Feature Pyramid Network (BiFPN) and Asymptotic Feature Pyramid Network (AFPN), and, for loss function, YOLOv8 introduces “distribution focal loss,” a new loss function that can better balance the positive and negative samples, thus improving the model’s learning efficiency during the training process. Finally, YOLOv8 adopts the anchor-free detection head, an upgrade that markedly improves detection accuracy, especially in dealing with small-size objects.

Figure 1 shows the network structure of the improved YOLOv8 algorithm. Improvements have been made to the backbone network, neck, and detection head, as illustrated in the figure. Specifically, the C2f-blocks in the original backbone network and neck have been replaced with the C2f-Clo blocks we constructed. Additionally, an EVC module has been introduced at the end of the backbone network. Finally, there is the lightweight shared convolution detection head designed by us.

C2f-Clo Block Construction

CloFormer is a lightweight visual transformer suitable for deployment on mobile and edge computing devices. CloFormer demonstrates excellent performance on several vision tasks with little additional computational burden. The Clo-block, which is the key to the CloFormer’s effectiveness, consists of a global branch and a local branch, which perceive low-frequency global information and high-frequency local information, respectively. Considering the excellent performance of the Clo-block, we try to embed it into the bottleneck of the C2f-block to construct a new block, called C2f-Clo, with which we replace the original C2f-block. Its network structure is shown in Figure 2; the Clo-block was placed behind the two convolution blocks in the bottleneck.

First, feature maps are extracted through a convolutional layer, then split to form multiple parallel branches. Each branch refines features through a “Clo-BottleNeck” unit containing residual connections (including specifically processed Clo-block and convolutional layers). Finally, all branch outputs are fused through concatenation, and then integrated through the final convolutional layer to generate the final feature representation.

Figure 3 describes the network structure of the Clo-block in detail. The left half is a global branch that uses the traditional attention mechanism but down-samples the Key (K) and Value (V) to reduce computation. Equation 1 describes the computation of the global branch.

The left global branch realizes cross-region feature interaction through pool, matrix multiplication, and Softmax to capture overall patterns; the right local branch performs fine-grained local feature extraction using depth-wise convolution (DWconv), element-wise multiplication, and Swish/Tanh activation functions. Both branches share the layer normalization input, and finally fuse global and local information through concatenation and fully FC layer, forming a complementary feature representation.

X_{global} = Attention (Q_{g}, Pool (K_{g}), Pool (V_{g}))

(1)

The right half is a well-designed local branch that fuses the global shared weights commonly used in traditional convolution and the token-specific context-aware weights appearing in attention to achieve efficient aggregation of local information at high frequencies. Specifically, it first extracts local information through DWconv with shared weights, and then uses a gating mechanism to generate context-aware weights for enhancing local features. The computation process of the local branch is shown in Equation 2. Finally, the features extracted from the two branches are concatenated and fed into a Multilayer Perceptron (MLP) to obtain the final output.

\begin{matrix} Q, K, V = FC (X_{in}) \\ Q_{l} = DWconv (Q) \\ K_{l} = DWconv (K) \\ A_{{ttn}_{t}} = FC (Swish (FC (Q_{l} ⊙ K_{l}))) \\ A_{ttn} = Tanh (\frac{Att n_{t}}{\sqrt{d}}) \\ X_{local} = Attn ⊙ V_{s} \end{matrix}

(2)

EVC Block Introduction

Many past experiences have demonstrated the positive effect of modulating intra-layer features to improve the performance of object detection algorithms; for example, some methods use attention mechanisms or visual transformer to learn compact intra-layer feature representations, and tune the intra-layer features to make the model focus on the important parts of the input information, while ignoring irrelevant or less informative parts. Such methods have achieved undeniable results, but they ignore corner regions, which are very important for dense prediction tasks such as pedestrian detection. To address this problem, we introduce an EVC mechanism, which is shown in experiments to remarkably improve the detection performance of small-size dense objects ( 21 ).

EVC consists of two parallel connected blocks—a lightweight MLP block for capturing global long-range dependencies and a learnable vision-center (LVC) block for obtaining local corner information—and the output levies of the two blocks are concatenated along the channel dimensions as the final output of the EVC. Figure 4 depicts the network structure of EVC in detail.

The upper part is a lightweight MLP block for capturing long-range dependencies (refer to the correlation relationships between distant pixels or regions in images) through group normalization (GN) and DWConv to extract global features; the lower part is an LVC block which introduces a codebook mechanism (codebook) to select key features of local corner regions (easily overlooked yet discriminative local regions in images), achieving region-level abstraction. The features processed by both branches are concatenated and then fused and output by a convolutional layer. Here, Conv-BN-ReLU extracts the input features in the order of “convolution-batchnormalization-ReLU,” forming a more robust feature representation.

MLP block: The DWconv-based residual block and the channel MLP-based residual block are concatenated to form a lightweight MLP block. Specifically, the input feature x is first fed to a GN layer, then it enters a DWconv layer, then passes through channel scaling and DropPath, and finally element-wise addition with x to obtain the output of the DWconv-based residual block, $\tilde{X}$ . This process can be expressed by Equation 3:

\tilde{X} = DWconv (GN (X)) + X

(3)

In a similar process to the one described above, $\tilde{X}$ as the input to the channel MLP-based residual block is first fed to a GN layer, then enters a channel MLP layer, then goes through channel scaling and DropPath, and finally undergoes element-wise addition with $\tilde{X}$ to get the output of the entire lightweight MLP, $MLP (X)$ . This process can be described by Equation 4:

MLP (X) = CMLP (GN (\tilde{X})) + \tilde{X}

(4)

LVC Block: LVC is an encoder with a fixed dictionary. Specifically, the input feature $X$ is encoded by a set of convolutional layers and then fed into the codebook. In the codebook, a set of scaling factors $S = {s_{1}, s_{1}, . . ., s_{k}}$ is used to make pixel features and visual codewords map the corresponding position information; from this, the information of the whole image relative to the i-th codeword $e_{i}$ is obtained, and then a BN-ReLU-mean layer is used to fuse $e_{1}, e_{2} \dots e_{k}$ to compute the output of the codebook, $e$ . After that, $e$ is fed to an FC layer and then to a $1 \times 1$ convolutional layer, and then multiplied channel-wise with the input feature $X$ to get the local corner region feature $Z$ . The above process can be summarized by Equations 5 and 6:

e = \sum_{i = 1}^{k} φ (e_{i})

(5)

Z = X \otimes (δ ({Conv}_{1 \times 1} (e)))

(6)

Finally, we sum the input feature $X$ with the local corner region feature $Z$ channel-wise to get the final output of the LVC block, $LVC (X)$ .

It should be noted that we found from a series of experiments that the best improvement was achieved by placing the EVC block at the back-end of the last layer of the backbone network; therefore, we build the network framework according to this principle.

Lightweight Shared Convolution Detection Head

The detection head is a key component of an object detection model, and a high-quality detection head can enhance the model’s localization and classification capabilities, leading to better performance in complex scenes. Optimizing the detection head structure is often an effective way to improve the overall performance of an object detection model, so we designed a novel lightweight shared convolutional detection head.

Conv_GN is the key component of the detection head, which consists of a convolution, a GroupNorm layer, and SiLu activation function. GroupNorm is a common normalization method, which has been proved to improve the performance of the detection head for localization and classification in Fully Convolutional One-Stage Object Detection (FCOS) ( 22 ). It divides the feature map channels into multiple groups, and then computes the mean and standard deviation within each group. Compared with BatchNorm, GroupNorm is insensitive to batch size and has higher robustness.

Figure 5 depicts our designed detection head in detail. Specifically, the input features from the neck are first fed into the first Conv_GN block and then flow into the Conv_GN blocks of the two shared convolutions. Finally, the regression heads predict the bounding box and confidence scores. The number of parameters is drastically reduced by using shared convolution; the model becomes lighter and more suitable for running on resource-constrained devices, which is also beneficial for improving inference speed. In addition, we scale the predictions to deal with the problem for inconsistent object scales by using the scale layer, where the scaling factor $scale$ is a learnable parameter that gets updated as the training process goes on.

HIoU Introduction

OC-SORT is a SORT-like multi-object tracking algorithm that utilizes position and motion information for more accurate prediction and matching by incorporating an observation-centered mechanism. It pays special attention to the detection results (i.e., observations) in each frame, and uses an observation-based matching strategy during data correlation which is more capable of dealing with the problems of object occlusion, overlapping, and fast movement, thus improving the stability and accuracy of tracking. Even though OC-SORT improves in handling short-term occlusions, tracking errors may still occur in complex scenes. For this reason, we introduce an HIoU; that is, a height state is introduced to modify the IoU to enhance the data association ability of the tracking algorithm.

In general, the height state can increase the data association ability in two aspects. First, in common pedestrian tracking datasets, the height of the bounding box reflects the distance between the object and the camera, to some extent, and the height state will be an effective distinguishing cue for certain overlapping objects. Second, the height state is an accurately estimated state with high robustness against pedestrian posture change.

The calculation of the HIoU is depicted in Figure 6. IoU is equal to the area of the intersection region of two boxes divided by the area of the concatenation region, and the Height Ratio (HR) is equal to the height of the intersection region of two boxes divided by the height of the concatenation region. We multiply the HR and IoU element-wise to obtain the final HIoU.

Experiment

We conducted a set of experiments on the MOT17 and MOT20 datasets to evaluate the performance of the algorithm. Compared with the baseline, our method demonstrates superior performance in the pedestrian tracking task, which indicates that our various improvements are fruitful.

Datasets

The CityPersons dataset is an open dataset designed specifically for pedestrian detection and tracking tasks, which focuses on pedestrian detection, localization, and tracking in autonomous driving and intelligent transportation systems. The dataset contains 5,000 images from street view and driving view. We trained the pedestrian detector with this dataset.

MOT17 and MOT20 are two datasets widely used for multi-pedestrian tracking tasks. MOT17 has a total of 14 video sequences, of which seven are training sets with labels, and the other seven are test sets. MOT20 is a newer version of the MOT series which has a total of eight video sequences, of which four are training sets and the other seven are test sets, and provides more complex scenarios including more object occlusions and faster motion than MOT17. Overall, both MOT17 and MOT20 contain diverse urban environments and pedestrian postures as well as complex lighting and weather conditions, and they are highly challenging multi-pedestrian tracking datasets which facilitate objective and accurate evaluation of the performance of our proposed method.

Metrics

In this paper, we evaluate our multi-pedestrian tracking method using the five commonly used metrics in the field of multi-object tracking: HOTA, MOTA, MOTP, IDF1, and IDSW ( 23 ). HOTA is the geometric mean of detection accuracy and association accuracy, and can comprehensively reflect the performance of detection and association. MOTA focuses more on the detection performance. MOTP is an important evaluation metric that reflects the accuracy in estimating the location of the object. IDFl provides a comprehensive evaluation for identity retention performance. There is also IDSW, which is the number of object identity switches, which can directly reflect the object identity retention capability of the tracking algorithm.

The multi-object tracking performance is typically measured by MOTA:

MOTA = 1 - \frac{\sum_{t} ({FP}_{t} + {FN}_{t} + {IDSW}_{t})}{\sum_{t} {GT}_{t}}

(7)

where

${FP}_{t}$ = the number of targets detected by the tracker in frame t,

${FN}_{t}$ = the number of true targets missed by the tracker in frame t,

IDSW = the number of incorrect ID switches of the same target trajectory during tracking, and

${GT}_{t}$ = the total number of real targets.

MOTP evaluates the algorithm’s ability to capture the target position by quantifying the overlap or distance error between the tracking box and the ground-truth box. The calculation formula of MOTP is as follows:

MOTP = 1 - \frac{\sum_{t, i} d_{t, i}}{\sum_{t} c_{t}}

(8)

where

$c_{t}$ = the number of matches in frame t, and

$d_{t, i}$ = the IoU between target i and its ground truth in the t-th frame.

IDF1 is the ratio of correctly identifed detections over the average number of ground-truth and computed detections:

IDF 1 = \frac{2 IDTP}{2 IDTP + IDFN + IDFP}

(9)

where

IDFN = the number of false negative IDs,

IDFP = the number of false positive IDs, and

IDTP = the number of true positive IDs.

Implementation Details

The implementation of these experiments was based on the following: 1) CPU: Intel i9-13900K; 2) GPU: NVIDIA GeForce RTX4090; and 3) ubuntu18.04, CUDA 11.3 and PyTorch 1.12.0. In addition, we used Python 3.8 as the programming language and loaded pre-trained model on ImageNet as initial weights. We used the Adam optimizer with an adaptive dynamically adjusted learning rate with an initial value set to 0.0001. The batch size was set to 16, and a total of 300 epochs were trained. An early stopping strategy was adopted to avoid overfitting.

Experiment Results and Analysis

We combined a series of detectors and trackers to conduct ablation and comparison experiments to validate the effectiveness of our proposed improvements. We used mobilenetv2 as the pedestrian re-identification model in the experiment.

First, we used the C2f-Clo block mentioned above, the EVC block, and the self-developed lightweight shared convolution detection head to combine to obtain eight different pedestrian detectors, and coupled these detectors with the OC-SORT tracker to perform sets of ablation experiments on the MOT17 and MOT20 datasets. The results of the experiments are shown in Tables 1 and 2, respectively.

Table 1.

Pedestrian Tracking Ablation Experiment Results on the MOT17 Dataset Using OC-SORT

Yolov8n	C2f-Clo	EVC	LSDH	HOTA↑	MOTA↑	MOTP↑	IDF1↑	IDSW↓
√				58.6	71.3	78.3	72.7	1,934
√	√			60.5	73.8	81.2	74.8	1,623
√		√		59.7	72.7	81.1	74.2	1,741
√			√	59.3	71.9	80.8	73.5	1,617
√	√	√		63.7	76.5	82.1	76.3	1,552
√	√		√	62.6	75.3	82.8	75.0	1,449
√		√	√	61.8	74.4	82.4	74.8	1,515
√	√	√	√	64.6	77.9	82.2	77.4	1,316

Note: EVC = explicit visual center; LSDH = the self-developed lightweight shared convolution detection head; OC-SORT = observation-centric simple online and real-time tracking; YOLO = you only look once; HOTA = higher order tracking accuracy; MOTA = multiple object tracking accuracy; MOTP = multiple object tracking precision; IDF1 = identification F1-score; IDSW = identity switches; √ = module included.

Bold values indicate the best results.

Table 2.

Pedestrian Tracking Ablation Experiment Results on the MOT20 Dataset Using OC-SORT

YOLOv8n	C2f-Clo	EVC	LSDH	HOTA↑	MOTA↑	MOTP↑	IDF1↑	IDSW↓
√				56.4	69.1	74.1	67.2	1,654
√	√			58.5	72.4	75.9	69.7	1,468
√		√		57.7	71.8	76.4	68.3	1,540
√			√	56.9	71.3	76.0	67.6	1,599
√	√	√		61.2	75.7	77.1	73.7	1,389
√	√		√	60.5	74.5	78.2	72.3	1,406
√		√	√	60.0	73.8	77.5	70.5	1,282
√	√	√	√	62.3	77.6	78.0	74.6	1,247

Bold values indicate the best results.

The results of the ablation test reveal the effects of different components on the overall tracking system performance. The tracking system’s HOT, MOTA, and IDF1 scores are relatively low when no components are added, and the performance of the tracking system shows an obvious upward trend with the gradual introduction of the C2f-Clo, EVC, and LSDH components. In particular, the tracking system reaches the highest values for the metrics of HOTA, MOTA, and IDF1 on the MOT17 dataset when these components are added at the same time, which are improved to 65.2%, 79.8%, and 78.7%, respectively, while IDSW is reduced to 1,286. Similarly, the tracking system reached the highest values for HOTA, MOTA, and IDF1 on the MOT20 dataset, which were improved to 62.9%, 77.9% and 76.1%, respectively, while IDSW was reduced to 962. Both sets of experiments demonstrated a remarkable improvement in the tracking stability and accuracy of the system with the addition of these components. It is worth noting that the inclusion of each component has a positive impact on system performance, but the degree of contribution is variable for different components; for example, the inclusion of C2f-Clo and EVC is particularly significant in improving system performance, while the introduction of LSDH contributes less. In addition, the combination of two components typically resulted in higher performance gains than one component alone, which suggests that there may be some synergistic effect among these components that work together to optimize the tracking capability of the system.

Further, we coupled the eight different pedestrian detectors mentioned above with the OC-SORT tracker with HIoU, and conducted comparative experiments at MOT17 and MOT20, to verify whether the introduction of HIoU is effective in enhancing the OC-SORT tracker. Tables 3 and 4 show the results of the experiments at MOT17 and MOT20, respectively. Looking closely at these results, it is easy to find that HOTA, MOTA, MOTP, and IDF1 are improved to different degrees, while IDSW is obviously reduced, which indicates that the OC-SORT tracker with HIoU has better performance.

Table 3.

Pedestrian Tracking Ablation Test Results for the MOT17 Dataset OC-SORT with Height-Modified IoU

YOLOv8n	C2f-Clo	EVC	LSDH	HOTA↑	MOTA↑	MOTP↑	IDF1↑	IDSW↓
√				59.7	72.7	80.0	74.5	1,889
√	√			61.7	75.3	82.5	76.1	1,577
√		√		61.5	74.6	82.4	75.5	1,702
√			√	60.6	73.0	82.8	75.2	1,594
√	√	√		65.1	78.6	84.0	77.9	1,515
√	√		√	64.1	76.4	84.2	76.4	1,408
√		√	√	63.2	75.7	83.9	76.7	1,473
√	√	√	√	65.7	79.8	83.8	78.7	1,286

Note: EVC = explicit visual center; IoU = intersection over union; LSDH = the self-developed lightweight shared convolution detection head; OC-SORT = observation-centric simple online and real-time tracking; YOLO = you only look once; HOTA = higher order tracking accuracy; MOTA = multiple object tracking accuracy; MOTP = multiple object tracking precision; IDF1 = identification F1-score; IDSW = identity switches; √ = module included.

Bold values indicate the best results.

Table 4.

Pedestrian Tracking Ablation Test Results for the MOT20 Dataset Using SORT with Height-Modified IOU

YOLOv8n	C2f-Clo	EVC	LSDH	HOTA↑	MOTA↑	MOTP↑	IDF1↑	IDSW↓
√				58.6	70.9	76.1	68.3	1894
√	√			60.8	74.5	77.6	71.3	1588
√		√		59.6	74.0	78.8	70.1	1703
√			√	58.9	73.0	78.1	69.1	1575
√	√	√		62.4	77.7	78.9	73.6	1522
√	√		√	61.8	76.8	80.4	73.0	1416
√		√	√	61.3	75.7	79.5	72.3	1470
√	√	√	√	62.7	78.2	80.2	75.7	1212

Bold values indicate the best results.

In addition to verifying the effectiveness of the above improvements, we have conducted several comparison experiments among the recent SOTA algorithms in the field of multi-object tracking and our proposed method on MOT17 and MOT20. The results of the experiments are shown in Tables 5 and 6. It is obvious that our method does not underperform compared with SOTA algorithms in recent years, with the highest MOTA and IDF1 score on MOT17 and MOT20, respectively.

Table 5.

Performance Comparison with Preceding SOTAs on MOT17

Method	HOTA↑	MOTA↑	IDF1↑	IDSW↓
FairMOT	59.3	73.7	72.3	3,303
TransCenter ( 24 )	54.5	73.2	62.2	4,614
CorrTracker ( 25 )	60.7	76.5	73.6	3,369
Semi-TCL ( 26 )	59.8	73.3	73.4	2,790
RelationTrack ( 27 )	61.0	73.8	74.7	1,374
SOTMOT ( 28 )	/	71.0	71.9	5,184
MAATrack ( 29 )	62.0	79.4	75.9	1,452
OC-SORT	61.7	76.0	76.2	2,199
StrongSORT ( 30 )	63.5	78.3	78.5	1,446
ByteTrack ( 31 )	63.1	80.3	77.3	2,196
BoT-SORT ( 32 )	65.0	80.5	80.2	1,212
SMILEtrack ( 33 )	65.3	81.1	80.5	1,246
Ours	65.7	79.8	78.7	1,286

Note: HOTA = higher order tracking accuracy; MOTA = multiple object tracking accuracy; IDF1 = identification F1-score; IDSW = identity switches.

Bold values indicate the best results.

Table 6.

Performance Comparison with Preceding SOTAs on MOT20

Method	HOTA↑	MOTA↑	IDF1↑	IDSW↓
FairMOT	54.6	61.8	67.3	5,243
TransCenter ( 24 )	/	61.9	50.4	4,653
CorrTracker ( 25 )	/	65.2	69.1	5,183
Semi-TCL ( 26 )	55.3	65.2	70.1	4,139
RelationTrack ( 27 )	56.5	67.2	70.5	4,243
SOTMOT ( 28 )	/	68.6	71.4	4,209
MAATrack ( 29 )	57.3	73.9	71.2	1,331
OC-SORT	60.5	73.1	74.4	1,307
StrongSORT ( 30 )	61.5	72.2	75.9	1,066
ByteTrack ( 31 )	61.3	77.8	75.2	1,223
BoT-SORT ( 32 )	63.3	77.8	77.5	1,257
SMILEtrack ( 33 )	63.4	78.2	77.5	1,318
Ours	62.7	78.2	75.7	1,212

Note: HOTA = higher order tracking accuracy; MOTA = multiple object tracking accuracy; IDF1 = identification F1-score; IDSW = identity switches.

Bold values indicate the best results.

Visualization Results and Analysis

The top and bottom rows of Figure 7 show the tracking results of the baseline algorithm and our proposed method on MOT17-03 at frames 135, 241, and 450, respectively.

Let us focus on the two pedestrians indicated by the yellow arrow and the green arrow in the picture. In frame 135, the pedestrian indicated by the green arrow is walking toward the location of the pedestrian indicated by the yellow arrow, and their IDs are #51 and #111, respectively. In the following frame 241, their positions overlap and, at this time, the ID of the pedestrian indicated by the green arrow still remains #111, while the pedestrian indicated by the yellow arrow is almost completely occluded, resulting in a missed tracking. In the following frame 450, the two pedestrians’ positions no longer overlapped, but the ID of the pedestrian indicated by the yellow arrow changed from the initial #51 to #111, while the pedestrian indicated by the green arrow gets a new ID. In summary, the IDs of the two pedestrians have changed because of occlusion.

Next, let us analyze the tracking results of our proposed method. At frame 135, their IDs are #45 and #100, respectively. At frame 241, although their positions overlap and the pedestrians indicated by the yellow arrows are almost completely occluded, there is no more mistracking and they still keep their IDs. At frame 450, the pedestrians are no longer overlapped, and their IDs are the same as the initial ones, which shows that our proposed method has quite good performance in multi-pedestrian tracking. The mistracking and ID switching problems in the baseline algorithm have been alleviated, which suggests that the improvements we made to the baseline algorithm are meaningful.

Conclusions

In this paper, we propose a detection-based multi-pedestrian tracking algorithm. We use a modified YOLOv8 as a pedestrian detector and a modified OC-SORT as a tracker, and the two are combined to achieve the multi-pedestrian tracking task. We build a C2f-Clo block by borrowing the Clo-block proposed in CloFormer, introduce an explicit vision center mechanism, design a lightweight shared convolutional detection head, and, finally, introduce an HIoU. Experiments on MOT17 and MOT20 show that all the improvements are fruitful, and our algorithm even achieves comparable performance to that of the SOTA in recent years, which shows that our work is meaningful. We will continue our research, and hope our work will push the field forward and contribute more to the field of multi-pedestrian tracking.

Footnotes

Author Contributions

The authors confirm contribution to the paper as follows: study conception and design: J. Liu, S. Zhang; data collection: J. Liu, Y. Zhao; analysis and interpretation of results: J. Liu, S. Zhang, L. Guo; draft manuscript preparation: J. Liu, S. Zhang. All authors reviewed the results and approved the final version of the manuscript.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Jian Liu

Shixin Zhang

References

Wang

Guo

Robust Pedestrian Multi-Object Tracking in the Intelligent Bus Environment. Measurement Science and Technology, Vol. 35, No. 10, 2024, p. 105401.

Wang

Liu

Chen

Wang

Optimization research on pedestrian multiobjects tracking model based on TBD strategy. Proc., International Conference on Algorithm, Imaging Processing, and Machine Vision (AIPMV 2023), Qingdao, China, 2024.

Fan

Huang

Guan

Rethinking Local Perception in Lightweight Vision Transformer. arXiv Preprint arXiv:2303.17803, 2023.

Girshick

Donahue

Darrell

Malik

, eds. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proc., 27th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, June 23, 2014–June 28, 2014, Columbus, OH, IEEE Computer Society.

Girshick

, ed. Fast R-CNN. Proc., 15th IEEE International Conference on Computer Vision, ICCV 2015, December 11, 2015–December 18, 2015, Santiago, Chile, Institute of Electrical and Electronics Engineers Inc.

Ren

Girshick

Sun

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, No. 6, 2017, pp. 1137–1149.

Gkioxari

Dollár

Girshick

Mask R-CNN. Proc., IEEE International Conference on Computer Vision, 2017, pp. 2961–2969.

Cai

Vasconcelos

, eds. Cascade R-CNN: Delving into High Quality Object Detection. Proc., 31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018, June 18, 2018–June 22, 2018, Salt Lake City, UT, IEEE Computer Society.

Liu

Anguelov

Erhan

Szegedy

Reed

C.-Y.

Berg

A. C.

, eds. SSD: Single Shot Multibox Detector. Proc., 14th European Conference on Computer Vision, ECCV 2016, October 8, 2016–October 16, 2016, Amsterdam, Netherlands, Springer Verlag.

10.

Metaxas

D. N.

ASSD: Attentive Single Shot MultiBox Detector. Computer Vision and Image Understanding, Vol. 189, 2019, 102827.

11.

Zhai

Shang

Wang

Dong

DF-SSD: An Improved SSD Object Detection Algorithm Based on DenseNet and Feature Fusion. IEEE Access, Vol. 8, 2020, pp. 24344–24357.

12.

Bewley

Ott

Ramos

Upcroft

, eds. Simple Online and Realtime Tracking. Proc., 23rd IEEE International Conference on Image Processing, ICIP 2016, September 25, 2016–September 28, 2016, Phoenix, AZ, IEEE Computer Society.

13.

Wojke

Bewley

Paulus

, eds. Simple Online and Realtime Tracking with a Deep Association Metric. Proc., 24th IEEE International Conference on Image Processing, ICIP 2017, September 17, 2017–September 20, 2017, Beijing, China, IEEE Computer Society.

14.

Cai

Xiong

Xia

Soatto

MeMOT: Multi-Object Tracking with Memory. Proc., IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, IEEE, New York, NY, 2022, pp. 8080–8090.

15.

Cao

Pang

Weng

Khirodkar

Kitani

Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking. Proc., IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, IEEE, New York, NY, 2023, pp. 9686–9696.

16.

Zhang

Wang

Zeng

Liu

FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking. International Journal of Computer Vision, Vol. 129, No. 11, 2021, pp. 3069–3087.

17.

Duan

Bai

Xie

Huang

Tian

CenterNet: Keypoint Triplets for Object Detection. Proc., IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, South Korea, IEEE, New York, NY, 2019, pp. 6568–6577.

18.

Henriques

J. F.

Caseiro

Martins

Batista

High-Speed Tracking with Kernelized Correlation Filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 37, No. 3, 2015, pp. 583–596.

19.

Bergmann

Meinhardt

Leal-Taixe

, eds. Tracking Without Bells and Whistles. Proc., 17th IEEE/CVF International Conference on Computer Vision, ICCV 2019, October 27, 2019–November 2, 2019, Seoul, Republic of Korea, Institute of Electrical and Electronics Engineers Inc.

20.

Zhou

Koltun

Krahenbuhl

, eds. Tracking Objects as Points. Proc., 16th European Conference on Computer Vision, ECCV 2020, August 23, 2020–August 28, 2020, Glasgow, Springer Science and Business Media Deutschland GmbH.

21.

Quan

Zhang

Tang

Centralized Feature Pyramid for Object Detection. IEEE Transactions on Image Processing, Vol. 32, 2023, pp. 4341–4354.

22.

Tian

Shen

Chen

FCOS: Fully Convolutional One-Stage Object Detection. Proc., IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, South Korea, IEEE, New York, NY, 2019, pp. 9626–9635.

23.

Huang

Zhu

Zeng

Qiu

Shen

SQE: A Self Quality Evaluation Metric for Parameters Optimization in Multi-Object Tracking. Proc., IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, IEEE, New York, NY, 2020, pp. 8303–8311.

24.

Ban

Delorme

Gan

Rus

Alameda-Pineda

TransCenter: Transformers with Dense Representations for Multiple-Object Tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 45, No. 6, 2023, pp. 7820–7835.

25.

Wang

Zheng

Pan

, eds. Multiple Object Tracking with Correlation Learning. Proc., IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2021, June 19, 2021–June 25, 2021, Virtual, Online, USA, IEEE Computer Society.

26.

Xiong

Yang

Wang

Xia

Semi-TCL: Semi-Supervised Track Contrastive Representation Learning. arXiv Preprint arXiv:2107.02396, 2021.

27.

Han

Wang

RelationTrack: Relation-Aware Multiple Object Tracking With Decoupled Representation. IEEE Transactions on Multimedia, Vol. 25, 2023, pp. 2686–2697.

28.

Zheng

Tang

Chen

Zhu

Wang

Improving Multiple Object Tracking with Single Object Tracking. Proc., IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual, IEEE, New York, NY, 2021, pp. 2453–2462.

29.

Stadler

Beyerer

Modelling Ambiguous Assignments for Multi-Person Tracking in Crowds. Proc., IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), Waikoloa, HI, IEEE, New York, NY, 2022, pp. 133–142.

30.

Zhao

Song

Zhao

Gong

Meng

StrongSORT: Make DeepSORT Great Again. IEEE Transactions on Multimedia, Vol. 25, 2023, pp. 8725–8737.

31.

Zhang

Sun

Jiang

Weng

Yuan

Luo

Liu

Wang

, eds. ByteTrack: Multi-object Tracking by Associating Every Detection Box. Proc., Computer Vision – ECCV 2022, Tel Aviv, Israel, Springer Nature Switzerland, Cham, 2022.

32.

Aharon

Orfaig

Bobrovsky

B.-Z.

BoT-SORT: Robust Associations Multi-Pedestrian Tracking. arXiv Preprint arXiv:2206.14651, 2022.

33.

Wang

Y.-H.

Hsieh

J.-W.

Chen

P.-Y.

Chang

M.-C.

H.-H.

, eds. SMILEtrack: SiMIlarity LEarning for Occlusion-Aware Multiple Object Tracking. Proc., 38th AAAI Conference on Artificial Intelligence, AAAI 2024, February 20, 2024–February 27, 2024, Vancouver, BC, Canada, Association for the Advancement of Artificial Intelligence.