Weakly labelled spatial-temporal sweet pepper data: Enabling higher quality detection,segmentation,and tracking.

Multi-object tracking

Multi-object tracking (MOT) is a fundamental task in computer vision and artificial intelligence. It has gained particular interest in the agricultural domain due to the ability to supply farmers with important information about their crop (Halstead et al., 2021; Smitt et al., 2023), particularly the yield and quality of the crop. While, this is one domain in particular that uses MOT, its use is widespread throughout the computer vision community. This includes common tasks such as autonomous driving (Luo et al., 2021), surveillance (Denman et al., 2015), sporting analytics (Cui et al., 2023), and robotics (Ahmadi et al., 2022).

The primary challenge in MOT is accurate association of detected objects across sequential frames, whether from a moving camera or from time-lapse footage. The key to this challenge is the ability to recognize and track objects in spite of varying appearances, occlusions, and distance between images (such as from dropped frames) (Zhang et al., 2021).

In this domain there are generally two forms of MOT: online (Bewley et al., 2016; Du et al., 2023; Zhang et al., 2022; ), and offline (Halstead et al., 2021; Smitt et al., 2023). Online approaches only consider the current and previous frames, while offline approaches consider the full (or partially full) set of frames at once, meaning they can use past, present, and future frames to associate and track objects.

Within these two distinct frameworks exist multiple MOT approaches. A simple and common form of MOT is tracking-via-detection or -segmentation (Du et al., 2023; Halstead et al., 2021; Zhang et al., 2022) which creates and propagates tracklets based on the overlap between detected bounding boxes or segmented regions. However, these approaches often fail when the robot moves considerably or there are large differences between frames, leading to poor tracking due to the lack in overlap between frames. Halstead et al. (2021) aimed to overcome some of this performance degradation by introducing a dynamic radius search region rather than directly using bounding box overlap. In their work they found that even with intermittent frame drop they were better able to track smaller objects in the scene compared to their contemporaries.

Other approaches to MOT include incorporating re-identification (Zhang et al., 2021) in the pipeline. This method is able to aggregate the similarity of objects in the scene which was shown to be beneficial in scenes where the appearance of objects do not change greatly throughout the sequence. Cheng et al. (2021) used spatial-temporal masked attention in their tracking approach, based on detections from Mask2Former (Cheng et al., 2022). This approach achieved promising results on two similar datasets. Heo et al. (2022) employ object token based instance association for MOT, which they call VITA. This is a three step end-to-end MOT approach that detects and assigns tracklets. The method achieves, for the time, state-of-the-art results across a set of datasets. Finally, Ying et al. (2023) incorporate contrastive learning into their MOT approach. This aims to ensure that objects that look similar obtain a similar feature vector, while different objects obtain a vastly different representation through the contrastive loss. Their approach was able to accurately track different objects (pedestrians and animals) in complex scenes and achieve state-of-the-art tracking performance on three different datasets.

While all of these approaches are able to generate state-of-the-art results at the time of publication, in most cases they have similar limitations where horticulture is concerned. Fruit in a glasshouse, like sweet pepper, can appear similar meaning re-identification or contrastive losses do not have enough information to develop unique feature vectors. This results in different objects appearing similar in these approaches. Similarly, particularly for tracking-via-detection and -segmentation approaches if there are dropped frames or large robot movements between frames, overlap between objects is limited. This results in poor performance, although Halstead et al. (2021) did mitigate this limitation somewhat. In this paper, we introduce a new sweet pepper dataset and evaluate different online and offline MOT approaches to gauge how they deal with common agricultural difficulties.

Multi-object tracking datasets

There are a vast number of multi-object tracking datasets, from pedestrian tracking (Leal-Taixé et al., 2015) to vehicle tracking (Geiger et al., 2012). However in the agriculture domain there are very few large tracking datasets, particularly with the additional information typically provided by a robotic platform (i.e. depth images). In general, existing agricultural datasets are small (Saraceni et al., 2024) in number of sequences, or constrained in the data they capture (Kirk et al., 2021). In the BUP-ST20 dataset, we aim to provide large, multifunctional dataset, suitable for tracking and video instance segmentation. Alternatively, the weak labels can be used to train image-based segmentation models to enhance performance.

Due to the rapid growth of the self-driving car industry, tracking datasets are centred around autonomous navigation. For instance, KITTI (Geiger et al., 2012), MOT15 (Leal-Taixé et al., 2015), and MOT16 (Milan et al., 2016) contain image sequences featuring vehicles and pedestrians. These datasets utilize static and moving cameras for data collection and include 50, 22, and 14 sequences, respectively. Recently, large-scale datasets, such as nuScenes (Caesar et al., 2020) with 1000 sequences and the Waymo Open Dataset (Sun et al., 2020) with 1150 sequences, have emerged. They provide more comprehensive tracking data for various driving scenarios. All these datasets provide bounding box annotations and object labels per sequence.

To enable tracking-via-segmentation, also known as multi-object tracking and segmentation (MOTS), Voigtlaender et al. (2019) extend KITTI and MOT16 by providing temporally consistent instance segmentation masks. The DAVIS 2017 (Pont-Tuset et al., 2017) and YouTube-VOS (Xu et al., 2018) datasets contain 150 and 3252 sequences, respectively, providing semantic segmentation masks and object identities across frames. However, these datasets do not support tracking of individual instances. In contrast, YouTube-VIS datasets (Yang et al., 2019, 2021) offer instance segmentation masks with consistent object identities. They include 2883 and 3859 sequences covering 40 categories, although the categories do not include agricultural objects.

In the field of agriculture, object tracking is more challenging due to the homogeneity of the objects in the scene. While some existing MOT datasets contain a substantial number of images, they often consist of fewer longer sequences, which may restrict the diversity of scenes and environmental conditions captured. Additionally, many provide only bounding boxes, limiting their applicability to tasks beyond traditional object tracking. In contrast, our dataset includes both instance segmentation masks and temporal identities, making it well-suited for more complex tasks such as video instance segmentation and tracking-by-segmentation. Saraceni et al. (2024) present a table vineyard dataset, containing 800 RGB images in total for four sequences. Kirk et al. (2021) introduce a strawberry dataset with four sequences, each containing 500 frames captured by an RGB-D camera. These MOT datasets provide bounding box annotations with consistent identities. LettuceMOTS (Hu et al., 2024) consists of 1308 RGB images across 12 sequences. Similarly, De Jong et al. (2022) present APPLE MOTS which initially contained 1673 images, has recently been increased to 2198 images across 12 sequences. These offer fully annotated instance segmentation masks with unique IDs instead of bounding boxes. Our proposed dataset, BUP-ST20, contains 16,240 images for 275 sequences, each with bounding boxes, instance segmentation masks, and temporal identities. To the best of our knowledge, BUP-ST20 is the largest spatial-temporal dataset in agriculture and the first in horticulture. The dataset has weakly labelled training and validation sets, while the evaluation set includes 3810 frames with hand-labelled ground-truth annotations, making it larger than existing datasets of its kind.

Object detection and segmentation

Object detection (Carion et al., 2020; Ren et al., 2015; ) is a crucial task in computer vision. It aims to identify and locate all relevant objects in a scene and describe them via a bounded region. This is particularly important in agriculture as we are only concerned with the crop or fruit of interest and want to ignore background information. Early CNN-based object detection approaches such as Faster R-CNN (Ren et al., 2015) and YOLO (Redmon et al., 2016) revolutionized performance. More recently, we have seen a shift to transformer-based architectures such as DETR (Carion et al., 2020), which has once again improved detection accuracy.

Object segmentation expands beyond simple detection to locate and classify an object on a per-pixel basis. Object segmentation can be further partitioned into: semantic segmentation (Ronneberger et al., 2015), instance-based semantic segmentation (He et al., 2017), and panoptic segmentation (Kirillov et al., 2019b). Semantic segmentation assigns class labels to pixels. This allows for each class to be represented by a single image mask, however it does not distinguish between unique instances. Semantic segmentation methods have recently been superseded by both instance-based semantic segmentation and panoptic segmentation due to their ability to relay more useful information, particularly in agriculture.

Instance-based semantic segmentation expands the pixel labels such that each unique object (instance) is assigned an individual ID (dog 1 is assigned label 1, dog 2 is assigned label 2). A pioneering approach is Mask R-CNN (He et al., 2017), which builds upon Faster R-CNN by incorporating an additional branch to predict instance segmentation masks. Alternative methods, such as SOLO (Wang et al., 2020a), segment objects based on spatial position, dividing the image into grids and assigning masks accordingly. These advancements make instance segmentation more applicable to real-world scenarios, particularly in agriculture, where distinguishing between individual fruits or plants is crucial for tasks such as yield estimation and disease detection.

Panoptic segmentation combines semantic and instance-based segmentation, in that you can have both an instance ID and a semantic map in the same image. Generally, objects considered as ‘things’ (fruit, vehicles, or people) are given a unique instance ID, while ‘stuff’ (soil, sky, or background vegetation) is only given a semantic label. This dual representation enables more comprehensive scene understanding. One of the earliest approaches is Panoptic FPN by Kirillov et al. (2019a), which incorporates a semantic segmentation branch in Mask R-CNN, allowing separate instance and semantic segmentation tasks to be merged. Similarly, Cheng et al. (2020) extends DeepLab (Chen et al., 2018) with an instance segmentation branch. It uses object centre estimates and clustering to segment instances before merging semantic and instance predictions. UPSNet (Xiong et al., 2019) employs a parameter-free panoptic head at the end of the network to improve results. Beyond CNN-based models, vision transformers have recently received wide use. MaX-DeepLab (Wang et al., 2021) integrates CNN and transformer architectures, extending Axial-DeepLab (Wang et al., 2020b) to predict instance masks directly. Lately, Cheng et al. (2022) introduced a transformer-based universal segmentation model, Mask2Former. Its key innovation is the use of masked attention in the transformer decoder to extract localized features. Notably, Mask2Former can also be applied to VIS without requiring architectural modifications (Cheng et al., 2021). VIS extends segmentation to the temporal domain, enabling the consistent tracking and segmentation of objects across frames. MaskTrack R-CNN (Yang et al., 2019) was the first VIS model, building upon Mask R-CNN by introducing a tracking branch to associate instances across a sequence. However, early approaches struggled to consistently maintain identity over long sequences. Transformer models that leverage self-attention mechanisms have been proposed to improve temporal consistency. MinVIS (Huang et al., 2022) utilizes an image-based transformer detector with bipartite matching to enhance object association. In contrast, VITA (Heo et al., 2022), introduced in the MOT section, generates object tokens from spatial features and implicitly incorporates temporal information to effectively capture motion dynamics. More recent VIS models such as IDOL (Wu et al., 2022) have applied contrastive learning to enhance object representations and refine tracking accuracy. Expanding this idea, CTVIS (Ying et al., 2023) which can be used as a MOT approach or for VIS, and was introduced in the MOT section. CTVIS employs contrastive learning and further optimizes object tracking by integrating a memory bank during training, improving object association and overall segmentation performance. To train these VIS models (excluding MinVIS) for agricultural scenarios, we require ID-consistent spatial-temporal datasets, such as BUP-ST20.

As our work creates pseudo-labels for panoptic segmentation we also investigate neural radiance fields (NeRF). NeRF (Mildenhall et al., 2021) is an MLP-based neural network that takes 2D image positions and associated viewing angles as input and outputs the colour and density of the corresponding 3D scene. By leveraging NeRF, we can generate a more coherent segmentation across frames, ensuring that objects retain their identities throughout a scene. Prior works, such as Semantic-NeRF (Zhi et al., 2021) extend NeRF by adding semantic labels into the network, yielding a semantic 3D view of the scene. Further advancing this approach, Kundu et al. (2022) and Fu et al. (2022) integrate semantic and instance-level information to obtain a panoptic scene representation. More recent approaches have improved NeRF’s efficiency by using feature grids as input. For instance, PAg-NeRF by Smitt et al. (2023) employs a permutohedral grid to encode poses and angles, and generates a 3D colour, depth, and panoptic view of the scene. In our study, we utilize PAg-NeRF to create ID-consistent panoptic predictions, ensuring robust performance in challenging agricultural environments.

MOT algorithms

We consider both online and offline MOT methods. Online approaches (Bewley et al., 2016; Du et al., 2023; Zhang et al., 2022) process each image in the sequence as they appear, meaning they only deal with the past and current images. By contrast, offline MOT approaches (Halstead et al., 2021; Smitt et al., 2023) consider the sequence in its entirety. This allows offline methods to aggregate information from the future, generally making offline approaches more accurate. However, their use is often inappropriate in real-world situations such as a robot actively sensing in the field.

We evaluate various offline and online approaches. We employ algorithms based on tracking-via-detection (Halstead et al., 2021), approaches that utilize re-identification (Zhang et al., 2021) and contrastive learning (Ying et al., 2023), spatial-temporal masked attention (Cheng et al., 2021), and object token based instance associate (Heo et al., 2022). Algorithms are deployed on the same dataset with the same inputs, BUP-ST20, and we evaluate them in a consistent manner using common MOT metrics that outline the performance of our chosen approaches on our novel dataset.

Metrics

The chosen MOT based metrics are: higher order tracking accuracy (HOTA) (Luiten et al., 2021), IDF1 (Ristani et al., 2016), and multiple object tracking accuracy (MOTA) (Bernardin et al., 2006). HOTA is the primary metric as it comprehensively evaluates both detection and association accuracy, and has been shown to produce the most complete evaluation of tracking approaches. This metric can be expressed as

H O T A α = D e t A α ⋅ A s s A α ,

(1)

The MOTA metric is defined by

M O T A = 1 − ∑ t F N t + F P + t + I D S W t ∑ t G T t ,

(2)

I D F 1 = 2 ⋅ I D T P 2 ⋅ I D T P + I D F P + I D F N ,

(3)

For the final two ablation studies we use video and image-based average precision metrics (AP) to assess segmentation performance. Video-based AP evaluates the segmentation accuracy across an entire sequence, considering spatial and temporal consistency. This metric is crucial for VIS tasks as it ensures that objects are correctly segmented over time while also maintaining their identities across frames. Similarly, image-based AP measures segmentation performance on a per-image basis, focussing on pixel-wise accuracy. This metric is particularly useful for evaluating the effectiveness of instance segmentation models, where precise object boundaries and classification accuracy are critical. Both metrics are reported at different intersection-over-union (IoU) thresholds, such as AP50 (IoU > 50%) and AP75 (IoU > 75%), providing a detailed performance breakdown.

Novel MOT approaches

In this section we describe the two novel assignment methods for MOT that can be added to tracking-via-detection approaches. First, we describe the standard tracking-via-detection approach that uses intersection-over-union (IoU) to aggregate tracklets over a sequence. Then we describe the dynamic radius (DR) approach of Halstead et al. (2021) as this forms the basis of our two novel dynamic radius-extended (DRE) algorithms, which are presented thereafter. We describe two DRE based approaches based on the presence of a Kalman filter or not. DRE-Sigma relies on the Kalman filter to dynamically update the search position, while DRE-Delta only relies on the detected motion and is needed for methods that do not explicitly include a Kalman filter.

Dynamic radius extended - Sigma

MOT approaches such as ByteTrack include a Kalman filter to predict object movement in the scene. This allows for the bounding box at t − 1 to be shifted to an estimated position at time t based on the covariance matrix. We exploit the internal Kalman filter to extend the DR approach outlined in the previous section; we call this dynamic radius extended - sigma (DRE-S).

The Kalman filter we employ has an 8 × 8 covariance matrix. The first four dimensions relate to the movement of the objects in 4D space multiplied by a heuristic weight (1/10). The next four dimensions are the velocity of the objects in 4D space multiplied by a second heuristic weight (1/80). Both weights come from the publicly released ByteTrack code and are used to maintain fair reproduction with the original work as stated in Zhang et al. (2022). In contrast to the ByteTrack filter, we use the associated movement in all directions as the input to the Kalman filter, where they only used movement in the fourth dimension and a static constant for the third dimension. We empirically found that using the actual movement and velocity of the objects improved results slightly, and thus we use this Kalman filter formulation for all evaluations. Our modified code can be found at, ¹ where we will release the two novel search criteria, the modified offline trackers of Halstead et al. (2018, 2021), and detail how to modify the ByteTrack (Zhang et al., 2022) code to be used with the updated Kalman filter. This Github repository will also outline the location of the repositories for the other MOT approaches used in this work. It will also describe how to download and use the dataset and provide the evaluation results with the best hyper-parameters used for each of the evaluated MOT techniques.

This Kalman filter formulation allows us to directly associate each value in the covariance matrix with a movement or velocity in a given direction. Primarily, we are concerned with movement in the x (left to right) and y (top to bottom) directions. To get the movement in these directions, we extract the diagonal of the covariance matrix and calculate the square root of the elements. The first two values in the resulting vector are the movement in the x and y directions respectively.

In both our novel DRE based matching criterion we create an elliptical shape rather than a circle as used in DR. For DRE-S we compute this ellipse using two distinct methods: maximum and summation. We evaluate these two methods to see which is able to best aggregate tracklets, and present the results in the following section. For the maximum approach we compute the x and y values of the ellipse using the standard radius of the DR method and the variance previously calculated; in this case either the x or y direction can be the major radius. The radii for the ellipse is calculated in the following manner,

Γ ∗ = m a x ( R , σ ∗ )

(4)

where, Γ_∗ is the radius in the x or y; from here on ∗ indicates that the same operation is used to obtain the result for x (e.g. Γ _x ) or y (e.g. Γ _y ). The radius from DR is represented by R, and σ_∗ indicates the variance in the specified dimension. Calculating the ellipse in this manner means that at its default it is the standard DR approach. We note that if we do not use the max function and use σ_∗ directly we can obtain very small search regions due to small σ_∗ values. Empirical investigations found that using a search ellipse defined solely by the variance in each direction obtained poor results.

The search ellipse can also be formed using a summation operation. In this case the two radii are calculated such that

Γ ∗ = R + σ ∗

(5)

where once again the Γ_∗ is the resulting radius in the two directions and R and σ_∗ are the same as in the maximum calculation. While some of our σ values are small, this approach generally ensures that the ellipse will be larger than that obtained by the DR approach.

Finally, the changed search region necessitates a different method to check if a detected object is within the search region of existing tracklets. We use the standard ellipse calculation for this with the following modifications:

D = ( C b , x − C a , x ) 2 Γ x ⋅ α + ( C b , y − C a , y ) 2 Γ y ⋅ α ,

(6)

where D is the calculated distance between a new centre (C_b,*) to an existing tracklet centre (C_a,*). The size of the ellipse in each direction (x, y) is indicated by the Γ variable. We also include a weighting factor (α) which allows us to empirically scale the search region as a hyper-parameter.

This ellipse function now indicates if an object is within the search region of an existing tracklet. If the value of D is in the range [0, 1] then the new object is within the ellipse, if it is greater than one then it is outside the search range. Only objects within the field of search are considered matches, and the object with the smallest D value is added to the tracklet.

Dynamic radius extended - Delta

Our second novel matching criterion is designed for MOT algorithms that do not have a Kalman filter, but still aggregate tracklets over time. Dynamic radius extended delta (DRE-D) is a tracking-via-detection addition that uses a moving average of the distance between tracked objects in images to create an elliptical search region. This allows for, depending on movement prediction, larger search areas to help maintaining tracklets when there are large differences between object detections (either from frame skips or from large robot movements).

While DRE-S takes the σ_∗ values from the covariance matrix as movement parameters, DRE-D takes a different approach. At initialization, the two ellipse radii (d_x,0 and d_y,0) are calculated as

d * , 0 = a b s ( ϕ i − ϕ j )

(7)

where, d_∗, is a vector that stores the movement distances in the x and then the y directions for the previous N timesteps, empirically set to 3. Then the absolute difference for x is calculated by the left and right most positions of the bounding box (ϕ_∗), and y is calculated as the top and bottom co-ordinates of the bounding box (ϕ_∗). This gives us a relatively large search ellipse at initialization.

To match new objects to existing tracklets we employ three different criterion for DRE-D: default, maximum, and summation. The default criteria is calculated using

Γ ∗ = m e a n ( d ∗ )

(8)

where Γ_∗ once again indicates the elliptical radius in the x and y directions, and mean(.) is the mean movement between images recorded in the tracklet. Both maximum (equation (4)) and summation (equation (5)) are calculated in a similar manner to DRE-S.

To match a new detection to the tracklet we calculate if it falls within the ellipse by using the same equation as that in the DRE-S method (equation (6)), and values of Γ_∗ derived using the default, maximum, and summation criterion. Similar to DRE-S, if D is in range [0, 1] this indicates that an object is close enough to the tracklet to be matched. Finally, once an object has been attached to a tracklet we update d_∗ in both directions, using a first-in-first-out approach to ensure only N distances are retained (N = 3 by default).

The offline tracking approach

Finally, we modify the tracking-via-detection approach of Halstead et al. (2021) to suit our purpose. In Halstead et al. (2021) the authors incorporated a kill zone and an initilisation zone, however we found these zones unnecessary. This was because the Mask2Former predictions were far more robust when compared to the Mask R-CNN (He et al., 2017) detections used in Halstead et al. (2021, 2018). Removing these zones allows us to track objects more accurately throughout the scene.

This offline tracker forms the basis for our DRE matching criterions as it showcases the ability to better track objects in the presence of large frame skips. Our GitHub repository includes our version of the code, however, in our evaluations we refer to this method using the original citations.

Multi-object tracking on BUP-ST20

The primary evaluation compares state-of-the-art tracking algorithms and our two novel matching criterion on BUP-ST20. We chose 12 MOT algorithms for this evaluation based on their performance on various tracking datasets. Ordered by the year of their release we include: Bewley et al. (2016); Wojke et al. (2017); Halstead et al. (2018); Zhang et al. (2021); Halstead et al. (2021); Cheng et al. (2021); Zhang et al. (2022); Heo et al. (2022); Du et al. (2023); Smitt et al. (2023); Cao et al. (2023); Ying et al. (2023). These selected MOT algorithms are used to validate the novel dataset as a challenging task due to the unconstrained shapes and occlusions of the sweet peppers.

Quantitative results

Table 3 outlines the performance of each of the selected MOT approaches on BUP-ST20. To make performance metrics easier to understand the best-performing systems in both the offline and online categories are shown in bold. As described in the metrics section we evaluate performance based on HOTA, MOTA, and IDF1 where each provide different statistics.

OPEN IN VIEWER

Table 3.

Table of the results on BUP20-track – order these based on the year they were released.

Method	Matching	HOTA	MOTA	IDF1
Online
Bewley et al. (2016) SORT	IoU	78.25	61.34	77.49
Wojke et al. (2017) DeepSort	IoU	74.47	48.15	69.49
Zhang et al. (2021) FairMOT	IoU+Re-ID	68.52	53.19	67.78
Zhang et al. (2022) ByteTrack	IoU	81.35	63.74	80.25
	DR	80.09	58.94	78.08
	DRE-D	80.40	59.91	78.54
	DRE-S	80.48	60.29	78.87
Du et al. (2023) StrongSort	IoU	76.35	61.47	77.40
Cao et al. (2023) OCSORT	IoU	80.10	65.67	80.28
Ying et al. (2023) CTVIS	Re-ID	79.81	51.12	76.86
Offline
Halstead et al. (2018)	IoU	82.12	68.54	80.45
Halstead et al. (2021)	DR	80.93	68.17	79.15
	DRE-D	79.85	66.45	78.07
Cheng et al. (2021) M2F-VIS	IoU	68.08	32.40	64.55
Heo et al. (2022) VITA	IoU	62.35	25.89	58.03
Smitt et al. (2023) PAg-NeRF	Learnt	81.22	69.69	75.66

Best performances are highlighted in bold.

As expected and as outlined in Table 3 the best-performing technique is an offline algorithm. This is due to the ability to consider past, present and future images, resulting in more robust tracking. For online approaches we see the standard ByteTrack (Zhang et al., 2022) algorithm achieves the best HOTA score, which is the principle tracking metric. With a score of 81.35, it beats the next best online system by an absolute margin of 0.87, however, in general all approaches achieve good results. Interestingly, FairMOT (Zhang et al., 2021), which includes re-identification of detected objects achieves the worst HOTA score at 68.52. We attribute this to the fact that even though sweet pepper are highly deformable, their appearance for re-identification is not sufficiently unique for one-to-one matching. For the other metrics, IDF1 and MOTA, OCSort (Cao et al., 2023) achieves the best performance with only a small reduction in HOTA.

For the offline approaches we see that he modified offline tracking approach (Halstead et al., 2018) with IoU achieves the best results with a HOTA score of 82.12. Compared to PAg-NeRF, which we used to create BUP-ST20’s pseudo-labels, we see an improvement of 0.9, however PAg-NeRF achieves the best MOTA score indicating that it has less missed tracks. Unfortunately, for M2F-VIS and VITA we see a considerable drop off in performance despite being offline methods. We attribute this performance drop to the challenging nature of BUP-ST20 which comprises significant of natural occlusions (other sweet pepper, leaves, and stems). These challenges are natural within a horticultural dataset, and as a result these methods frequently miss detections or detect false positives. Furthermore, the homogeneity of the sweet peppers leads to inaccurate object associations, further hampering performance.

Overall, our novel matching criterion, while achieving close to best performance, are in general not able to improve upon IoU matching. For the offline DRE based approaches we saw that the summation approach was best for DRE-D and the maximum was best for DRE-S. For the offline DRE-D approach, the best-performing ellipse creator was obtained by using the maximum technique, however, there was only a small difference when compared to the summation technique. As such we recommend selecting the summation approach for all of our novel DRE based matching criteria, and for all ablation studies using the DRE criteria we have used this ellipse creation technique. This is in contrast to Halstead et al. (2021), who found that DR performed better than IoU for their application. We attribute some of this performance degradation to the described benefits of the DR approach, primarily, that DR is better for small objects. As we have previously stated Mask2Former has a known issue in that it does not detect small objects very accurately and in BUP-ST20 there are a number of small objects in the scene. This is particularly evident in the right most image of Figure 7 where the ground-truth object is not detected. Missed detections of this type limit tracking ability as we do not detect all relevant objects that would otherwise improve tracker performance (small objects). In future, to achieve better results, a more accurate detection model which better detects small objects could in turn improve the performance of the DR and DRE matching criteria.OPEN IN VIEWER

Figure 7.

Various examples of the challenges of the dataset; red bounding boxes indicate the detections and green bounding boxes represent the ground truth. In images (a) and (b) only the detections are displayed as there are no ground truth sweet pepper in the current row. (a) An example of detections prior to depth filtering, where all objects in the scene are more than 1.2 m from the camera and should not be detected. (b) Demonstrates the same image as in (a) after depth filtering, however, only a single sweet pepper in the deeper rows is detected. In the middle left of the image there is a single red incorrect bounding box. (c) Shows where leaves or peduncles are detected as sweet pepper. Also in the middle top there is an example of the detection routine failing due to occlusion. Finally, (d) is an example of where Mask2Former fails with small objects. There are three green bounding boxes (of small objects) which are completely missed.

Overall, the best approach for accuracy is the offline tracking algorithm based on Halstead et al. (2018), achieving best performance in HOTA and IDF1 and comparable performance for MOTA. However, offline trackers are not always appropriate, particularly for online robotic platforms. If an online tracker is required, OCSort or ByteTrack offer the best tradeoff between HOTA, MOTA, and IDF1.

Ablation I: Skipping frames

In the first ablation study we evaluate the considered state-of-the-art MOT algorithms when we introduce random frame skipping and low frame rates in the BUP-ST20 dataset. This simulates real-world situations where edge devices such as robots have limitations in processing capacity or speed inconsistencies. The random frame-skipping experiment is based on the Poisson distribution with a λ = 1, in all evaluations we use the same set of randomly distributed frames ensuring fair comparison between MOT techniques. Using λ = 1 ensures that the distribution of the frames is centred around having either 0 or 1 skips, with a smaller probability of having larger skips. This is more reflective of what is witnessed in real-world situations where there is a majority of no or very small skips. For the low frame rate evaluations we decimate the frames by 2, 5, and 10. As the standard camera used in the BUP20 dataset was captured at 15 frames per second we now have approximately: 7, 3, and 1 frames per second.

For both the random skipping and low frame rate we anticipate that the IoU based matching criterion will perform worse than the other methods. This is due to the strict matching thresholds, due to the distance between objects in the images. In Table 4 we only report the HOTA metric as this best describes the tracking performance of the selected approaches. For all of the MOT techniques, we do not perform a comprehensive hyper-parameter sweep for this ablation study. Instead we select approximately five of the best-performing hyper-parameter sets from the results produced when evaluating for Table 3 and only evaluate on these. Based on these hyper-parameters we only display the best-performing approach. In the released GitHub repository we will clearly display the best-performing hyper-parameters for each method.

OPEN IN VIEWER

Table 4.

Random skipping frames and low frame rate results with the HOTA metric reported for each method.

Method	Matching	λ1	2f	5f	10f
Online
Zhang et al. (2022) ByteTrack	IoU	73.20	76.68	39.91	28.35
	DR	74.84	77.93	55.06	31.54
	DRE-S	75.41	79.57	45.47	30.42
	DRE-D	73.03	77.94	51.38	31.02
Zhang et al. (2021) FairMOT	IoU+Re-ID	38.08	47.61	18.55	27.43
Cao et al. (2023) OCSort	IoU	67.41	73.05	22.66	29.17
Offline
Halstead et al. (2018)	IoU	70.01	77.88	43.27	24.94
Halstead et al. (2021)	DR	70.54	75.46	55.28	30.26
Halstead et al. (2021)	DRE-D	73.67	77.16	65.52	49.89

Finally, we also only select a subset of MOT techniques for this ablation study. We select ByteTrack (Zhang et al., 2022) (with IoU, DR, DRE-S, and DRE-D) as this best represents online approaches that use the state-of-the-art Kalman filter. As the best-performing approach for MOTA and IDF1 in Table 3 we include OCSort (Cao et al., 2023), and finally due to being a considerably different re-identification based approach we include FairMOT (Zhang et al., 2021) for robustness. For the offline approaches we select Halstead et al. (2018) for the IoU matching and Halstead et al. (2021) for DR and DRE-D. We do not further evaluate PAg-NeRF in this evaluation as it is has a learnt matching criteria and would therefore need to be retrained and optimized on this specific data. This would mean that the assumption of ‘random skipping’ is removed as it would be trained specifically for these sequences.

Table 4 once again splits the evaluated MOT techniques into offline and online, where the bold numbers indicate the best-performing approach for each column. Starting with the online approaches we see that for random frame skips, the introduction of the dynamic radius sigma approach to the already accurate Kalman filter of ByteTrack performs the strongest. This is able to achieve an absolute performance improvement of 2.2 over the standard IoU based approach. This is in contrast to the results seen in Table 3 where the IoU based ByteTrack performed the best. This outlines the benefit of incorporating movement based on the Kalman filter with an elliptical shape when there is potential for frame skipping. In this, and the reduced frame rate experiment, we used summation approach for the ellipse formation. We attribute this to the ability to provide richer movement context in the searching radius. For the reduced frame rate experiment we see that, in general, the standard dynamic radius approach outperforms the others. This is particularly noticeable in the 5f evaluation where it scores an absolute improvement of approximately 10 over the dynamic radius sigma approach. However, for the 10f decimation method we see that the DR and DRE-D approaches are commensurate. For both FairMOT and OCSort we see considerable degradation in performance when either frame skipping or low frame rates are present. Overall, where intermittent frame skipping is possible it is clear that selecting the DRE-S method is beneficial, although, for consistently low frame rate issues it is best to consider the standard dynamic radius approach.

The three offline approaches tell a slightly different story to the online approaches which we attribute to the possibility of looking into the future as well as the past. For these approaches, due to the fact that the Kalman filter is not available, the DRE-D approach achieves either considerable performance boosts (λ1, 5f, and 10f) or commensurate performance (2f). Once again, for the DRE-D approach the summation method is used for shape creation. For the 5f and 10f the selection of DRE-D is clear; we are able to achieve an absolute improvement of approximately 10 (5f) and 20 (10f), outlining its ability to better aggregate tracklets when the frame rate is low. The HOTA scores for these two low frame rate evaluations (5f and 10f) are the best across both the offline and online techniques.

For the online approaches no single approach offers best performance across all scenarios. Where frame skipping may occur the DRE-S approach is beneficial with the summation ellipse calculation. In low frame rate situations there is actually a degradation in performance when selecting this approach and in fact the standard DR approach is more appropriate. For the offline tracking the performance gains of the DRE-D method are considerable, both for frame skipping and low frame rates. This makes it, along with the summation approach, the clear criterion choice.

From a qualitative standpoint there are obvious situations where our novel criteria succeeds and fails. The most common scenarios are outlined in Figure 8, which are consistent across both random frame skips and low frame rates.OPEN IN VIEWER

Figure 8(a) is an example of where the IoU method is able to outperform the DRE methods. In this example, the detection from f1 (which can be shifted depending on the base technique) is used at f2. The detections at f2 are then compared to the f1 based detections, in this case the f2 detections are erroneously small (do not detect the entire object). This means that even though there is a small overlap between the f1 and f2 bounding boxes, it is enough to be considered a match, and the tracklet is updated. Unfortunately, for the DRE methods, even though the ellipse is shifted based on the movement parameters, the centre of the new object in f2 is outside of the search radius and thus a match is not found. This trend could be somewhat rectified with an improved object detection routine that can allow for occlusion. It should be noted here that in f3 due to the considerable frame skip in the scene the IoU based method completely fails and the tracklet is no longer retained.

The novel DRE criterion shows considerable gains over the IoU based matching when there are occlusions from other sweet pepper. In Figure 8(b) we see a clear example of this. There are two sweet pepper close together and the IoU matching simply attempts to aggregate the object with the highest overlap, which is not always beneficial. The DRE method allows for this by shifting the ellipse and searching based on the Euclidean distance for objects that are within the search radius. Due to this internal shifting and search radius the DRE method is also able to track the object as it shifts greater distances: f2 → f3.

Ablation II: Video instance segmentation

This ablation study aims to show the applicability of our large-scale spatial-temporal dataset for VIS models. Existing datasets in the agricultural domain are often limited in size and lack comprehensive annotations, making it challenging to develop robust segmentation models. In contrast, our dataset offers a vast number of annotations, including bounding boxes, segmentation masks, and consistent IDs across frames, which makes it well-suited for VIS tasks.

We evaluate state-of-the-art VIS models including CTVIS Ying et al. (2023), Mask2Former-VIS Cheng et al. (2021) and VITA Heo et al. (2022). These models are chosen due to their high performance on benchmark datasets such as YouTube-VIS 2021 (Yang et al., 2021) and OVIS (Qi et al., 2022). We use 8 NVIDIA A100 GPUs with 80 GB of memory each, and follow baseline settings for each model. We use Swin-L as the backbone, a frame resolution of 960 × 540, and a learning rate of 0.00,005 for all models. CTVIS, Mask2Former-VIS, and VITA are trained on BUP-ST20 for 8k, 11k, and 14k iterations respectively, with a batch size of 8. Pre-trained weights are used for each model.

In this ablation study we use the video-based average precision (AP) metric. While tracking metrics (HOTA, MOTA, IDF1) focus on the accuracy of object associations across frames, AP provides an assessment of the segmentation accuracy of each tracklet. As the primary goal is to evaluate how effective the models segment objects over time rather than just track their identities, AP serves as a more suitable metric here. In addition to the VIS performance comparison, we also calculate AP scores for the standard ByteTrack approach and Halstead et al. (2018), which have the two best HOTA scores in Table 3. While video instance segmentation models provide segmentation masks, this MOT evaluation relies on bounding boxes. To enable a direct evaluation, we extract bounding boxes from the predicted segmentation masks and compute the AP accordingly. This allows us to assess the advantages of video-based segmentation models while maintaining comparability with existing MOT methods.

Overall, from Table 5 we see that CTVIS (Ying et al., 2023) is consistently able to outperform other approaches in all metrics by a considerable margin. This performance can be attributed to the use of contrastive learning and a memory bank during training, which enables enhanced object representation and segmentation accuracy. However, as shown in Figure 9, CTVIS still encounters object association errors. In the main evaluation on BUP-ST20 we saw that M2F-VIS and VITA performed commensurate with the best-performing approaches, yet this is not the case here. For VITA this is particularly evident as it does not explicitly incorporate temporal information, limiting its ability to handle complex horticultural scenarios. In both of these approaches there are missed/incorrect detections and a number of object association errors caused by a myriad of reasons including insufficient object representations.OPEN IN VIEWER

The final two considered approaches, ByteTrack and Halstead et al. (2018), were the best-performing algorithms in the main evaluation. In Table 5 we see considerably poorer performance compared to CTVIS. The primary reason for Halstead et al. (2018) failing is that it employs a high IoU threshold value (0.9) meaning it only focuses on high-quality detections. This strict association causes poor AP results. While ByteTrack has multiple association stages it still has a number of false positives. This has a negative impact on overall detection accuracy, reducing performance.

OPEN IN VIEWER

Table 5.

AP results on BUP-ST20 evaluation set.

	AP b	AP b 50	AP b 75	AP m	AP m 50	AP m 75
Ying et al. (2023) CTVIS	56.8	79.8	61.4	56.7	81.0	61.8
Cheng et al. (2021) M2F-VIS	29.4	53.7	27.2	32.9	62.0	30.1
Heo et al. (2022) VITA	23.6	43.9	21.9	31.3	58.3	29.7
Zhang et al. (2022) ByteTrack	21.6	36.7	22.5	—	—	—
Halstead et al. (2018)	30.4	47.3	32.5	—	—	—

Based on this ablation study, it is clear that for video instance segmentation, CTVIS offers the most robust performance. This, in conjunction with the results of Table 3, highlights CTVIS as an effective tool for both MOT and VIS. These results also prove that BUP-ST20 as a weakly labelled spatial-temporal dataset can be utilized for video instance segmentation in the agricultural domain. Furthermore, beyond traditional tracking-by-detection methods, BUP-ST20 provides segmentation masks, enabling the application of tracking-by-segmentation approaches. This capability allows for more precise object tracking, as it leverages pixel-level segmentation.

Ablation III: Instance-based semantic segmentation improvement

In Ablation III, we leverage the pseudo-labels in BUP-ST20 to explore whether they can improve instance-based semantic segmentation. Generally, for instance-based semantic segmentation hand-labelled masks are used to train segmentation models. Creating these annotations is a laborious process however, and employing weak labels instead of fully annotated data can substantially reduce annotation time while maintaining or even enhancing performance. The fact that our novel dataset has 7914 weakly labelled images compared to just 124 images in BUP20, means that we have almost two orders of magnitude more images to train the models on.

We explore the use of three techniques to exploit these pseudo-labels. First, we train a Mask2Former model using the filtered BUP20 dataset, with 124 annotated images, we refer to this as the baseline. We then train a Mask2Former model using BUP-ST20, with 7914 weakly labelled images, to directly compare the benefits of including the weak labels. Finally, we train CTVIS on the video sequences obtained from the weak labels and then make use of the underlying Mask2Former model; CTVIS consists of a Mask2Former model with an additional re-identification component.

We evaluate all the models on the hand-labelled evaluation set of BUP20. To maintain evaluation conformity, we depth filter all masks and reject any masks further than 1.2 m from the camera. The metric used in this ablation study is image-based average precision (AP). We break this metric down into multiple values: AP, AP50, AP75, APs (small), APm (medium), and APl (large); APs is [0², 72²], APm is [72², 144²], and APl is [144², :] pixels.

The results, in Table 6, highlight the benefits that weak labels provide for instance-based semantic segmentation. When using the weak labels to train Mask2Former (BUPST20-M2F vs BUP20 filtered-M2F) we see that, in general, there is superior performance. However, this performance improvement is still relatively small, improving the AP by 2.4 points for the single class setting and by just 0.3 points for the multi-class setting. One area where there is obvious improvement across both the Single Class and Multi Class settings is with small objects. We attribute this to the fact that the weakly labelled dataset has more small objects than the standard BUP20 dataset, meaning that during training we see more of these objects creating a model more robust to this aspect. The greatest benefits for using weak labels for the task of instance-based semantic segmentation occur when we train the system in a video-instance based setting.

OPEN IN VIEWER

Table 6.

Instance segmentation for Mask2Former models on the BUP20 filtered evaluation set.

	AP	AP50	AP75	APs	APm	APl
Single class
BUP20 filtered-M2F	56.7	79.2	62.1	41.5	88.8	100
BUPST20-M2F	59.1	81.6	65.5	46.1	87.6	100
BUPST20-CTVIS	62.1	82.3	67.3	48.9	90.2	100
Multi class
BUP20 filtered-M2F	45.5	59.0	50.2	32.5	70.7	100
BUPST20-M2F	45.8	60.8	50.2	33.9	70.7	100
BUPST20-CTVIS	58.3	72.9	63.8	38.9	83.9	100

For the video-based segmentation approach, we observe that CTVIS outperforms image-based segmentation methods in both single- and multi-class scenarios. Specifically, it surpasses the baseline in overall AP for single and multi-class tasks by 5.4 and 12.8 points, respectively. This considerable performance improvement occurs across all metrics and we attribute this to the ability of CTVIS to extract spatio-temporal consistent features through contrastive learning to perform re-identification.

The results demonstrate that BUP-ST20 can enhance image-based instance segmentation performance, particularly for small objects, which is crucial in robotic based agricultural applications. Rather than using fully annotated small datasets, exploiting a large-scale weakly labelled dataset improves the model’s robustness while reducing annotation effort. Beyond image-based segmentation, BUP-ST20 is well-suited for more complex tasks such as VIS and MOT. These tasks inherently require a deeper understanding of spatial-temporal consistency and object association, making them more challenging than image-based segmentation. The large-scale and structured nature of BUP-ST20 provides an opportunity to develop and evaluate models specifically designed for these complex scenarios.

Overall, the results displayed in the primary and ablation studies show that our novel BUP-ST20 dataset is a valuable resource to the agricultural domain. It contains considerably more hand-labelled evaluation images that can be used for MOT, VIS, and instance-based semantic segmentation. It also includes a large training set composed entirely of pseudo-labels, which can be highly valuable for training more accurate models across a multitude of tasks. We show here that while the dataset is a valuable addition to this domain, there is still considerable work that needs to be completed before phenotypic traits can be accurately recognized in agricultural scenes.