Multiview Bird’s-Eye View Fusion for Enhanced Three-Dimensional Object Detection in Vehicle

Abstract

Three-dimensional (3D) object detection plays a critical role in connected and autonomous vehicles (CAVs) because it enables precise localization and classification of surrounding obstacles by estimating their spatial properties from two-dimensional (2D) images. This capability effectively addresses the limitation of traditional 2D detectors, which lack depth information. However, image-based 3D detection remains challenging in complex scenarios, particularly under occlusion or limited fields of view, such as at intersections or on curved roads. To address these issues, vehicle-to-infrastructure (V2I) cooperative perception frameworks leverage vehicle-to-everything (V2X) communication to transmit sensor data from roadside units to CAVs, enabling multiview feature fusion from both vehicle-mounted and infrastructure-side cameras. In this paper, a V2IFormer, a novel V2I cooperative 3D object detection framework based on bird’s-eye view (BEV) representations is proposed. Instead of transmitting raw images or high-level detection outputs, the V2IFormer transmits BEV features to reduce bandwidth while preserving essential spatial information. On the infrastructure side, a HeightNet module with a linearly-increasing discretization strategy is introduced to predict adaptive height distributions, improving long-range depth perception. On the vehicle side, multiview image features are lifted into the BEV space using depth distribution prediction, enhancing depth accuracy in close-range regions. A deformable mutual-attention module is further used to adaptively fuse BEV features from both sides, selectively focusing on informative regions while suppressing irrelevant content. Extensive experiments on the dataset for infrastructure–vehicle cooperative perception via vehicle-to-everything communication (DAIR-V2X) benchmark demonstrate that V2IFormer achieves state-of-the-art performance, with a mean average precision of 55.64 for 3D detection and 59.63 for BEV detection, significantly outperforming existing fusion methods.

Keywords

vehicle–infrastructure cooperative bird’s-eye view connected and autonomous vehicles three-dimensional object detection

Introduction

Accurate three-dimensional (3D) object detection is a critical component in autonomous driving. It enables precise localization and classification of surrounding entities, including vehicles, pedestrians, and cyclists. Unlike two-dimensional (2D) detection, which only identifies objects on a plane, 3D detection estimates their position, size, and orientation from 2D images, providing a more comprehensive understanding of the driving environment. However, monocular vehicle-mounted cameras face inherent limitations because of their low installation height, which leads to frequent occlusion and a restricted field of view. According to Waymo’s safety report, occlusions account for 25% of major accidents involving autonomous vehicles ( 1 ), highlighting the need for advanced perception systems.

Roadside infrastructure, equipped with elevated cameras, offers a complementary perspective with a wider field of view, reducing occlusion and extending detection range. Existing 3D detection approaches can be broadly divided into Light Detection and Ranging (LiDAR)-based and camera-based methods. LiDAR offers high geometric precision; its high cost and limited performance in detecting small or distant objects hinder large-scale deployment. In contrast, camera-based systems, especially monocular or multiview configurations, offer a scalable and cost-effective solution, as evidenced by Tesla’s vision-only Full Self-Driving (FSD) pipeline ( 2 ). Vision-based roadside systems are preferred over LiDAR because of lower costs and seamless integration with urban infrastructure, such as traffic poles ( 3 , 4 ). However, roadside monocular systems face inherent depth ambiguity, particularly over long distances. Recent methods such as BEVHeight (bird's-eye-view height-based 3D detection) ( 5 ) and MonoGAE (monocular geometry-aware estimation) ( 6 ) improve depth estimation by leveraging ground-relative constraints, but still face challenges in detecting distant and occluded objects. Vehicle-to-infrastructure (V2I) cooperative perception offers a promising solution by transmitting high-viewpoint sensor data to connected and autonomous vehicles (CAVs) in real time.

The intermediate fusion strategy integrates sensor data from both onboard vehicle sensors and roadside infrastructure into a unified bird’s-eye view (BEV) representation. This design enables extended perception coverage, mitigates occlusion effects, and ensures reliable performance under limited communication bandwidth. Because of their ability to encode rich spatial context within a globally consistent coordinate system, BEV representations have become a widely adopted paradigm in camera-based autonomous driving systems. A fundamental challenge in BEV-based perception lies in transforming perspective-view image features into accurate and dense BEV features. Recent advances, such as BEVDepth ( 7 ) and BEVFormer ( 8 ), address this by learning geometric projections and spatiotemporal associations, respectively. These frameworks have significantly advanced 3D scene understanding and have been successfully applied to core autonomous driving tasks, including 3D object detection ( 9 ), semantic segmentation ( 10 ), and map construction ( 11 , 12 ). In addition, BEV-based representations provide a common spatial foundation for V2I cooperative perception ( 13 , 14 ). Despite these advancements, existing V2I perception methods face notable limitations when applied to real-world multi-agent collaboration. For instance, BEVFusion ( 7 ) fuses multi-modal sensor inputs (e.g., camera and LiDAR) within the BEV space and achieves strong detection performance. However, its substantial computational cost and bandwidth demand hinder its suitability for distributed V2I scenarios. Meanwhile, a monocular-based 3D detection model (ImVoxelNet) ( 15 ) lifts 2D features into voxel space using predicted depth. While computationally efficient, its performance deteriorates in long-range detection and under occlusions because of limited depth precision. Other cooperative frameworks, such as vehicle-to-everything vision transformer (V2X-ViT) ( 16 ) and vehicle–infrastructure multiview interaction (VIMI) ( 14 ), adopt attention-based or concatenation-based fusion mechanisms. Although these strategies offer initial improvements, they struggle to resolve the spatial and semantic inconsistencies inherent in cross-view data. Global attention mechanisms, such as cross-attention, assign equal importance to all spatial locations, leading to inefficiencies in filtering irrelevant information and amplifying sensor noise. Similarly, naive feature concatenation lacks the capacity to adaptively align heterogeneous inputs, resulting in poor geometric consistency and suboptimal fusion.

V2I cooperative perception has emerged as a promising paradigm for enhancing environmental awareness. By enabling the real time sharing of high-elevation sensor data from infrastructure to CAVs, V2I systems can significantly extend perception capabilities beyond the inherent limitations of onboard sensors. Enabled by 5G and V2X communication technologies ( 17 ), these systems support multiview data fusion to improve situational awareness. However, a fundamental trade-off exists between detection accuracy and communication bandwidth. Transmitting raw sensor data retains rich detail but demands high bandwidth; transmitting only detection results reduces bandwidth but sacrifices spatial fidelity ( 13 , 14 , 16 ). To balance this trade-off, many recent methods adopt intermediate feature representations for transmission, particularly BEV features, which provide a unified spatial framework for multisensor fusion. BEV-based approaches offer strong spatial context and support efficient fusion of vehicle-side and infrastructure-side features. However, transforming image-view features into accurate BEV features remains a major challenge, especially under monocular setups.

To address the challenges of limited bandwidth and multi-source feature fusion in V2I cooperative 3D object detection, V2IFormer is proposed, a novel framework based on BEV representations. Instead of transmitting raw images or high-level predictions, V2IFormer transmits intermediate BEV features to reduce bandwidth while preserving spatial detail. On the infrastructure side, a HeightNet module with a linearly-increasing discretization (LID) strategy is introduced to predict adaptive height distributions and improve long-range depth perception. On the vehicle side, image features from multiple views are lifted into the BEV space using depth distribution prediction based on the lift-splat-shoot (LSS) framework ( 18 ), which enhances accurate depth perception at close range. A deformable mutual-attention module then adaptively aligns and fuses dual-view BEV features by focusing on informative regions and suppressing irrelevant noise, ensuring robustness under occlusion and varying viewpoints. Experiments on the dataset for infrastructure–vehicle cooperative perception via vehicle-to-everything communication (DAIR-V2X) benchmark demonstrate that V2IFormer achieves state-of-the-art performance in complex driving scenarios. The main contributions of this paper are summarized as follows.

V2IFormer is proposed, a novel V2I cooperative 3D object detection framework that transmits BEV features to balance perception accuracy and bandwidth efficiency. By integrating vehicle-mounted and infrastructure-side monocular data, it enhances robustness under occlusion and limited visibility.

Specialized BEV generation modules are designed on the infrastructure and vehicle sides. On the infrastructure side, a HeightNet module is introduced with an LID strategy to predict adaptive height distributions, enhancing long-range depth perception. On the vehicle side, image features from multiple views are lifted into BEV space using depth distribution prediction based on the LSS framework, which improves depth accuracy in close-range regions.

A deformable mutual-attention module is employed to adaptively fuse dual-source BEV features by selectively focusing on informative regions while suppressing irrelevant noise. The effectiveness of V2IFormer is validated on the DAIR-V2X benchmark, where it achieves state-of-the-art 3D detection performance in complex driving scenarios.

The structure of this paper is as follows. The Literature Review surveys the state-of-the-art in 3D object detection, collaborative perception, and BEV scene understanding, outlining recent advancements and identifying current limitations. The Methodology details the proposed V2IFormer architecture, which features dual-side BEV generation and a deformable mutual-attention fusion mechanism. Experimental Evaluation reports the experimental evaluation on the DAIR-V2X benchmark, highlighting the method’s accuracy and robustness. Finally, Conclusion summarizes the findings and explores directions for future research.

Literature Review

Camera-Based 3D Object Detection

Current 3D object detection techniques are classified into two main categories based on the type of sensing modality: (1) image-based methods; and (2) LiDAR-based point cloud methods. LiDAR offers high detection accuracy and precise spatial measurements; however, its high cost, bulky hardware, and limited effectiveness in detecting small or distant objects hinder its widespread deployment in real-world applications. In contrast, camera-based 3D object detection estimates 3D bounding boxes using image data from monocular cameras. This approach is increasingly favored over LiDAR for several compelling reasons. First, cameras are significantly more cost-effective and widely available, making them suitable for large-scale deployment in consumer vehicles and roadside infrastructure ( 19 , 20 ). For example, Tesla’s FSD system demonstrates the viability of a vision-only solution by achieving advanced driver assistance capabilities using a suite of eight cameras ( 2 ). Second, cameras capture rich visual cues, including color, texture, and contextual information, that are essential for accurate object classification and comprehensive scene understanding, especially in complex environments ( 8 ). Third, recent advances in deep learning, particularly in monocular depth estimation and neural network architectures, have significantly improved the performance of image-based 3D detection systems. Tesla’s FSD performance in urban scenarios with minimal human intervention highlights this progress ( 21 ). Despite challenges such as depth ambiguity, occlusion, and sensitivity to environmental conditions, camera-based 3D object detection strikes a favorable balance between cost, scalability, semantic richness, and system integration feasibility. These advantages make it a promising solution for autonomous driving and infrastructure-assisted perception and broader intelligent transportation applications.

Camera-based 3D object detection methods can be categorized into vehicle-mounted and infrastructure-based approaches. Early works extend 2D detectors to jointly predict 2D and 3D object attributes. Notable examples include fully convolutional one-stage 3D object detection (FCOS3D) ( 22 ), detection transformer for 3D object detection (DETR3D) ( 23 ), and the position embedding transformation for multi-view 3D object detection (PETR) series methods ( 24 , 25 ), which incorporate geometric priors or temporal context to enhance spatial reasoning. Alternatively, some methods operate directly in 3D space by lifting 2D image features into structured 3D representations. For instance, orthographic feature transform (OFT) ( 26 ) introduces orthographic feature transforms, while LSS ( 18 ) predicts depth distributions to generate BEV features. BEVDepth ( 7 ) leverages LiDAR supervision to refine depth prediction, and cross-view depth-aware transformer representation (CrossDTR) ( 27 ) improves spatial understanding using depth-aware embeddings and transformers. However, these methods are constrained by occlusions and limited fields of view inherent to onboard sensors.

Compared with vehicle-mounted systems, roadside cameras provide elevated viewpoints and broader scene coverage, effectively compensating for the limited field of view and occlusion issues inherent to onboard sensors. Despite these advantages, infrastructure-based detection faces challenges such as depth ambiguity, reduced accuracy for distant objects, and persistent occlusions. To mitigate these issues, BEVHeight ( 5 ) predicts height distributions to enhance depth estimation from monocular inputs. Camera-based roadside perception (CBR) ( 28 ) removes extrinsic calibration dependence by using multi-layer perceptron (MLP) -based feature transformation at the cost of geometric precision. Furthermore, public benchmarks such as DAIR-V2X ( 13 ) and Rope3D ( 29 ) have been developed to evaluate the effectiveness of roadside perception systems.

Cooperative Perception

V2I cooperative perception enhances situational awareness by enabling information sharing between vehicles and infrastructure. It helps mitigate occlusions and extends the perception range. Based on the data format used for communication, fusion strategies can be classified into early, late, and intermediate fusion. Early fusion transmits raw sensor data, such as camera images, which preserves detailed information for aggregation but requires high communication bandwidth. Late fusion transmits final detection outputs such as 3D bounding boxes, reducing bandwidth but sacrificing contextual richness. For example, Cooper ( 30 ) uses early fusion to broadcast LiDAR data, which incurs substantial communication costs. In contrast, late fusion methods ( 13 ) transmit predictions from each CAV, but their performance suffers from the lack of shared context and reliance on individual detection quality.

Intermediate fusion offers a better trade-off by transmitting compact feature maps, such as BEV representations. These features retain spatial information while significantly lowering communication overhead. Roadside units encode raw sensor inputs into intermediate features using local neural networks and transmit them to CAVs via low-latency V2X links. CAVs can then receive feature maps from multiple viewpoints, increasing spatial diversity. Effective fusion of these features is essential. Aggregation methods such as feature-wise maximum ( 31 ) and summation ( 32 ) are commonly applied. Attention-based approaches, such as attention-based fusion (AttFuse) ( 4 ), V2X-ViT ( 16 ), and cooperative bird's-eye-view transformer (CoBEVT) ( 33 ), leverage transformer architectures for better alignment. Inspired by these, an intermediate BEV fusion framework is proposed that employs deformable mutual-attention to adaptively combine vehicle and infrastructure features.

BEV Scene Understanding

BEV perception plays a key role in spatial reasoning by projecting sensor information into a unified top-down space. The core challenge lies in lifting perspective-view image features to the BEV domain. Existing methods fall into two categories: (1) implicit; and (2) explicit lifting.

Implicit lifting methods learn image-to-BEV mappings using neural networks without enforcing geometric constraints. MLP-based methods, such as view parsing network (VPN), convert image features into BEV space for downstream tasks such as segmentation and road layout estimation ( 34 ). Transformer-based approaches, such as PETR ( 24 ) and BEVFormer ( 8 ), use cross-attention between BEV queries and image features to learn spatial correspondence. BEVFormer further incorporates temporal fusion for improved temporal consistency. However, these methods often neglect camera intrinsics and extrinsics, which are crucial for precise spatial alignment.

Explicit lifting methods rely on camera geometry and estimated depth to project features accurately into BEV. Classical methods such as inverse perspective mapping (IPM) ( 35 ) assume a flat ground and fixed intrinsics, but are prone to sampling artifacts. To overcome the ill-posed nature of monocular depth recovery, Pseudo-LiDAR ( 36 ) reconstructs 3D point clouds from estimated depth. More recent models, such as LSS ( 18 ) and BEVDepth ( 7 ), predict dense depth distributions and use calibrated camera parameters for BEV transformation. BEVHeight ( 5 ) extends this by predicting height distributions rather than absolute depth, improving localization under uncertainty.

Methodology

V2IFormer leverages V2X communication to integrate perception from both vehicles and infrastructure. It consists of three key modules: an infrastructure-side BEV branch, a vehicle-side BEV branch, and a fusion module that combines the BEV features from both sources, as shown in Figure 1.

Figure 1.

Overview of V2IFormer framework.

The framework consists of three main components: (1) the infrastructure-side branch; (2) the vehicle-mounted branch; and (3) the BEV feature fusion module. The infrastructure-side predicts adaptive height distributions using the HeightNet module, while the vehicle-side lifts multiview image features into BEV space using depth prediction. The BEV feature fusion module combines these features with deformable mutual-attention for final 3D object detection.

Infrastructure-Side Branch

In monocular 3D perception, depth prediction is a widely used approach for constructing BEV representations. At close range, objects occupy a relatively large number of pixels, providing rich visual cues such as texture and clear contours, which facilitate accurate depth estimation. Under these conditions, depth-based methods can achieve reliable 3D reconstruction by exploiting strong geometric priors. However, in long-range scenarios, the effectiveness of depth prediction declines significantly. Distant objects appear much smaller in the image, often covering only a few pixels, resulting in insufficient spatial evidence for reliable depth inference. This leads to unstable or biased predictions, limiting the applicability of depth-based methods in V2I settings that require long-range and occlusion-resilient perception.

To address this, inspired by height-based modeling strategies such as BEVHeight ( 5 ), an alternative approach is adopted that estimates object height as an intermediate geometric representation. Unlike depth, object height is relatively invariant to distance and less sensitive to projection distortion. In structured traffic environments, common object categories, such as vehicles and pedestrians, exhibit consistent height patterns, making height a more robust cue for 3D reasoning. Furthermore, infrastructure-mounted cameras, because of their elevated viewpoint, can better leverage height information to infer object location in 3D space with reduced ambiguity. This makes height-based modeling particularly suitable for roadside perception tasks that emphasize long-range detection and occlusion resilience. Based on these insights, the HeightNet module is introduced, a roadside BEV feature extraction module centered on height prediction. HeightNet employs a LID strategy ( 37 ) to model height distributions, enabling precise localization of distant targets and enhancing the geometric consistency of BEV features.

HeightNet

To enable robust and geometry-aware BEV feature extraction from roadside monocular images, HeightNet is introduced, a dedicated height prediction network that estimates pixel-wise height distributions from 2D image features. Unlike depth, object height exhibits greater invariance across distances and is less sensitive to scale distortions, making it a more reliable geometric cue for long-range perception.

As shown in Figure 1, HeightNet consists of two main components: (1) a context branch; and (2) a height estimation branch. The context branch extracts semantic features from the input 2D image features using a series of residual blocks ( 38 ) and a channel attention module. These features are then refined by a deformable convolution (DCN) layer ( 39 ), which enhances spatial adaptability and focuses on object-relevant regions. In parallel, the camera’s intrinsic parameters $I$ , such as focal length and principal point, and extrinsic parameters $E$ , including rotation and translation vectors that define the camera’s position and orientation in 3D space, are encoded through two MLPs to generate camera-aware features $F_{cam}$ . These features are then fused with the contextual features to improve the network’s adaptability across different viewpoints and camera configurations.

Height prediction is formulated as a classification task by discretizing the continuous height range into a predefined number of bins. To ensure accurate and consistent modeling across varying distances, LID ( 37 ) is adopted to partition the height interval. LID increases the bin width linearly with height, which balances sampling resolution across short and long ranges. LID yields more consistent relative error across the full height spectrum, offering precision for nearby objects and robustness for distant targets. This design choice significantly enhances the quality of height-aware BEV features, particularly under long-range and occluded conditions, which are prevalent in V2I scenarios.

Formally, the discretized bin center $h_{i}$ in LID is defined as,

h_{i} = h_{\min} + (h_{\max} - h_{\min}) \times \frac{i (i + 1)}{N_{H} (N_{H} + 1)}

(1)

where $N_{H}$ represents total number of bins and $[h_{\min}, h_{\max}]$ represents the target height interval. This formulation ensures that the discretization adapts smoothly to the distribution of object heights in real-world traffic scenes.

The height prediction module, as shown in Figure 1, is designed to estimate pixel-wise height distributions from image features. The network architecture begins with a set of residual blocks ( 38 ), which process the input 2D image features $F^{2 d}$ . To enhance geometric adaptability, these features are conditioned on camera parameters $F_{cam}$ , which encode intrinsic and extrinsic calibration data. The features are then refined using a DCN layer ( 39 ), which dynamically adjusts the sampling positions to better capture object boundaries and spatial context. Finally, the processed features are passed through a prediction head $ψ_{h} (\cdot)$ to output per-pixel height distributions. The height prediction process can be formulated as,

H^{pre} = ψ_{h} (DCN (Res (F^{2 d} ∣ F_{cam})))

(2)

where $Res (\cdot)$ represents the residual block operations, $DCN (\cdot)$ represents the deformable convolution, $ψ_{h} (\cdot)$ represents the prediction network, and the output $H^{pre} \in R^{N_{H} \times \frac{H}{16} \times \frac{W}{16}}$ represents the predicted discretized height distribution over $N_{H}$ height bins, at a spatial resolution reduced by a factor of 16 relative to the input image.

Height-Based 2D-to-3D Feature Projection

To construct a 3D representation from 2D image features, the predicted height distribution $H^{pre}$ is leveraged to lift pixel-level features into a frustum-shaped 3D space. This process enables spatial reasoning in the height dimension, which is particularly useful for generating BEV representations from monocular images.

First, extract the semantic contextual features $F^{context}$ from the 2D input features $F^{2 d}$ using a series of convolutional operations and channel attention mechanisms. These context features are then combined with the height distribution $H^{pre}$ using an outer product operation.

F_{height}^{3 d} = F^{context} \otimes H^{pre}

(3)

where $\otimes represents$ the outer product across the channel and height dimensions. This results in a volumetric feature tensor $F_{height}^{3 d} \in R^{C_{c} \times N_{H} \times \frac{H}{16} \times \frac{W}{16}}$ , where $C_{c}$ represents the number of channels and $N_{H}$ represents the number of discretized height bins.

To enable downstream BEV-based processing, the frustum-shaped feature volume is projected into the ego vehicle coordinate system using a three-stage geometric transformation pipeline, as shown in Figure 2.

Figure 2.

Height-based two-dimensional–three-dimensional (2D–3D) projector

Step 1: Image Pixel to Camera Coordinates

For each image pixel $p_{imag} = (u, v)$ , a reference point $P_{ref}^{cam}$ is defined at unit depth (i.e., depth = 1) in the camera coordinate system. Using the camera intrinsic matrix $I$ , the mapping is computed as,

P_{ref}^{cam} = I^{- 1} [\begin{matrix} u \\ v \\ 1 \end{matrix}]

(4)

where $I$ represents the intrinsic matrix.

I = [\begin{matrix} f_{x} 0 c_{x} \\ 0 f_{y} c_{y} \\ 0 0 1 \end{matrix}]

(5)

where $f_{x}$ and $f_{y}$ represent the focal lengths in the horizontal and vertical directions, respectively, and $(c_{x}, c_{y})$ represents the principal point (optical center) of the image.

Step 2: Camera to Virtual Coordinate System

The reference point is transformed into the virtual coordinate system as,

P_{ref}^{virt} = T_{cam}^{virt} P_{ref}^{cam}

(6)

where $T_{cam}^{virt}$ encodes the extrinsic transformation (rotation and translation) from the camera to the virtual coordinate system. The virtual coordinate frame is defined so that the origin lies at the camera’s optical center, with the y-axis pointing downward (toward the ground), the $z$ -axis pointing forward, and the $x$ -axis to the right.

Step 3: Virtual to Ego Coordinates

For each height bin $h_{i}$ , the corresponding 3D point in the ego vehicle coordinate system is computed using the principle of similar triangles.

P_{ego}^{height} = T_{virt}^{ego} \times \frac{H - h_{i}}{y_{virt}} \times P_{ref}^{virt}

(7)

where $T_{virt}^{ego}$ transforms coordinates from the virtual frame to the ego frame, $H$ represents the known camera height above the ground, and $y_{virt}$ represents the vertical coordinate of $P_{ref}^{virt}$ . The factor $\frac{H - h_{i}}{y_{virt}}$ scales the reference point to the appropriate height slice. This process assigns each pixel height pair $(u, v, h_{i})$ to a unique 3D position in the ego coordinate system. The full 3D volume is then projected onto the BEV plane by applying sum-pooling along the height dimension $N_{H}$ . The result is a compact 2D BEV feature map that retains semantic and geometric context, and is well-suited for downstream tasks such as object detection and multiview feature fusion.

Vehicle-Mounted BEV Features Extraction

In contrast to the infrastructure-side pipeline, which typically transforms 2D features into BEV representations using height distributions, the vehicle-mounted perception module independently generates BEV features by transforming image data captured from multiple onboard cameras. These features are projected into a unified top-down BEV space centered on the ego vehicle, enabling spatially consistent reasoning within the vehicle’s surrounding environment.

The BEV space is discretized into a fixed-resolution 2D grid of size $x \times y$ , representing a BEV of the environment. Each grid cell corresponds to a fixed physical area of $d \times d$ m on the ground plane (e.g., $d = 0.5 m$ ). To populate this grid with meaningful semantic features, raw image data must be lifted from the 2D image plane into 3D space using depth information, and then reprojected into the BEV plane based on their real-world positions. Given the absence of direct depth sensing in monocular inputs, the LSS paradigm is adopted ( 18 ), which estimates depth distributions for each pixel and uses them to construct a 3D volumetric representation before collapsing it into a 2D BEV feature map.

Let $I_{vehicle} \in R^{N_{v} \times 3 \times H \times W}$ represent the multiview red–green–blue (RGB) images captured by $N_{v}$ vehicle-mounted cameras, each with spatial resolution $H \times W$ . These images are first passed through a shared 2D convolutional backbone to extract visual features, followed by a feature pyramid network to generate multiscale representations. The output is then downsampled to $\frac{1}{16}$ of the original resolution, producing intermediate feature maps for each view.

This output is further processed through two parallel branches.

Depth estimation branch: Outputs per-pixel discrete depth distributions $D_{vehicle} \in R^{N_{v} \times N_{d} \times \frac{H}{16} \times \frac{W}{16}}$ , where $N_{d} = \frac{r}{Δ r}$ represents the number of depth bins covering the range $[1, r]$ m (typically $r = 100$ m, $Δ r = 1$ m), modeling the probability of each pixel belonging to different depths along its viewing ray.

Context feature branch: Outputs semantic features $D_{context} \in R^{N_{v} \times C_{F} \times \frac{H}{16} \times \frac{W}{16}}$ , where $C_{F}$ represents the channel dimension of the feature space.

For each pixel, $N_{d}$ candidate 3D points are sampled along its camera ray by applying inverse projection using the known camera intrinsics and extrinsics. Each 3D point is assigned a semantic feature by weighting the corresponding context feature with the associated depth probability, resulting in a set of probabilistically weighted 3D point features across all camera views.

These 3D features are then projected onto the ground plane and mapped to the corresponding BEV grid cells. To aggregate information from multiple overlapping points that fall into the same grid cell, a pooling operation is applied, producing the final BEV feature map $F_{vehicle}^{bev} \in R^{x \times y \times C_{F}}$ . This representation encodes high-level semantic and geometric information aligned in a globally consistent spatial frame centered on the ego vehicle. Importantly, the BEV features generated from vehicle-mounted cameras can be seamlessly integrated with those produced by the infrastructure-side pipeline. This fusion enables complementary advantages: vehicle-side sensors provide dense near-field coverage, while infrastructure-side sensors offer broader and more stable field of view, especially beneficial in complex occlusion scenarios or when cooperative perception is required.

Vehicle–Infrastructure BEV Feature Fusion

The BEV features $F_{road}^{bev}$ and $F_{vehicle}^{bev}$ exhibit fundamentally different error characteristics because of their observation perspectives. Simple feature stacking fails to align features from the same spatial region accurately. As shown in Figure 1, the V2I BEV Fusion Block leverages deformable mutual-attention to suppress spatial interference from irrelevant regions and effectively fuse roadside and vehicle-mounted BEV features in relevant areas. BEV queries are employed to extract features from the roadside and vehicle-side BEV data. Instead of processing all spatial positions, the model focuses on key locations near each BEV query for feature aggregation, adaptively assigning attention weights to these points. This approach, inspired by deformable mutual-attention in the literature ( 40 ), achieves efficient and precise feature integration.

Given the BEV features $F_{road}^{bev} \in R^{x \times y \times C_{F}}$ and $F_{vehicle}^{bev} \in R^{x \times y \times C_{F}}$ , a $1 \times 1$ convolution is applied to align their feature channels. Subsequently, query BEV features $Q_{p} \in R^{x \times yy \times C_{F}}$ are generated as follows:

Q_{p} = {conv}_{1 \times 1} ([F_{road}^{bev}; F_{vehicle}^{bev}])

(8)

where $Q_{p}$ represents the query at position $p = (x, y)$ from the concatenated features. These queries merge roadside and vehicle features, capturing spatial context effectively. For precise fusion, deformable mutual-attention is applied to the roadside feature map, computed as,

\begin{matrix} DeformAttn (Q_{p}, p, F_{road}^{bev}) = \\ \sum_{m = 1}^{M} W_{m} [\sum_{k = 1}^{K} A_{mqk} \times W_{m}^{'} F_{road}^{bev} (p + Δ p_{mqk})] \end{matrix}

(9)

where $M$ represents the number of attention heads, $m$ indexes the attention head, $K$ represents the number of sampled keys, $k$ indexes each key, $A_{mqk} \in [0, 1]$ represents the attention weight, and $\sum_{k = 1}^{K} A_{mqk} = 1$ , $Δ p_{mqk}$ represents the learned offset from the reference point $p$ , $F_{road}^{bev} (p + Δ p_{mqk})$ represents the feature at the sampled location, $W_{m}$ aggregates the multi-head outputs, and $W_{m}^{'}$ represent the input projection weight.

The same process applies to the vehicle-side feature $F_{vehicle}^{bev}$ . The fused BEV features $F_{fused}^{bev}$ at spatial position $p$ are then computed as,

\begin{matrix} F_{fused}^{bev} = BevFuse (F_{road}^{bev}, F_{vehicle}^{bev}) \\ = \sum_{p = 0}^{HW} [Q_{p}^{'} + \sum_{V \in {F_{road}^{bev}, F_{vehicle}^{bev}}} DeformAttn (Q_{p}, p, V)] \end{matrix}

(10)

where $H$ and $W$ represent the height and width of the BEV map, and $Q_{p}^{'}$ represents the sampled query feature at position $p$ from either the roadside or vehicle-mounted BEV branch.

This approach offers two key benefits. First, it selectively aggregates features from reliable locations, avoiding corrupted regions such as those affected by occlusions or sensor noise. Second, it assigns lower attention weights to less reliable areas, prioritizing high-quality features.

Experimental Evaluation

The proposed V2I collaborative perception framework is evaluated on the DAIR-V2X and V2XSet data sets. The experimental setup and key characteristics of the data set are outlined. Then, a comparative analysis between V2IFormer and representative state-of-the-art methods is conducted.

Data Set

Most existing V2I perception studies rely on simulated data sets with idealized V2X communication, limiting their applicability in real-world scenarios. In contrast, this evaluation is conducted on the DAIR-V2X data set ( 13 ), which captures real-world data. Released in 2022 by Baidu Apollo and the Institute for AI Industry Research at Tsinghua University, DAIR-V2X is the first publicly available benchmark designed for vehicle–infrastructure cooperative autonomous driving.

The data set covers 10 km of urban roads, 10 km of highways and 28 intersections within Beijing’s High-Level Autonomous Driving Demonstration Zone. It encompasses diverse traffic scenarios under varying weather conditions (sunny, rainy, and foggy) and lighting environments (day and night), offering comprehensive support for perception research. DAIR-V2X provides rich multi-modal data, including synchronized camera images, LiDAR point clouds, timestamped annotations, and calibration information. Specifically, the DAIR-V2X-C subset contains 38,845 annotated frames for cooperative detection. The experiments use the (vehicle–infrastructure cooperative synchronized (VIC-Sync) portion of DAIR-V2X for collaborative perception (DAIR-V2X-C), consisting of 9,311 vehicle–infrastructure synchronized frame pairs. Annotations, originally given in world coordinates, are converted into vehicle coordinates for evaluation. The official data split, 17,955 training frames, 4,822 validation frames, and 2,694 test frames, is followed; the karlsruhe institute of technology and toyota technological institute (KITT) I-style average precision metric is adopted for 3D object detection ( 19 ). Figure 3 shows synchronized vehicle and infrastructure views from the data set.

Figure 3.

DAIR-V2X data set: (a) infrastructure-side camera view; and (b) the synchronized view from the vehicle-mounted camera, highlighting the cooperative perception scenario.

The V2XSet is a large-scale V2X perception data set generated in simulation using car learning to act (CARLA) ( 41 ) and open cooperative driving automation (OpenCDA) ( 42 ). It is specifically designed to address the lack of fully public data sets for evaluating real-world challenges, such as localization errors and transmission delays. The V2XSet uniquely integrates realistic noise into V2X collaborative perception. The data set contains 11,447 frames, corresponding to 33,081 samples when accounting for frames per agent, and is split into 6,694, 1,920, and 2,833 samples for training, validation, and testing, respectively. V2XSet covers 55 distinct scenes across eight towns in CARLA. These scenes span five roadway types: (1) straight segments; (2) curvy roads; (3) mid-blocks; (4) entrance ramps; and (5) intersections. Intersection scenes are deliberately oversampled because of their inherent complexity, high traffic volume, and severe occlusions. Each scene lasts up to 25 s and involves between two and seven interacting agents. All agents, including vehicles and infrastructure units, are equipped with 32-channel LiDAR sensors operating at 10 Hz with a 120 m range. The sensor placement is carefully designed to highlight geometric disparities. Infrastructure sensors are mounted on traffic or street light poles at 4.3 m and are deployed at key locations such as intersections, mid-blocks, and ramps. This high-altitude placement contrasts with the lower mounting height of vehicle sensors. Therefore, the inherent variation in installation height leads to distinct measurement patterns that reflect the differing sensor perspectives of infrastructure and vehicle agents.

Evaluation Metrics

The official evaluation metrics defined by the DAIR-V2X data set are adopted ( 13 ) to assess detection accuracy and communication efficiency. Specifically, average precision (AP) ( 20 ) is used to evaluate 3D object detection performance, while Average Byte (AB) quantifies transmission cost. For 3D detection, AP is computed by comparing predicted results with ground-truth annotations provided in the DAIR-V2X-C subset. Following the Vehicle–Infrastructure Cooperative 3D Detection evaluation protocol, objects from the vehicle’s egocentric perspective are evaluated, considering those located within a predefined 3D region in the vehicle coordinate frame: $x \in [x_{\min}, x_{\max}] m$ , $y \in [y_{\min}, y_{\max}] m$ , and $z \in [z_{\min}, z_{\max}] m$ . Detection performance is reported over the full 0–100 m range using a 3D Intersection-over-Union (IoU) threshold of 0.5. Two components are evaluated: (1) ${AP}_{3 D}$ for full 3D bounding boxes; and (2) ${AP}_{BEV}$ for BEV projections. In parallel, AB captures the average size (in bytes) of transmitted data, enabling an assessment of perception quality under communication constraints.

Implementation Specifics

Comparative experiments were conducted between the proposed V2IFormer and several state-of-the-art baselines. For late fusion settings, ImVoxelNet ( 15 ) was adopted as the monocular detector applied to each view independently. For early fusion, PointPillars ( 43 ) serves as the LiDAR-based detection method. On the vehicle side, multiview camera fusion techniques were benchmarked, including BEVFormer ( 8 ), which projects image features into BEV space, and VIMI ( 14 ), a recent multiview fusion approach. On the infrastructure side, BEVDepth ( 7 ) is used to aggregate multi-camera inputs from elevated perspectives. For fair comparison, all models use a ResNet-101 backbone for image encoding. Input images are resized to $864 \times 1, 536$ pixels during training. The data augmentation strategy of BEVDepth is followed, applying random cropping, scaling, flipping, and rotation to images, and random scaling, flipping, and rotation to BEV representations. All models are trained for 150 epochs using the AdamW optimizer ( 44 ), with an initial learning rate of $2 \times 10^{- 4}$ , distributed across eight NVIDIA RTX 3090 GPUs.

Quantitative Results

Analysis on the DAIR-V2X-C data set, which defines short-range as 0–30 m, mid-range as 30–50 m, and long-range as 50–100 m, reveals that methods relying solely on vehicle-mounted (Only-veh) or infrastructure-side (Only-inf) perception exhibit significant limitations, particularly in long-range detection. Quantitative evaluation results are reported in Tables 1 and 2, covering 3D object detection and BEV-based detection across multiple range intervals. Overall, the proposed V2IFormer outperforms all baseline methods for detection accuracy and range robustness. V2IFormer achieves an overall 3D detection performance of 55.64% in ${AP}_{3 D}$ and 59.63% in ${AP}_{BEV}$ . Compared with the representative late fusion method ImVoxelNet, which yields 26.56% ${AP}_{3 D}$ and 31.40% ${AP}_{BEV}$ , V2IFormer improves accuracy by 29.08 percentage points and 28.23 percentage points, respectively. Against the early fusion method PointPillars, which achieves 50.03% ${AP}_{3 D}$ and 53.73% ${AP}_{BEV}$ , V2IFormer still gains an additional 5.61 percentage points and 5.90 percentage points in 3D and BEV detection, respectively, showing its superiority even over strong baselines with direct access to LiDAR data.

Table 1.

Three-Dimensional (3D) Detection Performance ( ${AP}_{3 D}$ , IoU = 0.5) on the DAIR-V2X-C Data Set

			Distance Range (m)
Fusion	Model	Overall	0–30m	30–50m	50–100m
Only-veh	VIMI ( 14 )	8.66%	19.11%	4.33%	0.20%
Only-inf	BEVFormer ( 8 )	8.80%	18.07%	3.71%	1.76%
Only-inf	BEVDepth ( 7 )	7.36%	16.23%	1.79%	0.18%
Late fusion	ImVoxelNet ( 15 )	26.56%	34.20%	17.20%	9.81%
Early fusion	PointPillars ( 43 )	50.03%	53.07%	60.38%	33.05%
Intermediate fusion	V2IFormer	55.64%	72.36%	59.06%	35.50%

Note: AP = average precision; IoU = intersection-over-union; Only-veh = vehicle mounted; Only-inf = infrastructure side; BEV = birds-eye view; DAIR-V2X-C = DAIR-V2X collaborative perception dataset; VIMI = vehicle–infrastructure multiview interaction; ImVoxelNet = image-to-voxel network.

Table 2.

BEV Detection Performance ( ${AP}_{BEV}$ , IoU = 0.5) on the DAIR-V2X-C Data Set

			Distance Range (m)
Fusion	Model	Overall	0–30m	30–50m	50–100m
Only-veh	VIMI ( 14 )	10.46%	22.42%	5.57%	0.42%
Only-inf	BEVFormer ( 8 )	13.45%	24.76%	6.46%	4.63%
Only-inf	BEVDepth ( 7 )	13.17%	26.42%	5.00%	4.82%
Late fusion	ImVoxelNet ( 15 )	31.40%	37.75%	21.21%	12.99%
Early fusion	PointPillars ( 43 )	53.73%	55.80%	64.08%	36.17%
Intermediate fusion	V2IFormer	59.63%	71.05%	63.04%	44.80%

The advantages of V2IFormer become even more evident under long-range detection (50–100 m), where accurate perception is typically more difficult because of reduced resolution, depth ambiguity, and increased occlusion. In this range, V2IFormer achieves 35.50% ${AP}_{3 D}$ and 44.80% ${AP}_{BEV}$ . These values exceed those of PointPillars (33.05% and 36.17%) and ImVoxelNet (9.81% and 12.99%) by significant margins. This suggests that the intermediate fusion strategy effectively aggregates cross-view information and enhances spatial consistency, especially in sparsely observed or occluded areas. Of note, PointPillars exhibits relatively strong performance in the 30–50 m mid-range interval (60.38% ${AP}_{3 D}$ and 64.08% ${AP}_{BEV}$ ), slightly outperforming V2IFormer in this specific range (59.06% and 63.04%). However, this advantage comes at the cost of higher bandwidth usage and performance drop at long ranges. In contrast, V2IFormer maintains consistent and well-balanced detection accuracy across all three intervals (short, mid, and long), demonstrating its robustness and adaptability to varying spatial configurations.

Communication overhead is a critical constraint for real time V2X cooperative perception systems, especially in practical deployments with limited wireless bandwidth. Table 3 presents the average byte (AB) required by each method, revealing a clear efficiency-accuracy trade-off across fusion strategies. V2IFormer requires only 146.24 KB of data transmission per frame, which is an order of magnitude lower than the early fusion baseline PointPillars (1,382.28 KB). Despite this reduction, V2IFormer retains and exceeds the detection performance of PointPillars, indicating its ability to minimize redundant information transfer by focusing on compact, semantically meaningful features. Compared with the late fusion approach ImVoxelNet, which transmits as little as 102.32 bytes per frame by independently processing monocular inputs, V2IFormer consumes moderately more bandwidth. However, this slight increase in data exchange yields substantial performance gains, +29.08% in ${AP}_{3 D}$ and +28.23% in ${AP}_{BEV},$ demonstrating that the cost is well-justified.

Table 3.

Average Byte (AB) Required for Each Method

Fusion	Model	Average Byte
Only-veh	VIMI ( 14 )	0
Only-inf	BEVFormer ( 8 )	0
Only-inf	BEVDepth ( 7 )	0
Late fusion	ImVoxelNet ( 15 )	102.32
Early fusion	PointPillars ( 43 )	1,382.28 K
Intermediate fusion	V2IFormer	146.24 K

Note: The symbol “K” represents kilobytes (KB), where 1 K = 1024 bytes. V2I vehicle-to-everything; Only-veh = vehicle mounted; Only-inf = infrastructure side; BEV = birds-eye view; VIMI = vehicle–infrastructure multiview interaction; ImVoxelNet = image-to-voxel network.

This evaluation underscores the efficiency of the proposed intermediate fusion strategy. It strikes a favorable balance between detection accuracy and communication cost, achieving near-optimal performance without incurring the excessive transmission burden typical of early fusion methods. Such a characteristic makes V2IFormer particularly suitable for scalable deployment in bandwidth-constrained vehicular environments, where minimizing latency and data traffic is essential.

A comprehensive evaluation on the DAIR-V2X-C data set was conducted, which defines short-range as 0–30 m, mid-range as 30–50 m, and long-range as 50–100 m. Among all compared methods, the proposed intermediate fusion approach, V2IFormer, consistently delivers the best performance across all distance intervals. It achieves 35.50% ${AP}_{3 D}$ and 44.80% ${AP}_{BEV}$ in the long-range (50–100 m) scenario, significantly surpassing early fusion (33.05% and 36.17%) and late fusion (9.81% and 12.99%) methods. Although early fusion (e.g., PointPillars) shows competitive accuracy at close and mid-ranges, it suffers from excessive communication overhead, transmitting approximately 1.38 MB of data per frame. In contrast, V2IFormer achieves higher detection accuracy, especially at long distances, and maintains a much lower communication load of 146.24 KB per frame, making it more suitable for real time cooperative perception systems with limited bandwidth. ImVoxelNet, relying solely on monocular camera inputs, struggles with detecting distant objects because of insufficient depth information. PointPillars performs better for infrastructure-side perception by directly processing LiDAR point clouds, which offer precise spatial geometry. However, its performance declines at longer ranges. In summary, V2IFormer effectively addresses three key challenges in V2X cooperative perception: (1) achieving high detection accuracy; (2) ensuring reliable long-range performance; and (3) maintaining efficient bandwidth usage for scalable deployment.

Handling Dynamic Weather and Lighting Conditions

V2IFormer was evaluated on real-world data sets that naturally encompass diverse and dynamically changing environmental conditions, including variations in challenging weather (e.g., rainy or foggy). While recent advances in 3D object detection have achieved remarkable performance on standard vehicle-centric benchmarks such as KITTI (19), nuScenes (45), and Waymo ( 46 ), their generalization capability under extreme or adverse conditions remains limited ( 47 , 48 ). To address this, previous works have curated dedicated data sets targeting challenging weather scenarios for robust vehicle detection ( 49 , 50 ). In contrast, the cooperative vehicle–infrastructure data set DAIR-V2X used in this paper is collected in the wild under a wide spectrum of real-world conditions without frame-level annotation of specific weather types. The absence of fine-grained environmental annotations precludes rigorous quantitative benchmarking under specific conditions, limiting comprehensive performance evaluation. Therefore, qualitative assessments were adopted to validate V2IFormer’s robustness in challenging scenarios. As shown in Figure 4, detection results under occlusion, foggy, and rainy conditions are compared using PointPillars ( 43 ) and the V2IFormer. V2IFormer consistently outperforms baselines, accurately detecting partially occluded vehicles, small pedestrian targets, and distant objects even in low-visibility or degraded illumination conditions. This superior performance stems from V2IFormer’s enhanced BEV representation, which effectively fuses vehicle-side and infrastructure-side BEV features.

Figure 4.

Visualization results under occlusion and adverse weather conditions, showing that V2IFormer accurately detects partially occluded vehicles, as well as distant vehicles and pedestrians in rainy and foggy scenes.

Quantitative Evaluation and Robustness Analysis on V2XSet

Table 4 presents the quantitative results on V2XSet under two evaluation settings. The Perfect setting assumes ideal V2X communication with accurate pose information and zero transmission delay, representing the upper bound of cooperative perception performance. In contrast, the Noisy setting introduces localization disturbances and communication delays, providing a more realistic and stringent benchmark for assessing practical robustness. Under the Perfect setting, all cooperative perception approaches outperform the single-agent baseline. The V2IFormer achieves the best overall performance, reaching 0.925 AP@ 0.5 and 0.864 AP@0.7, outperforming all existing fusion methods. Other cooperative models, such as V2X-ViT (0.882 AP@0.5 and 0.712 AP@0.7) and DiscoNet (0.844 AP@0.5 and 0.695 AP@0.7), also exhibit significant improvements over No Fusion, confirming the benefits of multi-agent feature fusion. The performance gap widens notably under the Noisy setting. Traditional fusion strategies degrade severely when exposed to pose errors and delay. Late Fusion and Early Fusion drop to 0.307 AP@0.7 and 0.384 AP@0.7, respectively, both falling below the No Fusion baseline (0.402 AP@0.7). These results highlight the vulnerability of the fusion mechanisms to spatial misalignment and temporal inconsistencies. In contrast, V2IFormer demonstrates remarkable robustness. It achieves 0.910 AP@0.5 and 0.780 AP@0.7 under the Noisy setting, outperforming all other models by a considerable margin. Compared with V2X-ViT (0.836 AP@0.5 and 0.614 AP@0.7), V2IFormer delivers a 0.166 AP@0.7 improvement, indicating a significantly stronger ability to handle V2X noise. In addition, V2IFormer exhibits the smallest performance drop between the Perfect and Noisy settings (0.084 AP@0.7). However, competing methods suffer much larger declines (e.g., V2X-ViT = 0.098, (open perception vehicle-to-vehicle) OPV2V = 0.177, and V2VNet = 0.184). These results demonstrate that the proposed bilateral BEV representation and deformable BEV feature fusion are highly effective in mitigating V2X noise, enabling V2IFormer to maintain stable and reliable perception performance in realistic deployment scenarios.

Table 4.

Three-dimensional (3D) Detection Performance Comparison on V2XSet

Models	Noisy		Perfect
Models	AP@0.5	AP@0.7	AP@0.5	AP@0.7
No fusion	0.606	0.402	0.606	0.402
Late fusion	0.549	0.307	0.727	0.620
Early fusion	0.720	0.384	0.819	0.710
DiscoNet ( 51 )	0.798	0.541	0.844	0.695
V2VNet ( 52 )	0.791	0.493	0.845	0.677
OPV2V ( 4 )	0.709	0.487	0.807	0.664
V2X-ViT ( 16 )	0.836	0.614	0.882	0.712
V2IFormer (this paper)	0.910	0.780	0.925	0.864

Note: The symbol “@” denotes the intersection-over-union (IoU) threshold used for evaluation (e.g., AP@0.5 and AP@0.7). V2I = vehicle-to-infrastructure; AP = average precision; V2X = vehicle-to-everything; V2V = vehicle-to-vehicle; OPV2V = open perception vehicle-to-vehicle; V2X-ViT = vehicle-to-everything vision transformer.

Conclusion

In this paper, V2IFormer was proposed, a novel V2I cooperative 3D object detection framework that integrates multiview image features from vehicle-mounted and infrastructure-side cameras into a unified BEV representation. To address the challenges of occlusion and limited field of view in complex driving scenarios, a HeightNet module is introduced on the infrastructure side to enhance BEV feature generation through height-aware modeling. On the vehicle side, depth distribution prediction based on the LSS framework is employed to lift perspective-view features into BEV space. Furthermore, a deformable mutual-attention module was designed to adaptively fuse features from both views, focusing on informative regions and suppressing irrelevant noise. Extensive experiments on the DAIR-V2X benchmark validate the effectiveness of the proposed approach. V2IFormer consistently outperforms existing state-of-the-art methods, particularly under conditions of heavy occlusion and degraded visibility. These results highlight the importance of cooperative perception and adaptive feature fusion in improving the robustness and accuracy of 3D detection. While V2IFormer demonstrates strong performance, there are several areas for improvement. One limitation is its reliance on synchronized vehicle–infrastructure data, which may not always be available in real-world scenarios. In addition, the framework could benefit from further optimization for real time deployment, especially when scaling to large autonomous driving systems with diverse sensor types. In future work, V2IFormer will be extended to integrate a wider variety of sensor modalities, such as radar and event-based cameras, and enhance its performance in less controlled, real-world environments. In addition, the potential for real time processing will be investigated and the system’s performance in large-scale autonomous driving applications will be evaluated.

Footnotes

Acknowledgements

We would like to express our gratitude to the editor, reviewers, and all those who have supported us in this research.

Author Contributions

The authors confirm contribution to the paper as follows: study conception and design: Guoqiang Mao, Tianxuan Fu; data collection: Tianxuan Fu, Keyin Wang; analysis and interpretation of results: Guoqiang Mao, Tianxuan Fu; draft manuscript preparation: Tianxuan Fu, Guoqiang Mao. All authors reviewed the results and approved the final version of the paper.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the State Key Laboratory of Integrated Services Networks, and the National Natural Science Foundation of China under Grant U21A20446. This research was also partially supported by the Wuxi Key Laboratory of Integrated Vehicle-Road-Cloud Smart Transportation System.

ORCID iDs

Guoqiang Mao

Tianxuan Fu

Keyin Wang

References

Schwall

Daniel

Victor

Favaro

Hohnhold

Waymo Public Road Safety Performance Data. arXiv Preprint arXiv:2011.00038, 2020.

Ouyang

Simonelli

Ricci

3d Object Detection from Images for Autonomous Driving: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 46, No. 5, 2023, pp. 3537–3556.

Qiao

Zulkernine

Adaptive Feature Fusion for Cooperative Perception Using Lidar Point Clouds. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2023, pp. 1186–1195.

Xiang

Xia

Han

OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication. In

2022 International Conference on Robotics and Automation (ICRA), IEEE, Philadelphia, PA, USA, 2022, pp. 2583–2589.

Yang

Tang

Yuan

Wang

Zhang

Chen

. BEVHeight: A Robust Framework for Vision-Based Roadside 3D Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 2023, pp. 21611–21620.

Yang

Zhang

Zhao

Wang

Huang

Zhang

Wang

MonoGAE: Roadside Monocular 3D Object Detection with Ground-Aware Embeddings. IEEE Transactions on Intelligent Transportation Systems, Vol. 25, No. 11, 2024, pp. 17587–17601.

Yang

Wang

Shi

Sun

Bevdepth: Acquisition of Reliable Depth for Multi-View 3d Object Detection. In Proc., AAAI Conference on Artificial Intelligence, Vol. 37, 2023, pp. 1477–1485.

Wang

Xie

Sima

Dai

BEVFormer: Learning Bird's-Eye-View Representation from LiDAR-Camera via Spatiotemporal Transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 47, No. 3, 2025, pp. 2020–2036.

Huang

Zhu

Bevdet: High-Performance Multi-Camera 3d Object Detection in Bird-Eye-View. arXiv Preprint arXiv:2112.11790, 2021.

10.

Peng

Chen

Liang

Cheng

. BEVSegFormer: Bird's Eye View Semantic Segmentation from Arbitrary Camera Rigs. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2023, pp. 5935–5943.

11.

Schramm

Vdisch

Petek

Kiran

B. R.

Yogamani

Burgard

Valada

BEVCar: Camera-Radar Fusion for BEV Map and Object Segmentation. In *2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, IEEE, Abu Dhabi, United Arab Emirates, 2024, pp. 1435–1442.

12.

Wang

Zhang

Sun

Towards Robust LiDAR-Camera Fusion in BEV Space via Mutual Deformable Attention and Temporal Aggregation. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 34, No. 7, 2024, pp. 5753–5764.

13.

Luo

Shu

Huo

Yang

Shi

Guo

Yuan

Nie

DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 21329–21338.

14.

Wang

Fan

Huo

Wang

Liu

Chen

Zhang

Y.-Q.

VIMI: Vehicle-Infrastructure Multi-View Intermediate Fusion for Camera-Based 3D Object Detection. arXiv Preprint arXiv:2303.10975, 2023.

15.

Rukhovich

Vorontsova

Konushin

ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2022, pp. 2397–2406.

16.

Xiang

Xia

Yang

M.-H.

V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer. In European Conference on Computer Vision (ECCV), Springer, Cham, Switzerland, 2022, pp. 107–124.

17.

Noor-A-Rahim

Liu

Lee

Khyam

M. O.

Pesch

Moessner

Saad

Poor

H. V.

6G for Vehicle-to-Everything (V2X) Communications: Enabling Technologies, Challenges, and Opportunities. Proc., IEEE, Vol. 110, No. 6, 2022, pp. 712–734.

18.

Philion

Fidler

Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV, Springer, Cham, 2020, pp. 194–210.

19.

Geiger

Lenz

Urtasun

Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Providence, RI, USA, June 16–21, 2012, pp. 3354–3361.

20.

Geiger

Lenz

Stiller

Urtasun

Vision Meets Robotics: The Kitti Dataset. The International Journal of Robotics Research, Vol. 32, No. 11, 2013, pp. 1231–1237.

21.

Brazil

Liu

M3D-RPN: Monocular 3D Region Proposal Network for Object Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, Seoul, South Korea, October 27–November 2, 2019, pp. 9287–9296.

22.

Wang

Zhu

Pang

Lin

FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, IEEE, 2021, pp. 913–922.

23.

Wang

Guizilini

V. C.

Zhang

Wang

Zhao

Solomon

DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries. In Proceedings of the 5th Conference on Robot Learning (CoRL), PMLR (Proceedings of Machine Learning Research, Vol. 164), Auckland, New Zealand, 2022, pp. 180–191.

24.

Liu

Wang

Zhang

Sun

PETR: Position Embedding Transformation for Multi-View 3D Object Detection. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVII, Springer, Cham, 2022, pp. 531–548. (Lecture Notes in Computer Science, Vol. 13687),

25.

Liu

Yan

Jia

Gao

Wang

Zhang

PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, Paris, France, 2023, pp. 3262–3272.

26.

Roddick

Kendall

Cipolla

Orthographic Feature Transform for Monocular 3d Object Detection. arXiv Preprint arXiv:1811.08188, 2018.

27.

Tseng

C.-Y.

Chen

Y.-R.

Lee

H.-Y.

T.-H.

Chen

W.-C.

Hsu

W. H.

CrossDTR: Cross-View and Depth-Guided Transformers for 3D Object Detection. In 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE, London, UK, May 29–June 2, 2023, pp. 4850–4857.

28.

Fan

Wang

Huo

Wang

Liu

Calibration-Free BEV Representation for Infrastructure Perception. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, Detroit, MI, USA, October 1–5, 2023, pp. 9008–9013.

29.

Shu

Shi

Wang

Tan

Ding

Rope3D: The Roadside Perception Dataset for Autonomous Driving and Monocular 3D Object Detection Task. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, New Orleans, LA, USA, June 18-24, 2022, pp. 21309–21318.

30.

Chen

Tang

Yang

Cooper: Cooperative Perception for Connected Autonomous Vehicles Based on 3D Point Clouds. In

2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), IEEE, Dallas, TX, USA, July 7–10, 2019, pp. 514–524.

31.

Chen

Tang

Guo

Yang

F-Cooper: Feature Based Cooperative Perception for Autonomous Vehicle Edge Computing System Using 3D Point Clouds. In *Proceedings of the 4th ACM/IEEE Symposium on Edge Computing (SEC 2019)*, ACM, Arlington, VA, USA, November 7–9, 2019, pp. 88–100.

32.

Marvasti

E. E.

Raftari

Fallah

Y. P.

Cooperative Lidar Object Detection via Feature Sharing in Deep Networks. U.S. Patent Application US 17/928,473, filed May 28, 2021, and published May 31, 2023. Assignee: United States Patent and Trademark Office (USPTO).

33.

Xiang

Shao

Zhou

CoBEVT: Cooperative Bird’s Eye View Semantic Segmentation with Sparse Transformers. arXiv Preprint arXiv:2207.02202, 2022.

34.

Yang

Liu

Pan

Projecting Your View Attentively: Monocular Road Scene Layout Estimation via Cross-View Transformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual Conference, June 19–25, 2021, pp. 15536–15545.

35.

Mallot

H. A.

Bülthoff

H. H.

Little

J. J.

Bohrer

Inverse Perspective Mapping Simplifies Optical Flow Computation and Obstacle Detection. Biological Cybernetics, Vol. 64, No. 3, 1991, pp. 177–185.

36.

Wang

Chao

W.-L.

Garg

Hariharan

Campbell

Weinberger

K. Q.

Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, June 16–20, 2019, pp. 8445–8453.

37.

Reading

Harakeh

Chae

Waslander

S. L.

Categorical Depth Distribution Network for Monocular 3D Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual Conference, June 19–25, 2021, pp. 8555–8564.

38.

Zhang

Ren

Sun

Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Las Vegas, NV, USA, June 27–30, 2016, pp. 770–778.

39.

Zhu

Lin

Dai

. Deformable ConvNets V2: More Deformable, Better Results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, June 16–20, 2019, pp. 9308–9316.

40.

Zhu

Wang

Dai

Deformable DETR: Deformable Transformers for End-to-End Object Detection. arXiv Preprint arXiv:2010.04159, 2020.

41.

Dosovitskiy

Ros

Codevilla

Lopez

Koltun

CARLA: An Open Urban Driving Simulator. In Proceedings of the 1st Annual Conference on Robot Learning (CoRL 2017), PMLR (Proceedings of Machine Learning Research, Vol. 78), Mountain View, CA, USA, November 13–15, 2017, pp. 1–16.

42.

Guo

Han

Xia

Xiang

OpenCDA: An Open Cooperative Driving Automation Framework Integrated with Co-Simulation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), IEEE, Indianapolis, IN, USA, September 19–22, 2021, pp. 1155–1162.

43.

Lang

A. H.

Vora

Caesar

Zhou

Yang

Beijbom

PointPillars: Fast Encoders for Object Detection from Point Clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Long Beach, CA, USA, June 16–20, 2019, pp. 12697–12705.

44.

Loshchilov

Hutter

Decoupled Weight Decay Regularization. arXiv Preprint arXiv:1711.05101, 2017.

45.

Caesar

Bankiti

Lang

A. H.

Vora

Liong

V. E.

Krishnan

Pan

Baldan

Beijbom

nuScenes: A Multimodal Dataset for Autonomous Driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Seattle, WA, USA, June 13–19, 2020, pp. 11618–11628.

46.

Sun

Kretzschmar

Dotiwalla

Chouard

Patnaik

Tsui

Guo

, et al. Scalability in Perception for Autonomous Driving: Waymo Open Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Seattle, WA, USA, June 13–19, 2020, pp. 2443–2451.

47.

Hahner

Sakaridis

Bijelic

Heide

Dai

Van Gool

LiDAR Snowfall Simulation for Robust 3D Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, New Orleans, LA, USA, June 18–24, 2022, pp. 16343–16353.

48.

Hahner

Sakaridis

Dai

Van Gool

Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, Montreal, QC, Canada, October 10–17, 2021, pp. 15263–15272.

49.

Dong

Kang

Zhang

Zhu

Wang

Yang

Wei

Zhu

Benchmarking Robustness of 3D Object Detection to Common Corruptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, June 18–22, 2023, pp. 1022–1032.

50.

Hendrycks

Basart

Mazeika

Zou

Kwon

Mostajabi

Steinhardt

Song

Scaling Out-of-Distribution Detection for Real-World Settings. arXiv Preprint arXiv:1911.11132, 2019.

51.

Ren

Chen

Feng

Zhang

Learning Distilled Collaboration Graph for Multi-Agent Perception. Advances in Neural Information Processing Systems, Vol. 34, 2021, pp. 29541–29552.

52.

Wang

T.-H.

Manivasagam

Liang

Yang

Zeng

Urtasun

V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK (virtual), August 23–28, 2020, Proceedings, Part II, Springer, Cham, 2020, pp. 605–621. (Lecture Notes in Computer Science, Vol. 12347).

Multiview Bird’s-Eye View Fusion for Enhanced Three-Dimensional Object Detection in Vehicle–Infrastructure Cooperative Systems

Abstract

Keywords

Introduction

Literature Review

Camera-Based 3D Object Detection

Cooperative Perception

BEV Scene Understanding

Methodology

Infrastructure-Side Branch

HeightNet

Height-Based 2D-to-3D Feature Projection

Step 1: Image Pixel to Camera Coordinates

Step 2: Camera to Virtual Coordinate System

Step 3: Virtual to Ego Coordinates

Vehicle-Mounted BEV Features Extraction

Vehicle–Infrastructure BEV Feature Fusion

Experimental Evaluation

Data Set

Evaluation Metrics

Implementation Specifics

Quantitative Results

Handling Dynamic Weather and Lighting Conditions

Quantitative Evaluation and Robustness Analysis on V2XSet

Conclusion

Footnotes

Acknowledgements

Author Contributions

Declaration of Conflicting Interests

Funding

ORCID iDs

References