What Is Perception?

In the realm of autonomous driving, the purpose of perception is to mimic the human eye to collect relevant information, providing necessary data for subsequent decision-making. Depending on the decision-making tasks at hand, perception can include many sub-tasks such as lane detection, 3D object detection, obstacle detection, traffic light detection, and so on. Based on the results predicted by perception, decisions are made; finally, corresponding actions (such as lane changes or overtaking) are executed.

How to Achieve Perception?

Since perception aims to mimic the human eye in acquiring information about the surrounding environment, it inevitably requires sensors to complete the information collection task. Currently, the sensors used in the field of autonomous driving include: cameras, lidar, and radar.

Forgot the image source, please delete if infringing.

As we can see, there are many types of sensors with varying costs, so how to use these sensors for perception tasks is a unique solution for each autonomous driving manufacturer.Pure Vision Perception SolutionsCurrently, Tesla is a typical representative of pure vision perception solutions.The advantages and disadvantages of pure vision perception solutions are also quite evident: Advantages: very low cost; Disadvantages: The images captured by cameras are 2D, lacking depth information, which must be learned through algorithms, leading to a lack of robustness.Sensor Fusion Perception SolutionsCurrently, most manufacturers adopt multi-sensor fusion solutions; their advantages and disadvantages are: Advantages: They can fully utilize sensors with different working principles to enhance overall perception accuracy in various scenarios. Additionally, if a certain sensor fails, other sensors can act as redundant backups, improving system robustness. Disadvantages: The cost is significantly higher than that of pure vision solutions due to the use of multiple sensors.Post-Fusion of SensorsPost-fusion refers to each sensor performing deep learning model inference on target objects separately, outputting results with their respective sensor attributes. The recognition results from each sensor are input into the fusion module, which assigns different confidence levels to the recognition results of each sensor in different scenarios and ultimately makes decisions based on the fusion strategy.The overall flowchart is as follows:

Image source: https://mp.weixin.qq.com/s/bmy9EsQaLNPQQKt9mPTroA

Advantages: Different sensors independently perform target recognition, allowing for good decoupling, and each sensor can serve as a redundant backup; at the same time, post-fusion solutions facilitate standard modular development, encapsulating interfaces for plug-and-play use by OEMs. For OEMs, the recognition results from each sensor are input into the fusion module, which assigns different confidence levels to the recognition results of each sensor in different scenarios and ultimately makes decisions based on the fusion strategy.Disadvantages: There are issues of “temporal perception discontinuity” and “spatial perception fragmentation”.Spatial Perception FragmentationDue to the installation angles of lidar and cameras around the vehicle, multiple sensor entities cannot achieve continuous coverage and unified recognition in the spatial domain, leading to cameras capturing only a small portion of the target, making it impossible to make correct detection results based on incomplete information, thus compromising the effectiveness of subsequent fusion.Temporal Perception DiscontinuityThe results captured by cameras are in frames, and a common perception method is to concatenate the detection results of continuous single frames, similar to post-fusion strategies, which cannot fully utilize useful information in the temporal sequence.Pre-Fusion of SensorsPre-fusion refers to aggregating data collected by various sensors together, synchronizing the data, and then fusing the raw data.The overall flowchart is as follows:

Image source: https://mp.weixin.qq.com/s/bmy9EsQaLNPQQKt9mPTroA

Advantages: This allows for earlier data fusion, making the data more correlated; for example, fusing the point cloud data from lidar with the pixel-level data from cameras results in less data loss.Disadvantages: The data obtained from different sensors (pixel data from camera images and point cloud data from lidar) have different coordinate systems; visual data is in 2D space while point cloud data from lidar is in 3D space. Therefore, there are two approaches for fusing heterogeneous data: Approach One: Using point cloud data to provide depth information in image space; Approach Two: Using visual data to provide semantic features in point cloud space, for point cloud coloring or feature rendering.

Image source: https://mp.weixin.qq.com/s/bmy9EsQaLNPQQKt9mPTroA

Thus, to ensure that data from different coordinate systems (pixel data, point cloud data) can be transformed into the same coordinate system for data fusion, perception tasks under the BEV (Bird Eye View) perspective have gradually gained widespread attention.In-Fusion of SensorsIn-fusion refers to first extracting features from the data collected by various sensors through neural networks, and then performing feature-level fusion on the multiple sensor features extracted by the neural networks, thereby increasing the likelihood of achieving optimal perception results. The features extracted from heterogeneous data are fused at the feature level in the BEV space, which minimizes data loss and reduces computational consumption (compared to pre-fusion), so strategies that employ in-fusion for perception tasks under the BEV perspective are quite common.

Perception Task Paradigm Under BEV Perspective

• Input camera data (2D images) into a feature extraction network to complete feature extraction for multiple camera data;
• Map all camera data features extracted through network learning to the BEV space;
• In the BEV space, perform fusion of heterogeneous data, merging the features mapped from image data in the BEV space with lidar point cloud features; (optional, as BEVFormer constructs BEV space features using only 6 cameras)
• Conduct temporal fusion, merging features from previous moments to enhance perception capability; (I personally believe that introducing temporal features can somewhat mitigate occlusion issues)
• Use the obtained BEV features for downstream tasks; (lane detection, obstacle detection, 3D object detection, etc., essentially making the entire model a multi-task learning model)

Advantages of Perception Under BEV Perspective

• Easier implementation of cross-camera and heterogeneous data fusion

When performing cross-camera fusion or heterogeneous data fusion, due to the different coordinate systems represented by different data, many post-processing rules are needed to correlate the perception results of different sensors, making the process quite complex. By performing fusion in the BEV space, mapping rules can be autonomously learned through the network, generating BEV features for downstream perception tasks, simplifying algorithm implementation, and allowing for direct expression of the size and orientation of objects perceived visually in the BEV space.

• Easier implementation of temporal fusion

When constructing the BEV space, temporal information can be easily fused, allowing the obtained BEV features to better accomplish some downstream perception tasks, such as speed measurement tasks.

• Mildly alleviates occlusion issues in perception tasks

Traditional 2D perception tasks can only perceive visible targets and are powerless against occlusions. However, in the BEV space, based on prior knowledge or utilizing temporal fusion, predictions can be made for occluded areas, thus “imagining” what objects may exist in those occluded areas. Although the “imagined” objects involve a degree of “imagination”, they still provide significant benefits to downstream regulation and control modules.

• Facilitates multi-task learning

When using traditional methods for perception tasks, target recognition, tracking, and motion prediction must be performed sequentially, resembling a “serial system” where upstream errors propagate downstream, causing cumulative errors. However, within the BEV space, perception and motion prediction are accomplished in a unified space, allowing for direct end-to-end optimization through neural networks, producing results in parallel. This not only avoids cumulative errors but also significantly reduces the role of manual logic, enabling the perception network to learn through data-driven methods and better achieve functional iteration.

References

1) Understand Feature-Level Fusion in BEV Space in One Article2) How to Achieve “Light High Precision Map” Urban NOH? Eight Highlights of Momenta Autonomous Driving

Welcome all angel round enterprises in the entire automotive industry chain (including the electrification industry chain)to join the group(Friendly connections including top investment institutions among 700 automotive investment institutions; a portion of quality projects will be selected for thematic roadshows to existing institutions));There are communication groups for leaders of scientific and technological innovation companies, as well as dozens of groups for the automotive industry, including complete vehicles, automotive semiconductors, key components, new energy vehicles, smart connected vehicles, aftermarket, automotive investment, autonomous driving, vehicle networking, etc. To join the group, please scan the administrator’s WeChat(Please indicate your company name)