Introduction:
This article is authorized by Yanzhi Intelligent Vehicles, authored by Aimme.
In the autonomous driving system, the perception module, acting as the “senses” to recognize the surrounding environment, is the absolute foundation for the safe and efficient operation of the entire system. Among these, visual perception is an incredibly important part. From L2 to L3 and L4, deep learning algorithms have continuously evolved, and the perception modules have also been upgraded. So, how should we choose the perception module for more advanced autonomous driving systems?
As we all know, the evolution from a distributed architecture to a centralized domain controller architecture has become an irreversible trend for the next generation of autonomous driving systems. For the next generation of autonomous driving systems under centralized domain control architecture, the domain controller has powerful hardware computing capabilities and rich software interface support, allowing more core functional modules to be centralized within the domain controller, greatly enhancing system functionality integration. This reduces the hardware requirements for perception and execution functions. However, the emergence of the domain controller does not mean the large-scale disappearance of the underlying hardware ECUs; many ECU functions will be weakened (software and processing functions downgraded, execution-level functions retained). Most sensors can also directly transmit data to the domain controller or process data preliminarily before sending it to the domain controller. Many complex calculations can be completed within the domain controller, and most control functions can also be carried out within the domain controller. Original ECUs only need to execute commands from the domain controller, meaning that peripheral components only focus on their basic functions while the central domain controller focuses on system-level functionality implementation. Furthermore, the standardization of data interaction interfaces will turn these components into standard parts, thereby reducing the development/manufacturing costs of these components.
As the “eyes” of autonomous driving, cameras are particularly significant. At the L2 stage, the entire visual perception unit of intelligent driving is generally placed in a component known as the camera assembly, which contains both the camera module itself and the software algorithm module that processes the environmental information perceived by the camera, such as ISP, encoding, neural networks, deep learning units, and other AI algorithms. However, in the next generation of advanced autonomous driving systems, these perceptual capabilities originally processed by the camera module will be centralized to the AI chip at the domain controller for processing. So, the question arises: for this type of autonomous driving system architecture, what changes will the requirements for the camera module itself undergo, and what new demands will arise?
This article mainly introduces the basic knowledge of vehicle-mounted cameras, including the basic principles of camera module imaging, types of cameras, methods for selecting cameras, installation methods, and common issues during camera installation. It provides a good reference for the design work of autonomous driving-related components.
The data and features determine the upper limit of machine learning, while models and algorithms only approach this limit.
1
Basic Principles of Camera Modules
The entire vehicle-mounted camera module is divided into several different larger modules. According to the different areas arranged and the different functions realized, the entire camera module is divided into front perception modules, cabin monitoring modules, imaging perception modules, and external imaging modules.
1. Types of Camera Functions
For advanced autonomous driving systems, taking the front perception system as an example, cameras can be categorized based on their types into: monocular perception modules for more robust long-distance small target detection and structural detection; binocular perception modules that are more sensitive to depth information such as distance and speed; and night vision camera modules that are more effective for detecting targets in night driving conditions. The following images represent the different detection function performances of several types of cameras.
2. Types of Camera Modules
From the perspective of camera structure, it mainly consists of lenses, bases, infrared filters, image sensors, PCBs, and FPCs. The two components that most influence imaging quality are the image sensor and the lens. The image sensor is the device that converts light signals into electrical signals, and it is the most important component in the camera, divided into CCD and CMOS categories. CMOS image sensor chips use CMOS technology, integrating the image collection unit and signal processing unit onto the same chip. Its working principle involves integrating the photosensitive element array, image signal amplifier, signal reading circuit, analog-to-digital conversion circuit, image signal processor, and controller onto one chip. Although the imaging quality is not as good as CCD, CMOS is favored by many manufacturers because it consumes less power (only about 1/10 of CCD chips), is smaller, lighter, has a higher integration level, and is cheaper. Currently, mainstream vehicle-mounted cameras use CMOS image sensor chips.
From the perspective of camera applications for different detection functions, they can be divided into front view, side view, rear view cameras, and cabin cameras.
3. Principles of Camera Detection
In fact, before entering the real image processing algorithms, the images entering the camera module have already undergone preliminary digital signal processing (ISP) at the semiconductor synthesis chip processing end of the camera module. This process includes the raw data processing (such as bad pixel correction, black level correction), color processing (white balance, denoising, demosaicing, GAMMA, etc.). Among them, the common 3A digital imaging technology utilizes AF auto-focus algorithms, AE auto-exposure algorithms, and AWB auto-white balance algorithms to maximize image contrast, improve the overexposure or underexposure of the main subject, and compensate for color differences under different lighting conditions, thereby presenting high-quality image information and ensuring accurate color reproduction, achieving perfect day and night monitoring effects. Another technology used is the wide dynamic range HDR technology, which allows the camera to see the characteristics of images under very strong contrast.
It should be noted that if the camera module’s algorithms are sophisticated enough, there will be no need for basic ISP image processing before the camera outputs to the backend AI chip for deep learning, allowing the AI chip to concentrate more computing power and resources on deep learning, neural networks, and other scenarios. This can not only greatly reduce the processing power consumption of the AI chip but also significantly improve the quality of raw data processing performance. Of course, the built-in ISP within the camera module actually raises higher requirements for whether it carries a good semiconductor processing chip, which is also a reason for increasing the cost of the camera module. In actual processing, the camera module and AI chip often undertake a dual ISP processing process.
2
Camera Imaging Performance Indicators
The imaging performance of the camera greatly affects the subsequent AI chip’s understanding of the environment, especially since its deep learning algorithms and computing power are strongly related to the raw data input from the camera module.The main factors affecting the raw data include image size, resolution, field of view, pixel size, dynamic range, and frame rate.
Overall, the main indicators of concern for the camera module are as follows:
1. Imaging Unit
Taking the CMOS camera imaging component commonly used by manufacturers as an example, this type of image sensor integrates the photosensitive element array, image signal amplifier, signal reading circuit, analog-to-digital conversion circuit, image signal processor, and controller onto one chip. In a CMOS chip, each pixel has its own signal amplifier, performing charge-to-voltage conversion independently. To read out the entire image signal, the output amplifier’s signal bandwidth must be wide, while the bandwidth of each pixel amplifier is relatively low, significantly reducing chip power consumption.
The factors affecting imaging quality include both subjective and objective elements. Subjective factors refer to the actual reflective light capacity of dynamic targets in the environment. For example, low illumination in tunnels or rainy and foggy days are objective factors that reduce imaging quality. To improve image quality caused by objective factors, active lighting (generally including DMS cameras, TOF cameras) or color compensation methods are often required; while subjective factors are related to the camera module itself, such as signal-to-noise ratio, resolution, wide dynamic range, grayscale, color reproduction, etc.
2. Integration
For the camera module, considering its processing capability needs to meet more raw scene requirements, it is necessary to integrate signal amplifiers, signal reading circuits, AD conversion circuits, image signal processors, and controllers onto one chip. This will achieve chip-level camera functions within the front-end module.
3. Acquisition Speed
The camera module needs to output each photosensitive element individually and must have multiple charge-to-voltage converters and row/column switch control, with a readout speed generally greater than 500f/s. For high-resolution camera modules, downsampling is often required to output sub-window images, which can achieve higher speeds when only outputting sub-window images. For example, the currently widely used Horizon J3 chip processes 8-megapixel images and requires downsampling before directly inputting to the serializer to meet J3’s processing capability.
4. Noise Processing
Currently, various camera suppliers prefer camera modules based on CMOS technology. These modules often lack PN junctions or silicon dioxide isolation layers, making it impossible to effectively isolate noise. The proximity of components and circuits can lead to significant interference. Therefore, higher demands for noise reduction technology on the front-end module are required.
5. Power Consumption
Previously, camera assemblies were often positioned at the front windshield, and the integrated AI chips would heavily utilize deep learning algorithms for extensive computations. Therefore, their power consumption was substantial. In the next generation of autonomous driving system architecture, if the camera merely performs raw image recognition, it can reduce power consumption from computational units.
3
Camera Peripheral Interfaces
For autonomous driving control systems designed with a centralized domain controller architecture, the input interface data from cameras will no longer be the CAN data that can be directly used for algorithm control but rather raw image data. Common camera raw data input interface formats include FPD LINK III, MIPI, and DVP.
FPD-Link was the first successful application of the LVDS specification. Due to the successful use of FPD-Link for the first time, many display engineers use LVDS terminology to replace FPD-Link. LVDS is also the primary method for camera image transmission, while the MIPI-CSI-2 (Camera Serial Interface) protocol is a sub-protocol of the MIPI Alliance protocol, widely used for its high speed and low power consumption. It is a transmission protocol specifically configured for the LVDS transmission medium.
For the entire autonomous driving system, each new module requires environmental adaptation. The adaptation process is as follows:
Using the vehicle’s external CAS as a carrier, simulation software such as CATIA and CAD is used to establish a simulation to determine the layout position, releasing the corresponding location information to the camera module supplier. The camera module supplier then performs module tuning, ISP tuning, hardware modifications, followed by subjective and objective testing. The results of the module tuning will be determined by the standards set by the industry or company. A module that passes the tuning can further undergo algorithm testing. The reasonableness of the camera module selection will be based on the final module qualification testing report.
4
Conclusion
In summary, the factors affecting the recognition performance of camera modules include module performance, layout position, detection environmental light, etc. Of course, image tuning can partially solve the impacts caused by the above issues, and later, standardization can guide the generation of subjective and objective testing standards, optimize algorithm requirements, and integrate them into product requirements.
END
Course Announcement
On September 24 at 7 PM, the Horizon Developer Special Session will officially start online! Senior engineer Sui Wei from Horizon 3D Vision will explain “The Application of 2.5D & 3D Vision Perception Technology in Indoor Robots”.
Group Application
