This article is an original piece from the WeChat public account “Dahua Imaging” and the Zhihu column “All in Camera”. Please indicate the source when reprinting. Dahua Imaging readers QQ group: 237427716
Our teaching videos on Dahua Imaging, including “Image Quality Testing and International Standards”, “CMOS Sensor Testing and International Standards”, and “Digital Imaging Systems”, are now available on Taobao. To purchase the courses, simply search for ‘Dahua Imaging’ on Taobao.
This article is translated from https://www.eetasia.com/news/article/ADAS-Key-Trends-on-Perception. If you are interested, you can check the original text. The translation may not be perfect, and if you have better suggestions, feel free to contact me.
Phenomena at the AutoSens 2019 conference:
The automotive industry is still seeking “robust perception” that can be used under any conditions, including but not limited to night, fog, rain, snow, and icy scenes.
From the results of the previous AutoSens 2019 conference, this field is not lacking in technological innovation. As technology developers, Tier 1 automotive manufacturers and OEMs are still looking for “robust perception” technology that can work normally under any road conditions, including night, fog, rain, snow, ice, and oil.
Although the automotive industry has not found a panacea, many companies have launched their new perception technologies or product concepts.
The latest test vehicles from Cruze are coming off the GM production line, equipped with sensors as shown in red. (Source: Cruze)
At this year’s AutoSens held in Brussels, the focus of discussion was on Advanced Driver Assistance Systems (ADAS) rather than Autonomous Vehicles (AV).
It is clear that the engineering community no longer denies but has reached a greater consensus that there is still a huge gap between the current state and the eventual rollout of commercially viable AI-driven autonomous vehicles, which is truly driverless.
It should be made clear that no one is saying that autonomous vehicles are impossible. However, Phil Magney, founder and head of VSI Labs, predicts that “Level 4 autonomous vehicles will be launched within strictly limited operational design domains (ODD) and will be based on very comprehensive and thorough safety cases. In limited operational design domains, such as specific routes, specific lanes, specific operating hours, specific weather conditions, specific times of day, and specific pick-up and drop-off points, etc.”
Phil Koopman, a professor at Carnegie Mellon University and CTO of Edge Case Research, pointed out that the foundation of every highly automated vehicle is “perception” technology, which needs to know where objects are. He explained that compared to human drivers, autonomous vehicles are weaker in their “predictive” ability, which is the understanding of context and predicting where perceived objects might go next.
Bringing Intelligence to Edge Devices
A new trend emerged at the conference, which is to make edge devices smarter. Many vendors are enhancing the intelligence of perception by fusing different sensor data on edge devices (e.g., RGB camera + NIR; RGB + SWIR; RGB + LiDAR; RGB + Radar).
However, there are differences of opinion among industry participants on how to achieve this goal. Some are fusing sensor data on edge devices, while others, such as Waymo, prefer to perform centralized fusion of raw sensor data on a central processing unit.
By 2020, Euro NCAP requires the use of Driver Monitoring Systems (DMS) as a primary safety standard, and many new monitoring systems of this kind appeared at AutoSens. These systems can monitor not only the driver but also passengers and other objects inside the vehicle.
One typical example is the new RGB-IR image sensor from ON Semiconductor combined with Ambarella’s advanced RGB-IR video processing SoC and Eyeris’s in-vehicle scene understanding AI software.
NIR vs SWIR
Whether inside or outside the vehicle, the need to see clearly in the dark highlights the reason for using IR. While ON Semiconductor’s RGB-IR image sensor uses NIR (Near-Infrared) technology, another participant at the exhibition, Trieye, further demonstrated a SWIR (Short-Wave Infrared) camera.
What is short-wave infrared (SWIR)? (Source: Trieye)
The advantages of SWIR include the ability to see objects under any weather/light conditions. More importantly, SWIR can also identify road hazards (such as ice) in advance because SWIR can detect the unique spectral response corresponding to the chemical and physical properties of each material.
However, due to the high cost of indium gallium arsenide (InGaAs) used to manufacture SWIR cameras, its use is limited to military, scientific, and aerospace applications. Trieye claims to have found a way to design SWIR using CMOS process technology. “This is the breakthrough we have achieved. Like semiconductors, we have used CMOS for mass production of SWIR cameras since day one,” said Avi Bakal, CEO and co-founder of Trieye. Bakal stated that compared to InGaAs sensors priced over $8,000, cameras using Trieye technology will be priced at the level of “tens of dollars.”
SWIR Camera Stripped (Source: Trieye)
Shortage of Labeled Datasets
One of the biggest challenges for artificial intelligence is the shortage of training data.More specifically, “labeled training data,” Magney said.Inference models are primarily related to data and how it is collected.Of course, training data needs to be labeled with metadata, which is very time-consuming.“
There was a heated discussion at AutoSens regarding the use of GAN (Generative Adversarial Networks) methods. According to Magney, in GAN, two neural networks compete to create new data.Given a training set, this technique learns to generate new data with the same statistics as the training set.
For example, Drive.ai is using deep learning to enhance the automation of data labeling to speed up the tedious data annotation process.
During a presentation at AutoSens, Koopman also talked about the daunting challenge of accurately labeling data.He expressed doubt that much data remains unlabeled because only a few large companies can afford to label large amounts of data.
In fact, AI algorithm startups participating in the exhibition also admitted that paying third parties for labeled data is quite painful.
GAN is one approach.However, Edge Case Research proposed another method that accelerates safer perception software development without needing to label all data.The company recently announced a tool called Hologram, which provides an AI perception stress test and risk analysis system.According to Koopman, Hologram can provide warnings for “problematic” or “best to check again” areas for petabyte-level (1T is 1024 times) unlabeled data by running it twice, alerting to collect more data or perform more training.
Another issue that arose at the conference was what to do with previously labeled datasets if automotive OEMs change the cameras and sensors used for data training.
David Tokic, Vice President of Marketing and Strategic Partnerships at Algolux, told EE Times that there are two things automotive engineers working on ADAS and AV care about:1) Robust perception under all conditions;2) Accurate and scalable visual models.
Today, camera systems deployed in ADAS or AV are varied and have significant differences.These differences mainly come from their lenses (different lenses provide different FOV), sensors, and ISP (image signal processing), and design parameters vary.If a tech company chooses a camera system, collects a large dataset, annotates it, and trains to build an accurate neural network model for that system.
But what happens when OEMs replace the camera systems originally used for training data?This change may affect perception accuracy because the neural network model tuned for the original camera is now processing a new set of raw data.
Does this require OEMs to retrain their datasets?
Tesla, Waymo, and GM/Cruise utilize a large number of cameras in their AVs (source: Algolux)
When asked about the interchangeability of camera systems, Magney from VSI Labs said:“Unless the camera specifications are the same, I don’t think it’s an option.”He pointed out, “For example, at VSI, we trained our neural network for FLIR thermal imaging, and the images used for training were collected according to the same specifications as the thermal imaging cameras we deployed later. We later changed the sensors, but the hardware specifications were the same.”
However, Algolux claims to have adopted a new approach that can “convert these previously created datasets in just a few days.”Tokic said the company’s Atlas camera optimization suite can achieve this by understanding the “prior” characteristics of the camera system (the characteristics of the camera and sensor) and applying them to the detection layer.Tokic said:“Our mission is to democratize camera technology for OEMs.”
AI hardwareIn recent years, many new AI processor startups have emerged, creating a trend in AI. Some have announced the arrival of a hardware renaissance. Many AI chip startups are targeting ADAS and AV as their market.
In response to this emerging AI accelerator market, Ceva showcased its new AI core and “Invite API” for the first time at the AutoSens conference.
However, it is strange that the new generation of feature-rich vehicles has not yet initiated the deployment of new AI chips—except for chips designed by Nvidia and Intel/Mobileye, as well as Tesla’s internally developed “Full Self-Driving (FSD) computing” chip.
On the other hand, ON Semiconductor’s RGB + IR camera system unveiled at AutoSens chose Ambarella’s SoC as its AI processor to perform in-vehicle monitoring tasks.
Eyeris CEO Modar Alaoui acknowledged that Ambarella is usually not referred to as an AI accelerator equipment (rather, it is a traditional video compression and computer vision chip company). He said, “We couldn’t find any AI chip that could support running 10 neural networks and capture 30 frames per second video using up to 6 cameras, all while consuming less than 5 watts. All cameras are running inside the vehicle to run Eyeris’s AI in-vehicle monitoring algorithms. However, Ambarella’s CV2AQ SoC meets the requirements and outperformed all other hyped accelerators.”
However, Alaoui hopes that his company’s AI software will be ported to three other hardware platforms at the Consumer Electronics Show in Las Vegas next January.
ON Semiconductor, Ambarella, and Eyeris demonstrated a new in-cabin monitoring system using three RGB-IR cameras. (Photo: EE Times)
At the same time, ON Semiconductor emphasized that Driver and Occupant Monitoring Systems (DMS) need to have the ability to “capture images under variable lighting conditions from direct sunlight to deep darkness.” It claims that “with its good near-infrared spectral response, the RGB-IR CMOS image sensor can provide Full HD 1080P image output through 3.0 µm back-illuminated (BSI) and three-exposure HDR technology.” Utilizing sensitivity to both RGB and IR light, the RGB-IR CMOS image sensor can capture color images under sunlight and NIR illumination.
Beyond Driver Monitoring Systems (DMS)Alaoui boasts that Eyeris’s AI software can perform complex body and facial analysis, passenger activity monitoring, and object detection. He added, “In addition to driver monitoring, we are also studying everything inside the car, including the surfaces of the seats and the steering wheel,” emphasizing that this startup has surpassed the current achievements of Seeing Machines.
Seeing Machines’ Director of Customer Solutions for Europe, Laurent Emmerich, has a different view. He said, “It is natural to go beyond monitoring the driver and cover the interior of the vehicle. We are also following up on this.”
He added that, compared to startups, Seeing Machines’ advantage lies in “the solid foundation of AI expertise we have accumulated over the past 20 years in computer vision.” Currently, the company’s driver monitoring systems have been “adopted by six automotive manufacturers and designed into nine projects.”
Moreover, Seeing Machines pointed out that they have also developed their own hardware—the Fovio driver monitoring chip. When asked whether this chip could also serve future in-vehicle monitoring systems, Emmerich explained that the chip’s IP will be applied to configurable hardware platforms.
RedundancyIncreasing redundancy, combining different forms of sensors, and installing them in vehicles is not only necessary for enhancing perception but also critical for safety.
Founded by former Withings CEO Cedric Hutchings, the startup Outsight launched a new highly integrated box composed of multiple sensors at AutoSens. He explained that Outsight’s sensor fusion box is designed to “provide perception through understanding and achieve localization by understanding the entire environment (including snow, ice, and oil on the road).” He added, “We can even sense the materials of the road using active hyperspectral sensing.”
When asked which company’s sensors are included in Outsight’s box, he declined to comment. “As we are still adjusting the right specifications and suitable features, we will not announce our main partners at this time.”
EE Times subsequently discussed with Trieye and indicated that Outsight would use Trieye’s SWIR camera. Hutchings explained that Outsight is promoting its sensor fusion box to Tier 1 manufacturers and OEMs, with sampling planned for the first quarter of 2020. Hutchings explained, “The box can provide ‘no related data’ to ensure safety and ‘true redundancy’ as an additional standalone system for Tier 1 manufacturers and OEMs.”
This box uses “no machine learning,” thus providing deterministic results to make it “verifiable.”
Aeye is also marketing its iDAR for the ADAS/AV market, a system that integrates MEMS LiDAR with high-definition cameras. AEye’s Vice President of Product Management, Aravind Ratnam, said that the system can operate in real-time by combining the two sensors and embedded AI to address certain extreme situations.
The company explained that the iDAR system is designed to combine 2D camera “pixels” (RGB) and 3D LiDAR “voxels” (XYZ) to provide a new type of real-time sensor data that can deliver more accurate, broader, and smarter information to AV computing at a faster speed.
AEye’s AE110 product features compared to industry benchmarks and capabilities (Source: AEye)
Ratnam stated during the presentation that AEye studied various use cases.“We reviewed 300 scenarios, selected 56 applicable cases, and narrowed them down to 20 scenarios,” discovering that integrating cameras, LiDAR, and AI is reasonable.
Ratnam showcased one of the scenarios, where a child suddenly chases a ball onto the street—right in front of the vehicle. The edge camera and LiDAR fusion can work faster, thus shortening the vehicle’s reaction time.He noted:“Our iDAR platform can provide very fast computing speeds.”
When asked about the advantages of edge sensor fusion, an attending Waymo engineer told EETimes that he was not sure if it would bring substantial changes.He asked:“Is it microseconds? I’m not sure.”
AEye is confident about the additional value its iDAR can provide for layered products.Through close collaboration with major partners Hella and LG, AEye’s Ratnam emphasized:“We have been able to reduce the cost of iDAR. We are now providing 3D LiDAR at ADAS prices.”“AEye will complete the integration of automotive-grade RGB and LiDAR systems with embedded AI in the next three to six months, with prices below $1,000,” Ratnam stated.
Automotive LiDAR system shipments (Source: IHS Markit)
Dexin Chen, Senior Analyst for Automotive Semiconductors and Sensors at IHS Markit, told attendees that LiDAR suppliers have been “over-marketing and over-promising.”He pointed out, “Looking ahead, while the physical advantages of LiDAR can be promoted, commercialization ultimately determines the outcome.”What is urgently needed is “standardization, alliances and partnerships, supply chain management, and AI collaborations.”