Technical Challenges Faced by Autonomous Driving SoC Chips in Complex Urban Road Scenarios

Autonomous driving SoC chips must meet requirements for high-precision perception, real-time decision-making, low power operation, and adaptability to extreme environments when dealing with complex urban road scenarios. The technical challenges can be summarized into the following six core dimensions:

______

1. Real-time and Accuracy Challenges of Multi-modal Sensor Fusion

1. Heterogeneous Data Synchronization and Calibration

   Urban roads require the fusion of data from over 10 sensors, including cameras (RGB/infrared), LiDAR, and millimeter-wave radar (MMW). However, there are timestamp discrepancies (in microseconds) and differences in spatial coordinate systems among the sensors. For example, cameras may lose lane line information in strong backlight, while LiDAR’s point cloud density decreases in rain and fog, necessitating millisecond-level data fusion through the SoC’s heterogeneous computing units (such as NPU-accelerated feature alignment).

2. Complexity of Dynamic Scene Modeling

   The behavior of urban traffic participants is unpredictable (e.g., pedestrians suddenly crossing, non-motorized vehicles going against traffic), requiring the SoC to complete dynamic obstacle trajectory prediction (e.g., LSTM + Transformer hybrid model) within 50ms. Black Sesame Intelligence’s BEV algorithm reduces perception latency at intersections to below 20ms through multi-view camera fusion, but requires the chip to support high-throughput data stream processing.

______

2. Perception Reliability Challenges in Extreme Environments

1. Adaptability to Severe Weather

   Severe weather conditions such as heavy rain and snow can degrade sensor performance:

   • LiDAR detection range decreases by 30%-50%

   • Cameras need a dynamic range covering 140dB (e.g., ON Semiconductor AR0234 ISP)

   • Millimeter-wave radar’s rain attenuation affects target detection accuracy

     requiring the SoC to integrate adaptive algorithms (e.g., multi-frame temporal fusion) and hardware redundancy design (dual sensor inputs), such as NIO ET9 achieving a 2+2 backup architecture with four Orin chips.

2. Long-tail Scenario Processing Capability

   Urban roads present numerous low-frequency but high-risk extreme scenarios (e.g., construction zones, animal crossings), necessitating the SoC to support a continuous learning framework (e.g., federated learning) to cover new scenarios through OTA updates. Tesla’s FSD chip trains long-tail data using the Dojo supercomputer, but the vehicle’s computing power must reserve 20% of resources for model iteration.

______

3. Contradiction Between High Computing Power Demand and Energy Efficiency

1. Surge in Computing Power Density

   L4-level autonomous driving requires simultaneous operation of algorithms for object detection (YOLOv8), semantic segmentation (SegFormer), and path planning (MPC), with computing power demands reaching 1000 TOPS (e.g., NVIDIA Thor). However, urban road scenarios are in a low to medium load state 80% of the time, necessitating the SoC to dynamically adjust voltage and frequency (DVFS) to keep average power consumption below 30W.

2. Memory-Compute Integration Technology Bottleneck

   The current HBM2E memory bandwidth (4.8TB/s) is insufficient to meet the parameter demands of Transformer models (e.g., GPT-3 level models require 10TB/s), leading to over 40% of computation time spent waiting for data. Samsung’s MRAM technology can improve energy efficiency to 100 TOPS/W, but its maturity for mass production is lacking, hindering the implementation of urban NOA scenarios.

______

4. Dual Pressure of Real-time Decision-making and System Safety

1. Millisecond-level Decision Latency

   The time from sensor input to control command output must be <50ms, but in urban roads, V2X communication latency (10-50ms) and algorithm inference latency (20-40ms) can accumulate, potentially leading to emergency braking failures. The SoC needs to integrate Time-Sensitive Networking (TSN) and deterministic scheduling algorithms, for example, Qualcomm Snapdragon Ride Flex achieves decision latency <10ms through a dual-core lockstep architecture.

2. Functional Safety Redundancy Design

   Must meet ISO 26262 ASIL-D level requirements (failure rate <10^-9/h), the SoC needs to adopt multi-chip heterogeneous redundancy (e.g., NIO ET9’s four Orin chips) or single-chip lockstep cores (Renesas R-Car H3). However, redundancy design increases chip area by 30% and costs by 40%, becoming a major obstacle for domestic chip mass production.

______

5. Challenges of Collaborative Optimization Between Software Algorithms and Hardware

1. Model Lightweighting and Computing Power Adaptation

   Urban NOA requires deploying large models like BEV + Transformer, but the native model’s computing power demand exceeds 500 TOPS. Optimization is needed through operator fusion (e.g., NPU microarchitecture customization) and quantization compression (INT8 precision loss <1%), with Horizon Journey 6 improving Transformer inference efficiency by three times through its self-developed BPU architecture.

2. Real-time Assurance of Operating Systems

   Microsecond-level task scheduling must be achieved on QNX/Linux systems, for example, Black Sesame Intelligence’s middleware reduces sensor data transmission latency to 2μs through shared memory pool technology, but resource competition issues still exist during multi-task concurrency.

______

6. Vehicle-Road Collaboration and Communication Bandwidth Limitations

1. V2X Data Interaction Bottleneck

   Roadside units (RSUs) need to transmit information such as traffic light status and construction warnings, but the bandwidth of the 5G-Uu frequency band is limited (100MHz), with latency fluctuations reaching ±20ms. The SoC needs to integrate 5G-V2X baseband chips (e.g., Huawei Balong 5000) and optimize the communication protocol stack, but this increases chip area and power consumption.

2. Edge Computing Collaboration Challenges

   Cloud-trained models (e.g., Waymo’s DGX cluster) need to collaborate with vehicle-side inference, but model compression (e.g., knowledge distillation) may lead to accuracy loss >5%. Black Sesame Intelligence supports dynamic loading of cloud models through the “Nine Shao” architecture, but vehicle-grade safety certification issues need to be resolved.

______

Conclusion and Trends

Autonomous driving SoCs need to find a balance in the computing power-energy efficiency-reliability triangle. Future breakthrough directions include:

• Photonic Chips: Intel’s optical interconnect technology can reduce inter-chip latency to the nanosecond level

• Memory-Compute Integration Architecture: Breakthroughs in the von Neumann bottleneck to improve energy efficiency

• Vehicle-Cloud Collaboration: Achieving dynamic adaptation of algorithms and scenarios through federated learning

With the mass production of domestic chips like Horizon Journey 6 and Black Sesame A2000, Chinese manufacturers are accelerating technological breakthroughs through scenario customization (e.g., Chinese-style intersection avoidance) and ecosystem collaboration (joint development among automakers, chip manufacturers, and algorithms), but the high-end market is still dominated by international products like NVIDIA Thor.