Meeting the real-time requirements of intelligent driving incurs high costs in hardware, energy consumption, algorithms, system optimization, and validation. However, these are essential conditions for ensuring the safe and efficient operation of intelligent vehicles in complex dynamic environments.01
What is Real-Time?
Real-time (实时性) refers to the ability of a system or program to respond to inputs within specific time constraints. In computer science and control systems, real-time means that the system must process inputs and produce outputs within a defined time frame to ensure functional correctness. A key characteristic of real-time systems is that they require not only correct functionality but also the completion of tasks within certain time limits.
Figure 1: Diagram of Real-Time Systems
In the field of real-time systems, it can be further divided into hard real-time, soft real-time, and quasi-real-time.
Hard Real-Time Systems have a set of strict deadlines, and missing a deadline is considered a system failure. Typical examples of hard real-time systems include aircraft sensors, spacecraft, and planetary probes.
Soft Real-Time Systems attempt to meet deadlines, but if a deadline is missed, the system does not fail, although service quality may degrade to improve response speed. Typical examples of soft real-time systems include entertainment audio and video streaming software. A timeout may affect the experience but will not lead to catastrophic consequences. In this case, the software may reduce 1080P to 480P to ensure smooth video playback.
Quasi-Real-Time Systems treat information delivered or computations completed after the deadline as invalid. Similar to soft real-time systems, missing a deadline does not cause system failure, but service quality may still degrade. Typical examples of quasi-real-time systems include financial forecasting systems and robotic assembly lines.
Additionally, it is important to note a common misconception: real-time systems are often discussed in conjunction with low-latency systems, but the core definition of real-time does not lie in low latency but in deterministic scheduling. This means that the system must guarantee the completion of specific tasks within a specific time. Therefore, the latency in real-time systems must be measurable, and a maximum allowable delay for tasks must be set.
02
Why Does Intelligent Driving Require Real-Time Performance?
Intelligent driving systems have a vast array of functionalities and stringent safety requirements. If an obstacle suddenly appears ahead and is not detected and processed in time, it could lead to catastrophic accidents. At the same time, the electronic and electrical architecture of vehicles is evolving towards integration and centralization, during which many traditional, single-function independent controllers are being integrated into high-performance computing platforms. Previously, it was easier to ensure real-time performance in simple and reliable systems within single controllers, but the situation becomes more complex in computing platforms, highlighting the importance of system real-time performance. Especially in intelligent driving systems, the core functionality and backbone of high-performance computing platforms is typically intelligent driving. The intelligent driving software must coexist and share resources with other software functionalities without compromising its real-time performance and safety. With the advent of electrification and intelligent connectivity, the number of digital functions in vehicles is increasing, along with the associated challenges.
Figure 2: Diagram of Emergency Steering to Avoid Collision
Moreover, intelligent driving systems not only impose real-time requirements on individual functions but also introduce additional priority dependencies within the overall link between sensors, control software, and actuator networks. This mixed-criticality system has higher requirements for safety-critical compliance, temporal and spatial isolation.
03
Intelligent Driving Requires Multiple Real-Time Capabilities
Since real-time performance is so crucial for intelligent driving, what specific levels of real-time performance are required? What is the acceptable response time tolerance? Is it 1 microsecond? 1 millisecond? 1 second? Or 1 minute? None of these are correct. Because the unit of real-time performance is not seconds or minutes, but rather the functional cycle curve.
Figure 3: Diagram of Functional Cycle Curve
As shown in the figure, different functions have varying sensitivities to real-time performance at different stages. For example, in traditional sensor fusion sub-functions, when the detected target’s speed is 30 m/s, a 10 ms time tolerance may lead to a spatial tolerance of 0.3 m among different sensors, affecting fusion performance. However, in the currently popular end-to-end AI architecture, a 10 ms time tolerance may have different dilution effects across different models.
For instance, in the AEB function, the requirements for end-to-end latency differ between extreme ghosting scenarios and conventional stopping in front of stationary vehicles (CCRS), leading to different deadline requirements. Besides time sensitivity, different functions may also have varying correctness requirements for task execution at different stages. For example, the execution precision required for the algorithm task implementing motor vector control is higher than that for the obstacle classification calculation in front, as the consequences of errors in the two calculations differ significantly.
04
What Costs Must Intelligent Driving Pay for Real-Time Performance?
As Chairman Mao once said, “It is not difficult for a person to do a good deed once; the challenge lies in doing good deeds consistently throughout one’s life and avoiding bad deeds.” This saying applies well to the real-time requirements in intelligent driving. Achieving real-time performance for a single task is not particularly difficult, but ensuring that hundreds or thousands of primary computational tasks meet real-time performance throughout their functional lifecycle on a complex computing platform, regardless of external conditions, is quite challenging. Of course, this difficulty is not insurmountable, but it comes at a significant cost. These costs are concentrated in several areas:
-
High-Performance Hardware Resource Costs
Intelligent driving requires processing a large amount of sensor data, including cameras, millimeter-wave radar, LiDAR, etc. The real-time processing of this data demands extremely high computational power. To meet this demand, high-performance processors (such as GPUs, TPUs, dedicated ASIC chips, or even custom chips developed at great expense) must be equipped. These hardware components are not only expensive but also increase system complexity and power consumption. Additionally, real-time processing requires fast storage and high bandwidth to ensure smooth data transmission. Higher-performance storage devices and communication systems also incur additional hardware costs.
Figure 4: Diagram of a High-Performance Computing Platform Collaborating with a Traditional T1 Chip Company
-
Energy Consumption and Heat Dissipation Costs
Real-time computing requires continuously executing complex algorithms and data processing, which significantly increases energy consumption. In electric vehicles, energy consumption directly affects range, so to meet real-time performance, it often requires sacrificing some range. Regarding heat dissipation, high-performance computing generates a lot of heat, necessitating the use of efficient cooling systems, such as air or water cooling. Designing and maintaining these cooling systems also increases energy consumption, costs, and design complexity.
-
Software and Complex System Optimization Costs
Developing algorithms that meet real-time requirements requires meticulous optimization, which is often much more complex than ordinary algorithms. Development teams need to invest more time and resources in software optimization to ensure algorithms can execute effectively under strict time constraints. Adjusting scheduling strategies, model pruning, model quantization, reducing copies, etc., each represent engineering optimizations that are “understood in theory but cumbersome in practice.” At the same time, intelligent driving systems often rely on distributed computing or remote communication, ensuring low-latency transmission between systems often requires adopting more complex communication protocols and advanced network architectures, which also increases development and deployment difficulty.
In real-time computing for intelligent driving, system errors or delays can lead to serious safety incidents. To avoid these issues, stricter safety monitoring and system redundancy designs must be implemented, which inevitably increases system complexity. Furthermore, to ensure the system meets real-time requirements in various complex traffic scenarios, extensive simulation, testing, and validation are necessary. These testing processes are time-consuming and require specialized testing environments and tools.
In summary, meeting the real-time requirements of intelligent driving incurs high costs in hardware, energy consumption, algorithm and system optimization, and validation, which can be abstracted into three major factors: human resources, material resources, and time. However, these are indispensable conditions for ensuring the safe and efficient operation of intelligent driving vehicles in complex dynamic environments. It is not a simple binary issue of whether it works or not, but rather a balance and trade-off that must be refined considering price, development cycles, and functional cycle curves.
Figure 5: Diagram of Product Management Balance Trade-offs
05
How to Implement Real-Time Performance in Intelligent Driving Software
Having discussed the costs and practical balance of achieving real-time performance, it is clear that the specific implementation methods for real-time performance are not uniform, but there are also patterns to follow. Next, we will draw from project practice experience to explore some “common denominators” for implementing real-time performance in intelligent driving software.
-
Adopting Real-Time Operating Systems (RTOS)
A true real-time computing system requires an operating system that runs in real-time mode and user code that can achieve deterministic execution. Running deterministic user code on a non-real-time operating system or running non-deterministic code on a real-time operating system will not yield real-time performance. Common high-performance real-time operating systems include:
(1) RT_PREEMPT: A Linux kernel patch that modifies the Linux kernel scheduling to make it fully preemptible.
(2) Xenomai: A POSIX-compliant co-kernel (or hypervisor) that provides a real-time kernel working alongside the Linux kernel. The Linux kernel is treated as the idle task (lowest priority task) of the real-time kernel scheduler.
(3) QNX Neutrino: A POSIX-compliant real-time operating system suitable for critical task systems. It employs a microkernel architecture, isolating each application, driver, protocol stack, and file system in independent address spaces outside the kernel. Faulty components do not cause other components or the kernel to crash; they can be restarted immediately with minimal impact on system performance.
(4) VxWorks: An operating system with a monolithic kernel. Its architecture allows the entire operating system to operate in kernel space, meaning all kernel services and user services reside in the same address space, which speeds up the execution of the operating system. It provides debugging capabilities, performance monitoring, memory management, CPU scheduling, and other operating system functions through system calls.
Figure 6: Common Real-Time Operating Systems
-
Edge Computing and Localized Processing
For vehicle-cloud collaborative intelligent driving systems, to avoid delays from remote cloud computing, critical data should be processed in real-time locally in the vehicle (e.g., sensor fusion, path planning, and vehicle control), and then filtered data or special data triggered by shadow modes can be uploaded to the cloud for further processing. This approach places real-time tasks in localized processing within the vehicle, reducing data upload volume and network communication latency.
-
Using Efficient Optimization Tools and Algorithms
(1) Hardware Acceleration: Using GPUs, FPGAs, or dedicated chips (such as NVIDIA Orin’s DLA and VIC, TDA4’s VPA) to accelerate compute-intensive tasks like deep learning model inference and image processing. These hardware accelerators can significantly enhance real-time performance.
(2) Stream Processing: Stream processing technology allows sensor data to be processed simultaneously with collection, rather than waiting for complete frames or batches of data, significantly reducing perception latency.
(3) Lightweight Algorithm Design: To achieve real-time performance, algorithms in intelligent driving must be meticulously optimized to ensure efficient use of computational resources. Algorithms should avoid complex nonlinear computations and accelerate execution through mathematical simplification and parallelization. For large models or complex algorithm libraries aimed at generalization, necessary pruning is also required to focus efforts on critical tasks.
(4) Resource Isolation and Priority Management: Resource isolation is an effective means to address complex systems. For example, using Linux cgroups, QNX Adaptive Partition, or containers or virtual machines with real-time features can ensure that critical tasks have sufficient resource allocation, preventing resource competition from degrading real-time performance.
Figure 7: Diagram of Resource Isolation in QNX
-
Time Synchronization and Low-Latency Communication
(1) High-Precision Time Synchronization: By using Network Time Protocol (NTP) or Precision Time Protocol (PTP), ensure time uniformity among various modules to avoid clock inconsistencies in data processing and decision-making. Time synchronization allows different functions on different controllers to obtain nearly identical clock timestamps, enabling interpolation or fitting techniques in backend data processing to reduce real-time sensitivity.
(2) Low-Latency Communication Protocols: To reduce data transmission delays within and outside the system, selecting low-latency communication protocols (such as CAN FD, TSN Ethernet) to connect sensors, actuators, and central controllers can greatly enhance system response speed.
-
Redundancy and Fault-Tolerance Mechanism Design
By introducing hardware redundancy (such as dual-controller designs) and algorithm redundancy (such as parallel execution of the same task and selecting the fastest result), the risk of system failure due to faults or delays can be reduced, thereby enhancing system reliability and real-time performance.
-
Strict Testing and Validation
(1) Combining Simulation Testing with Real-World Road Testing: To ensure real-time performance meets requirements in different scenarios, extensive validation through large-scale simulation environments is necessary, combined with substantial real-world road testing. Simulation platforms can use tools like Carla, Apollo, and Prescan to simulate various traffic scenarios and validate the system’s real-time performance in complex situations. The recently popular world model can undoubtedly provide new powerful means for simulation testing.
(2) Real-Time Stress Testing: By designing extreme condition test scenarios, such as emergency braking, complex urban road conditions, and adverse weather, or increasing system load by running additional software that raises CPU usage or flooding the communication network with extra data, the system’s real-time performance and stability under high load can be tested to ensure effective operation under extreme conditions.
06
Conclusion
In the context of intelligent driving, real-time performance is not only a requirement for system performance but also an important guarantee for driving safety and reliability. To achieve real-time performance, optimization must be conducted across multiple dimensions, including hardware, software, and algorithms. Although this process faces high costs and technical complexity, these investments are essential to ensure that vehicles can make quick and accurate decisions in complex dynamic environments, ultimately driving intelligent driving towards a safer and more efficient future.
Figure 8: Mind Map of Real-Time Performance Discussion in Intelligent Driving
References:1.https://design.ros2.org/articles/realtime_background.html2.https://www.tttech-auto.com/knowledge-platform/real-time-scheduling-safe-autonomous-driving3.https://www.automation.com/en-us/articles/december-2023/continental-telechips-smart-cockpit-computersAuthor | JoneProduced by | Yanzhi Automotive
