Challenges and Enhancements for Functional Safety in Linux Systems

1. Insufficiencies and Challenges of Functional Safety in Linux Systems

The functional safety requirements for Linux systems in autonomous driving systems must meet the ISO 26262 ASIL B+ level (specific requirements depend on the actual automation level of intelligent driving and the functional allocation on the SOC). This is a significant pain point in the industry. The gap between the open-source Linux system and existing safety standards is mainly reflected in the following aspects:

1.Absence of a Trustworthy Design and Verification Process:The development requirements for software in ISO 26262 primarily stem from the design and verification processes (part 6), as software does not inherently fail randomly; failures are systemic and caused by human factors. Therefore, robust processes are needed to ensure and eliminate these failures. Since Linux was released before relevant safety standards, the kernel’s source code comes from community contributions, and some parts of the code are ambiguous. Additionally, the design information required for development and testing is often missing, and simple documentation cannot support the reliable design and verification basis defined in ISO 26262 for QM or ASIL.

2.Absence of Accountability: The development of open-source Linux relies on extensive contributions from the open-source community. Contributors use a peer-review-like mechanism through mailing lists, and maintainers of each subsystem value their reputation and the trust of lower-level integrators. However, this does not guarantee the quality and safety of code or functional modules for specific use cases. Furthermore, Linux release versions may include incomplete features and unresolved issues, leading to systemic failures in functional safety. To ensure the stability and safety of Linux, users and developers must take on the responsibility for secondary development, testing, and problem resolution.

3.System Complexity: Given the large codebase of the Linux kernel, which is continuously expanding, the cyclomatic complexity of the software may exceed the specifications in typical safety development processes. The execution of unit tests faces significant workload challenges, making it difficult for even experienced engineers to deeply understand all existing code, especially the overall software architecture. This is a key factor in effectively addressing potential vulnerabilities and preventing systemic failures in functional safety.

4.Real-time and Reliability:The native scheduling module of the Linux kernel cannot ensure that safety-critical tasks have hard real-time guarantees. Although real-time patches can significantly improve the scheduling real-time performance of Linux, it is necessary to select a reliable kernel branch version and conduct extensive testing to ensure overall system reliability.

5.Correlational Failures: The Linux system is a monolithic kernel system, with many subsystems running in kernel mode address space. Although the MMU mechanism isolates kernel mode from user mode, security-related software modules cannot be absolutely isolated from non-security-related modules in kernel mode. Currently, there is no solution for memory isolation in kernel mode within the community. It is worth noting that as a monolithic kernel, if correlational failures are not resolved, secondary development of security mechanisms in the native kernel may lead to more systemic failures and further reduce the reliability of Linux.

2. Attempts and Deficiencies in Adapting Linux for Safety-Critical Applications

1. Technical Attempts

1)Brute Force (Brute Force Authentication): Traditional methods are used to perform security certification on the Linux kernel and packages. However, due to the large codebase, along with user-space public libraries and some open-source software, the certification cost is extremely high. It requires code reuse across multiple projects to dilute costs, and due to the open-source nature of the projects, the invested assets must return to upstream, which creates issues in the business model.

2)Strip Down (Code Reduction): Reducing the codebase of the Linux kernel and packages to retain only the essential parts for device operation, thereby reducing the certification workload. However, this operation may be suspected of transforming a monolithic kernel into a microkernel, losing its ecological advantages. Additionally, it may lead to Linux code branches (forks), creating technical debt, requiring re-analysis, customization, and certification for subsequent community updates, resulting in high maintenance costs.

3)Kernel Modifications (Kernel Modifications): Modifying the kernel to achieve real-time performance, but without a complete correlational analysis, each modification introduces new security risks and increases development and certification complexity.

4)Safety Monitor (Safety Monitoring): Using a safety-certified monitor to oversee non-safety-certified applications on Linux. Wind River attempted this 10 years ago and found that the monitor required complex proprietary technology, affected system performance, and needed proprietary drivers to be added in the hypervisor, making it impractical.

2. Standard-Level Attempts:

Modifying safety certification standards to adapt to existing software like Linux, which is already widely used in the industry. A typical example is ISO/PAS 8926, which proposes classifying “existing architectural elements” based on software source and complexity (similar to complexity assessments in hardware component identification). This aims to allow well-tested Linux to be used in ISO 26262 certified products without complete certification. The modifications proposed by this publicly available standard may be included in the third edition of ISO 26262 in 2027, but this approach may undermine confidence in existing safety standards, posing risks.

3. Enhancements for Safety and Real-Time Performance in Linux

1. Execution Time Safety

The core objective of execution time safety is to ensure that safety-critical tasks are reliably executed within specified time frames, meeting hard real-time and priority constraints. This can be optimized and enhanced through the following:

1)Selecting community long-term support version branches (LTS) to improve version stability.

2) Adapting and validating the PREEMPT_RT patch for soft real-time transformation. Converting most interrupts to threads and moving time-consuming processing to user space (interrupt bottom halves) to reduce the execution time of kernel-mode interrupt service routines (ISR).

3)Selecting and configuring appropriate real-time schedulers (FIFO/RR) to ensure tasks can be executed efficiently and in real-time.

4)Avoiding memory allocation in RT contexts (such as during RT thread execution) to prevent waiting and disrupt determinism and responsiveness. It is recommended to limit memory allocation to the system initialization phase; parse symbols during system startup; and lock pages after allocation using mlock_all() (with MCL_CURRENT/MCL_FUTURE flags) to prevent paging.

5) Using high-resolution timers to enhance the real-time performance of tasks that require timing execution. Reducing scheduling clock interrupts (e.g., CONFIG_NO_HZ_IDLE: stop periodic ticks when the CPU is idle).

6) Disabling CPU frequency scaling.

7) Considering the security of public libraries and dynamic link libraries (from aspects such as reliable loading and linking, interface robustness, memory resource management, exception handling, and thread safety).

8) Partitioning interrupt priorities (e.g., IRQF_TRIGGER_FALLING) to ensure that safety-related interrupts (such as sensor interrupts) are prioritized.

9) Optimizing the interrupt handling process to reduce interrupt handling overhead and latency.

10) Configuring interrupt affinity to bind specific hardware interrupts (IRQ) to specific CPU cores, preventing non-safety interrupts from interfering with safety interrupt handling.

2. Execution Order Safety

The core objective of execution order safety is to ensure the safety and predictability of resource access in multi-tasking and core-sharing environments, avoiding system failures caused by deadlocks, data races, or synchronization delays. This can be optimized and enhanced through the following:

1) Using semaphore and mutex mechanisms for concurrency control to ensure that real-time tasks can cooperate correctly and achieve mutual access to resources.

2) Constraining the CPU/memory usage range of tasks using cpusets.

3) Performing RCU optimizations, CONFIG_RCU_NOCB_CPU: when enabled, it offloads RCU callback management from clock ticks to unbounded kernel threads, avoiding RT CPU jitter; CONFIG_RCU_BOOST: enabling RCU priority boosting to avoid blocking RCU readers from extending the grace period, preventing high load from blocking RCU callbacks (recommended to enable for RT applications or high-load scenarios).

4) Designing reasonable priority mechanisms (e.g., priority inheritance, priority ceiling mechanisms) and lock mechanisms (spin operation delays, spin wait optimizations) to avoid deadlocks, data races, or synchronization delays.

2. Safety Monitoring

The core objective of safety monitoring is to ensure that interrupt handling does not interfere with the execution of safety-critical tasks, avoiding system failures caused by interrupt storms or processing delays. This can be optimized and enhanced through the following:

1) Monitoring critical interrupts to check if the interrupt frequency meets expectations, determining if there are excessive interrupts or interrupt storms. Monitoring interrupt entry and exit times to check for timeouts, and monitoring interrupt priority switches to see if they exceed the nested level limit, determining if there are interrupt nesting issues.

2) Monitoring critical tasks (including task execution time monitoring, kernel critical task execution timing monitoring, kernel resource statistics and monitoring, interrupt execution, memory, I.O., network, driver, etc. monitoring).

3. Secure Communication

The core objective of secure communication is to ensure the integrity, confidentiality, and real-time performance of data transmission within the system (e.g., between kernel mode and user mode, inter-process) and between cores and outside the system (e.g., vehicle bus). This can be optimized and enhanced through the following:

1) Ensuring the integrity of IPC communication code.

2) Ensuring the integrity of various network communication protocols in user space, such as DDS, Some/IP, etc.

3) Verifying the integrity of safety-related data transmission, including configurations for E2E, QoS, etc.

4. Address Space Security

As a monolithic kernel system, a significant amount of code runs in kernel mode address space, which leads to functional safety software modules being unable to isolate from other software modules to avoid interference. It is necessary to achieve memory isolation for safety-critical modules to address the “correlational failure” issue of monolithic kernels. This can be optimized and enhanced through the following:

1) Memory isolation mechanisms: Implementing kernel-mode memory isolation solutions (such as page table isolation, hardware virtualization technologies) to restrict the memory access range of safety modules.

2) Managing memory usage quotas and stack monitoring, such as using CANARY stack overflow sentinels.

3) Memory protection: Enabling memory protection units (MPU) or memory management units (MMU) with permission controls, setting read-only / non-executable (NX) attributes.

4) Controlling program access permissions to system resources through privilege level settings (distinguishing between kernel mode and user mode).

5) Providing a virtual file system to unify the file system interface, hiding differences between different file systems, replacing user program requests for kernel service interfaces (file operations / device control).

6) Providing integrity checks for the file system to validate configuration data.

5. Secure Boot

The core objective of secure boot is to ensure that there are no latent random hardware failures affecting safety during the driving cycle, while ensuring that the software loaded during the boot process (such as Bootloader, kernel, drivers) has not been tampered with, establishing a complete trust chain from hardware to applications, including:

1) The integrity of driver frameworks and various drivers provided by chip manufacturers (chip BSP).

2) Reliable boot verification of the system: including hardware fault checks (PBIST, LBIST, MBIST, etc.) and software checks (init process verification, C library critical file verification, critical configuration data checks).

3) Trusted boot verification of the system: SecurityBoot step-by-step verification, ePBF loading signature verification, combined with KASLR (Kernel Address Space Layout Randomization), ensuring the integrity check of the kernel image during secure boot.

6. Security Tools

Providing tool support during the development, testing, and auditing phases to ensure that the Linux system meets functional safety standards, including but not limited to the following:

1) Compilation, configuration, and generation tools, such as boomtimes generation tools, supporting GCC, and security compilation options.

2) Security proofs for the Linux C/C++ standard libraries used.

3) Static analysis tools, including tools for checking security flaws in kernel code and analyzing scheduling real-time performance.

4) Testing and validation tools, such as KUnit (unit testing), LTP (system testing) to validate the effectiveness of security mechanisms, simulating fault injection (e.g., memory errors).

5) Configuration detection tools, such as checksec.sh to check kernel compilation options (such as stack protection, RELRO), ensuring that security hardening measures are effective.

References:

1. “Research Report on Functional Safety of Autonomous Driving Linux Systems 1.0″ – AUTOSEMO Community

2. “Functional Safety of Intelligent Vehicle Software” – Wu Dandan

3. “Balancing Operating Systems for Safety-Critical Applications” – WNDRVR

4. “RT Linux in Safety-Critical Systems: The Potential and the Challenges”