People often ask me how to conduct functional safety testing and what the differences are compared to previous testing methods. I believe there isn’t much difference, but I place significant emphasis on Fault Injection Testing (FIT). This testing method is mentioned in Part 4 and Part 6, generally requiring ASIL to be ++. In fact, it is difficult to justify not performing fault injection for other levels as well, because the core of functional safety is the safe handling of faults.
According to ISO 26262, fault injection testing is a software testing technique used to assess the robustness of safety-critical systems by deliberately introducing faults or errors into the system. The goal is to verify that the system can detect, respond to, and recover from faults in a safe and controlled manner, without compromising the functional safety of the system. Fault injection testing is part of the overall software verification and validation process to ensure that the system meets the required safety integrity level (ASIL).
In ISO 26262, several types of fault injection testing can be performed, including:
Data-based fault injection: Intentionally introducing faults or errors into the system by manipulating input data, altering internal states, or triggering error conditions.
Structural fault injection: Introducing faults or errors into the system by modifying source code, changing execution paths, or interrupting memory access.
Environmental fault injection: Exposing the system to environmental conditions that may cause faults or errors, such as extreme temperatures, voltage fluctuations, or electromagnetic interference.
Time-related fault injection: Altering the timing of events within the system, such as delays or race conditions, to trigger faults or errors.
The meaning of fault injection varies slightly at different testing levels. At the unit testing level, the ISO 26262 standard summarizes fault injection testing as “injecting arbitrary faults, such as corrupting variable values, introducing code mutations, or corrupting CPU register values.” The concept of integration testing is similar, but the methods vary slightly. Arbitrary faults are injected by corrupting software and hardware components to test safety mechanisms.
Essentially, fault injection testing aims to analyze the system’s ability to detect faults and respond to them in a way that transitions to a safe state.
Once the functional safety requirements are clear and there is a clear understanding of their allocation, we can compile a list of fault types. With this list and a complete picture of the system architecture, we can design fault injection test cases.
The ISO 26262 standard does not provide detailed instructions on how to conduct fault injection testing but recommends using certain specific methods to inject faults into the system. To ensure that fault injection testing covers the safety requirements at all stages of the safety lifecycle, ISO 26262 suggests conducting fault injection testing during unit, integration, and functional testing. Therefore, before detailing the execution methods of the tests, let us first examine fault injection at each stage.
At the unit level, testing aims to analyze the gaps in fault handling and fault tolerance at an early stage, specifically at the code level (unit stage). A unit at the code level has several elements; each relevant element must have faults injected to achieve sufficient coverage.
Unit Testing
Variables:Corrupting variable values to verify whether faults in the variable can be detected, prevented, or corrected. For example, if a safety airbag system detects a sudden drop in vehicle speed, the vehicle speed variable can be corrupted to validate the test results.
Interfaces:Each unit provides specific interfaces to other units for initiating certain functions or sharing data. Testing these interfaces by injecting faults such as early/late calls helps analyze their behavior.
Data received by interfaces: What happens when the data received by a unit interface exceeds boundary values? For example, if a speedometer receives speed data of 190 km/h while its maximum display capability is 170 km/h, this situation can be tested by corrupting the data provided by the unit interface.
Functions executed under different instances: Functions can be executed on a unit basis according to system requirements, which can be sequential, trigger-based, or executed at a predetermined frequency. Units that meet ASIL D must be tested according to that sequence/order.
Integration Testing
Fault injection testing at the integration level is a verification of the software architecture, where faults injected into the integrated software aim to test the interfaces responsible for implementing safety mechanisms, specifically whether they correctly detect and handle faults such as memory corruption and data corruption. Fault injection testing at the integration level can help detect common causes of faults and assist in achieving uninterrupted operation.
Fault injection interfaces must be validated to ensure they can:
(1) Provide the correct state of safety objectives
(2) Achieve safety objectives upon fault detection
Fault injection in functional testing (embedded software testing)
Software safety requirements must be tested at this stage, and the HSI is also complete. The faults injected during functional testing are of the “functional” type. Error injection in communication protocols (such as CAN, LIN, or Ethernet) is the most common testing method.
As part of the verification and validation process, development teams can perform data-based fault injection to assess the system’s ability to handle unexpected input data. For example, they can simulate faults in the camera system providing input for ADAS and observe how the system responds.
Additionally, teams can perform structural fault injection by altering the system’s source code or execution paths to introduce faults and evaluate the system’s ability to detect and recover from these faults. For ADAS functions, they might modify the code responsible for detecting lane departures and observe how the system responds.
Summary
Fault injection testing, although strongly recommended only for ASIL D systems, is a reliable method for checking system robustness and fault tolerance early in the development cycle. After conducting a cost-benefit analysis, even lower ASIL systems can implement it to enhance reliability and robustness.
It is also worth mentioning that larger projects or teams often perform automated testing, even running test cases overnight. Unfortunately, FIT often causes the automated environment to crash, making it difficult to maintain automation; this requires manual verification, which can be labor-intensive and relatively less efficient.