The focus of building virtual platforms often lies in system operation, especially in the field of digital twins for automotive and aerospace applications. Only by ensuring that the virtual platform can operate the system correctly can the software run reliably on the virtual platform.
To ensure the system operates correctly, it is often necessary to address how to test for anomalies, faults, and other issues. This process is commonly referred to as “fault injection,” which involves deliberately introducing faults into the target system to accelerate system failure and evaluate the system’s reliability by observing its behavior in response to faults.
Fault injection can generally be divided into:
1. Hardware-based fault injection: This refers to introducing faults into the target system’s hardware using additional hardware.
2. Simulation-based fault injection: This involves creating a hardware simulation model of the target system using a standard hardware description language during the system design phase, and then inserting fault injection units within that model to achieve fault injection.
3. Software-based fault injection: This refers to simulating hardware or software faults by modifying the memory image of the target system according to a specific fault model.
It is well known that performing fault injection directly on hardware is both complex and expensive, and it is difficult to precisely control the timing and location of the injection, which can easily damage the target system’s hardware. The virtual platform is a solution to circumvent this issue.
Fault Injection in Virtual Platforms
As a digital twin of the hardware system, the virtual platform can provide fault injection at any state and any location, such as hard disk failures and Denial of Service (DoS) attacks, thus conveniently allowing for cost-effective, highly reliable, and infinitely repeatable fault injection for software.
The advantages of using virtual platforms for fault injection include but are not limited to:
1. The environment of the virtual platform is completely deterministic, making fault analysis simpler as environmental factors do not need to be considered;
2. Automated multi-combination fault injection can be achieved simply by inserting scripts, with testing volume and speed even surpassing that of the hardware itself;
3. Using virtual platforms for fault injection can save approximately 90% of the costs compared to direct fault injection on hardware.
Generally, fault injection in virtual platforms requires extending the system model. Certain types of faults can be introduced through simple model state changes (such as modifying register values, memory contents, etc.), but specific faults usually need to be present in the model. In practice, in addition to pre-designed, intentional fault injections, there will also be unexpected faults that cannot be clearly defined as “faults.”
Unexpected “Clock Drift” Events
The operating system on the virtual platform suddenly crashes, displaying a “divide by zero” fault—most likely due to clock drift.
We know that during actual hardware operations, processor cores on different chips typically do not exhibit exactly the same timing due to individual clock cycles often deviating at certain points in time, commonly referred to as “clock jitter,” while “clock drift” refers to deviations caused by physical design and other reasons.
In fact, time on the virtual platform is tightly coordinated, and the clock drift that is commonly present in real hardware is not modeled, leading to issues that do not occur on hardware but only appear on the virtual platform.
Unexpected “Simultaneous Insertion of Two Boards” Events
In tests to determine whether the system software can correctly detect and activate a newly inserted circuit board (specifically referring to hot-plugging a circuit board onto a powered backplane), when two circuit boards are simultaneously added to the system on the virtual platform, the software crashes immediately. By hot-plugging two circuit boards simultaneously on the corresponding real backplane and comparing stack traces, it can be determined whether the bug lies in the virtual platform or the system itself. In fact, at the moment of completing the countdown and simultaneously inserting the two boards, the backplane was burned out.
Although the probability of such a situation occurring is low, the cause of the fault could be a short circuit or overload, but regardless of the outcome, it is a result of a situation not considered during the initial design—perhaps this is what developers truly need to pay attention to.
▲ Dual board hot-plug fault
SkyEye: A Practical Tool for Fault Injection
The Tianmu All-Digital Real-Time Simulation Platform SkyEye is a hardware behavioral simulation platform based on visual modeling, allowing engineers to inject faults at will in a virtual testing environment, pause or reverse execution at any time, and stably reproduce defects, helping to break through hardware configuration boundaries and solve software issues.
For more content related to fault injection,please click to view >>
References:
[1] Tong Xiangpeng, Wang Wenfeng. System Testing Methods and Applications Based on Fault Injection [C]// 2013 First China Aerospace Science and Technology Conference. 0.
[2] Intentional and Accidental Fault Injection in Virtual Platforms
END
Recommended Reading
Product Introduction
DiJie Software is a leading provider of embedded DevOps solutions in China, with the mission of “Supporting the Leap of China’s High-End Equipment Manufacturing Industry” and aiming to “Become a World-Class Basic Software Supplier”.
