Abstract
This article discusses the definition of guardian threads, one of the three major threads in embedded systems. What is its status among various threads? What role does it play? How can this thread be integrated into practical engineering? What details need to be considered?
Definition of Guardian Threads
Definition
A thread that operates independently of user threads, runs continuously, and periodically monitors the entire system for abnormal states to maintain system safety.
Safety-critical systems must always exhibit predictable and reliable behavior; otherwise, the system may fail, putting lives at risk. Even with the most rigorous testing, it is impossible to exhaustively test all input combinations, and the system may behave in unexpectedly dangerous ways when affected by unseen input combinations during development.
Safety-critical systems typically employ fault-tolerant designs, ensuring that they do not encounter unacceptable risk incidents when faced with unexpected inputs. Guardian threads are part of this fault-tolerant design.
Abnormal States
When mentioning guardian threads, one might first think of watchdog threads: they reset the system when a task is blocked or the program hangs, thus protecting the system. However, feeding the watchdog is just one of the functions of guardian threads. The abnormal states they address include, but are not limited to:
- Program hang
- Thread deadlock
- Critical thread crash, such as unexpectedly entering a termination state
- Critical thread significantly blocked but not exceeding the maximum allowed time
- Stack exceeding warning threshold
- CPU exceeding warning threshold
- Other critical system parameters, such as CPU temperature, etc.
The first three represent service stoppage and must be included, while the others are set based on actual needs. All abnormal behaviors that can affect the normal provision of services by the system should be included in the list of abnormal states. Abnormal states can be set at various levels, and different levels of abnormal states are monitored and then handed over to the problematic thread for processing.//Product requirements – Function definition – Abnormal state definition and classification – Abnormal response
Characteristics
- Independent of main thread dependencies, runs continuously, and its lifecycle depends on user threads; when user threads end, this thread also terminates
- Periodically performs safety checks, with intervals not exceeding the maximum allowed time for abnormal states
- Inheritance: threads related to the guardian thread created also belong to the guardian thread (a single guardian thread may not meet the needs for different abnormal states)
- Monitors the entire system rather than just a single safety component
- Runs in the background, and during monitoring, it does not affect the normal operation of user threads
Design Framework of Guardian Threads
Classic Design Framework
Taking program hangs, thread deadlocks, and other service stoppage-type abnormal states as monitoring targets, we can use the classic watchdog function as an example:
- Driver initialization, initializing the watchdog, setting the maximum feeding time
- Creating the guardian thread, enabling the watchdog, and starting operation
- Creating user threads and registering them with the guardian thread as monitored objects
- User threads operate normally, periodically reporting to the guardian thread, which completes safety checks
- If a user thread is abnormal and does not report to the guardian thread on time but has not exceeded the specified time, the guardian thread records the abnormal status and hands it over to the problematic thread for decision-making
- If a user thread is abnormal and does not report to the guardian thread on time, exceeding the specified time, the guardian thread stops feeding the watchdog, controls the settings to enter an emergency safety state, and forcibly restores the system.
The above describes the classic design framework of guardian threads, which seems to monitor user thread hangs and then restart the device. Is directly restarting the device sufficient for such dangerous situations of service stoppage?This process overlooks the following factors:
- Priority of the guardian thread
- Restarting the device is not necessarily safe
- During abnormal handling, no traces are retained; theoretically, the issue will eventually recur. How to leverage the abnormal thread to locate the essence of the problem is key.
Priority
The priority of the guardian thread directly determines whether it can effectively monitor abnormal states. Let’s hypothesize in order:
- Guardian threads run in the background, so they cannot interfere with user threads, hence a low priority?
- Guardian threads monitor the entire system, which is of high importance, so a high priority?
Low Priority
If the priority of the guardian thread is lower than that of user threads, under normal circumstances, user threads use semaphores, message queues, and other communication methods to report safety to the guardian thread, and the guardian thread feeds the watchdog regularly, which is fine. However, during system stress testing, if the preceding user threads run for too long, it may lead to the guardian thread starving and being unable to feed the watchdog, causing the system to reboot incorrectly.
High Priority
If the priority of the guardian thread is higher than that of user threads, under normal circumstances, there is also no problem. However, its high priority may preempt the normally running user threads, making it unable to detect the starvation state of lower-priority user threads, ultimately leading to system performance degradation or the system becoming abnormal while the device still cannot reboot.
Demonstration
So what should the status of guardian threads be? To address this question, let’s look at the position of the “watchdog interrupt” in the interrupt vector table (independent watchdog interrupt). The following image shows part of the interrupt vector table for STM32:
It can be clearly seen that the window watchdog WWDG_IRQHandler is positioned first among user-level interrupts, aside from system exceptions. This means that under the same interrupt priority configuration, the system will prioritize running this interrupt.
Thus, the answer seems evident: if the priority of the guardian thread is set too low, during high user thread loads and prolonged blocking times, it indeed induces starvation of the guardian thread, leading to erroneous system reboots. Some engineers even implement guardian functions in idle tasks, which further reduces the probability of errors. How should we address the issue of high-priority guardian threads potentially affecting the normal operation of user threads?
First, guardian threads generally run periodically, and their running cycle is much longer than that of user threads; user threads operate on a millisecond scale, while guardian threads may operate on a second scale or longer. The two running cycles are not on the same order of magnitude, so occasionally running a guardian thread will not impose much load on the system.
Note: The actions of guardian threads should not be overly complex; even if they run occasionally, they must not affect the normal timing of user threads during this brief period.
Then, user threads can notify the guardian thread of their status information through mechanisms such as sending semaphores or events, which can be set so that the guardian thread only starts after all monitored user threads have sent information, remaining in a blocked state beforehand. Therefore, it is generally unlikely for user threads to starve, and the situation where the guardian thread, relying on its higher priority, runs first while user threads fail to send status information in time will not occur.
Safety Protection
When the guardian thread detects extreme abnormal states that trigger the watchdog to reboot the device, it should switch the entire system to a safe state in advance. Any safety-critical system must include a safe state; at this point, we need to utilize the last 10% of the time (for example, based on requirements) before the specified feeding cycle arrives to notify the problematic thread or the guardian thread itself to complete the safe state switch. For instance, the motor of a machine tool should immediately stop running to prevent the motor from exceeding safety limits during the reboot process.
Sometimes, simply rebooting the device does not end the issue, as some devices need to run continuously. For example, an ICU ventilator should be able to immediately return to its pre-abnormal working mode and continuously supply oxygen to the patient to prevent suffocation if an automatic reboot occurs while the doctor is absent.
Therefore, the behavior after rebooting is also crucial; generally, this is considered within the scope of risk analysis-related documents, and thorough risk analysis and response should be conducted.
Location Tracking
To proactively solve problems rather than passively defend, it is necessary to notify the logging thread to complete problem recording before the device reboots after an abnormal condition occurs. The records should include:
- Key system parameters: runtime, temperature, working mode, status of each user thread, etc. (based on actual needs, primarily reflecting the overall characteristics of the system)
- Stack call information: recording specific stack information so that maintenance personnel can determine the specific information of the thread during the abnormality to trace where the problem occurred.
It is essential to record this in non-volatile media, and since such writes take longer, please allow for sufficient and safe time.
Supplement:
For hot reboots (non-power-off reboots) caused by the watchdog, data can be recorded in RAM, which is very fast. When storing data in RAM, the storage address must be predetermined; it must not be within the normal execution address range of RAM, or else the watchdog will refresh and initialize this area of data after rebooting the device. Space can be allocated in unused RAM areas. However, some devices may refresh all of RAM during hot reboots, which cannot be avoided.
In the program’s memory layout, there will always be some RAM areas not occupied by .data, .bss, heap, and stack; we can designate these areas specifically for storing “reboot logs” or “status information.” As for how to find the reserved RAM area, one can research as there is plenty of information available.
Finally, note that the saved RAM area needs to have MPU-type protection measures to prevent accidental writes by other programs, and remember to unlock the MPU area before normal writes.
Low Address
+-----------------------+
| .text (Flash) | <- Stores code and constants
|-----------------------|
| .data (Flash) | <- Stores initial values of .data segment (only occupies Flash)
+-----------------------+
| |
| Reserved | <- May have other segments or reserved areas
| |
+-----------------------+
| |
| RAM |
| |
+-----------------------+
| .data (RAM) | <- Stores initialized variables at runtime (values copied from Flash)
|-----------------------|
| .bss (RAM) | <- Stores uninitialized/zero-initialized variables at runtime (cleared at startup)
|-----------------------|
| |
| Heap | <- Dynamically allocated memory, grows towards high addresses
| ↑ |
| | |
| | |
| ↓ |
| |
| Stack | <- Function calls and local variables, grows towards low addresses
|-----------------------|
| Interrupt Vector Table| <- Interrupt vector table (usually located at the start of Flash or RAM)
+-----------------------+
High Address