Intelligent Terminal Control System, “Mysterious Freeze” After Running for Several Hours
We developed an intelligent gateway based on STM32 + FreeRTOS for a client, featuring:
- Multithreaded tasks: sensor data collection, CAN communication, UI refresh, log writing
- Using mutexes to protect shared resources (such as LCD, Flash, serial port)
- Priority design was reasonable (or so we thought)
The system performed well in laboratory tests, but within a week of mass production, the client reported:
“The device sometimes freezes after running for a few hours, the UI does not refresh, communication drops, and the only solution is to restart.”
The problem was extremely bizarre:
- The watchdog was ineffective (tasks were still running, but “no activity”)
- No interruptions in logs
- Hard to reproduce on-site
- The team had not slept well for a week
🔍 Initial Investigation: All Tasks are “Stuck”, but the System Has Not Crashed
We captured <span>uxTaskGetSystemState()</span>, idle task status, and CPU usage, and found:
- All tasks were in
<span>eBlocked</span>or<span>eSuspended</span> - No tasks were
<span>Running</span> - CPU utilization was extremely low
- FreeRTOS had not crashed, nor had it entered HardFault
The system seemed to be “paused”, but who pressed the pause button?
🧠 Suspicious Directions: Locks? Queues? Priorities? Task Scheduling Imbalance?
We suspected a deadlock, so we checked each one:
xSemaphoreTake(xMutex, portMAX_DELAY);
...
xSemaphoreGive(xMutex);
But all locks appeared in pairs, theoretically, there should be no deadlock.
We even tried:
- Disabling some tasks → Deadlock still occurred
- Adding logs for investigation → Log task itself was stuck
- Enabling stack overflow hooks → Not triggered
- Watchdog timeout not triggered → Some tasks were still feeding the watchdog
The team was completely trapped in the fog of the “phantom deadlock”.
💡 Turning Point: Introducing Segger SystemView, the Truth Finally Emerges!
We decided to try Segger SystemView, which is one of the official real-time tracing tools for FreeRTOS.
The steps are as follows:
Add SystemView support in the STM32 project:
SEGGER_SYSVIEW_Conf();
Enable macros in FreeRTOSConfig.h:
#define configUSE_TRACE_FACILITY 1
#define configUSE_STATS_FORMATTING_FUNCTIONS 1
Connect to PC via serial or RTT for real-time tracking of task switches, events, and lock contention
🎯 Finally Discovered the Real Culprit!
During SystemView recording, we discovered an important sequence of events:
- Low-priority task A (log task) acquired the serial port mutex
<span>xUartMutex</span> - High-priority task B (CAN communication task) attempted to acquire that lock and became blocked
- Medium-priority task C (UI refresh)kept running, occupying the CPU
- Task A could not run → Could not release the lock → Task B waits forever → Deadlock!
🔥 The Real Problem:Priority Inversion
Low-priority task A held the lock → High-priority task B was blocked → Medium-priority task C preempted the CPU, causing A to never get a chance to run.
This phenomenon is a classic priority inversion.
FreeRTOS uses the priority inheritance mechanism by default to resolve this:
When a low-priority task holds a mutex, if a higher-priority task requests that lock,the low-priority task will temporarily “raise its priority” until it releases the lock.
But we configured it incorrectly!
📛 Fatal Configuration Error: Wrong Type of Mutex Used!
The code we used to create the lock was:
xSemaphoreCreateBinary(); // ❌ Regular binary semaphore, no priority inheritance
The correct one should be:
xSemaphoreCreateMutex(); // ✅ Mutex with priority inheritance
Moreover, the FreeRTOS documentation clearly states:
Only locks created with xSemaphoreCreateMutex support priority inheritance.
✅ Fix Plan: Break the Phantom Deadlock Curse!
🥇 First: Use <span>xSemaphoreCreateMutex()</span> for All Mutex Resources
We searched the entire project for all instances of <span>xSemaphoreCreateBinary()</span> used for resource protection and uniformly replaced them with:
xMutex = xSemaphoreCreateMutex();
Binary semaphores are suitable for event synchronization, not for resource mutual exclusion!
🥈 Second: Reorganize Task Priorities to Avoid “Intermediate Task Preemption”
We adjusted the task priority structure:
| Task | Priority (Before) | Priority (After) |
|---|---|---|
| CAN Communication Task | 3 | 3 |
| Log Writing Task | 1 | 2 |
| UI Refresh Task | 2 | 1 |
Ensure that low-priority tasks waiting for high-priority tasks can get a chance to run as soon as possible.
🥉 Third: Continuously Integrate SystemView to Establish Task Behavior Baseline
We integrated SystemView into the continuous integration process, recording task run graphs daily to prevent “behavior drift”:
- Run time of each task
- Lock wait/release time
- Interrupt frequency / context switch frequency
Once abnormal behavior occurs, an alert is triggered immediately.
🧪 Test Results After Optimization
| Project | Before Optimization | After Optimization |
|---|---|---|
| Deadlock Frequency | 1-3 times per day | 0 (running continuously for 45 days) |
| Freeze Duration | Indefinite (requires restart) | None |
| Watchdog Trigger | ❌ Not triggered | ✅ Can recover from exceptions |
| Data Transmission Stability | Fluctuating | Stable throughout |
| Team Man-Hours | 1 month chasing ghosts | ✅ SystemView identified in 5 minutes |
🧠 Summary: The Real “Deadlock” in FreeRTOS May Not Be Due to Code Errors, but Rather the Wrong Type of Lock Used!
| Problem | Cause | Solution |
|---|---|---|
| Random Freezes | Priority Inversion Deadlock | ✅ Use Mutex |
| Lock Release Without Opportunity | Intermediate Priority Task Preemption | ✅ Priority Inheritance Mechanism |
| Unable to Locate | Deadlock is not a crash, the system is still “running” | ✅ SystemView Tracking |
| Deadlock Workaround | Correct Lock Usage + Task Elevation | ✅ Dual Improvement in Performance/Stability |