The incident occurred very suddenly.
That morning, shortly after eight, I had just brewed a cup of tea and was about to sit down when a call came in from the warehouse, saying that all the AGVs were unresponsive, their wheels trembling, with some smoking and others completely powered down.
When we arrived on site, we saw AGVs one after another incapacitated on the ground, some with jammed wheels, others with their driver boards charred black, and the air filled with the smell of burnt circuit boards. The dispatch system had already triggered an alarm, and the production line was halted.
Initially, we suspected unstable voltage or communication errors, but after investigation, we found that the issue was not trivial; the root cause lay in the PWM configuration of the STM32.
Controlling the Motors
Our AGV system uses the STM32F407, controlling the motors through complementary PWM output to manage the upper and lower arms of a three-phase brushless motor. The driver board utilizes the IR2101 chip, which includes basic bootstrap and overcurrent protection but lacks automatic dead time functionality.
Each AGV controls two sets of motors, with three pairs of MOSFETs managing the UVW three-phase output. The STM32 outputs complementary PWM waveforms via timer TIM1, alternating the conduction of the upper and lower arms.
These PWM signals are complementary, meaning that when the upper arm is conducting, the lower arm must be off, and vice versa. If both are on simultaneously, it results in a short circuit, allowing current to flow directly from the power supply, potentially burning out the MOSFETs or even causing the board to explode.
To prevent this situation, we originally included dead time in the PWM waveforms.
Those Few Nanoseconds Became the Catalyst
Dead time is not complicated; it simply means waiting a few tens to hundreds of nanoseconds after turning off the upper arm before turning on the lower arm. This ensures that the two MOSFETs do not conduct simultaneously at the same clock point.
The functionality provided by the STM32 is hidden in a register called BDTR, where the DTG bit is responsible for this. We had previously configured it correctly, typically setting the dead time around 500ns, which is safe and does not overly hinder PWM precision.
The problem arose during a version update.
A young colleague, while debugging the PWM frequency, felt that the dead time affected output precision and commented out the dead time configuration code, leaving only the main output enable:
TIM1->BDTR = (1 << 15); // Enabled MOE, did not set DTGThis line of code seemed fine; the PWM waveform still appeared, and the complementary output was present, but in reality, the dead time was eliminated.
The result was that the upper and lower arms were conducting almost simultaneously at the transition points of each PWM cycle.
During laboratory testing, there were no issues due to light loads and stable power supply, but when deployed in the warehouse with dozens of AGVs running simultaneously, the increased current and load caused the MOSFETs to fail.
Waveform Analysis Reveals the Truth
We connected the burnt main control board of the AGV to an oscilloscope to examine the waveform, and the problem was immediately apparent.
The complementary PWM waveforms had almost no gaps between them, and the switching points appeared as a straight line. The conduction times of the upper and lower arms overlapped, creating an instantaneous short circuit that pulled the current to its maximum.
The simple current limiting on the driver board could not contain such a large surge current, leading to the MOSFETs burning out and the motors twitching, causing the control system to crash.
In that moment, the few nanoseconds of dead time became the fatal flaw of the entire system.
What We Did After the Incident
We quickly rolled back the code and reconfigured the dead time:
uint8_t dtg = calculate_deadtime_ns(500); // Convert to DTG encoding
TIM1->BDTR = (1 << 15) | dtg;This <span>calculate_deadtime_ns</span> is a simple conversion function we wrote to calculate the corresponding dead time encoding based on the timer frequency and prescaler. The DTG field of the STM32 is not written directly in nanoseconds but encoded according to a segmented rule, so it needs conversion.
To prevent similar incidents from occurring again, we also implemented several measures:
- Standardized the PWM initialization process, encapsulating all PWM-related configurations into functions and prohibiting direct register manipulation;
- Added dead time checks before BDTR configuration, triggering an alarm and exiting if the dead time is set to 0;
- Incorporated a brake input on the driver board, which automatically pulls down the PWM output if the current exceeds a certain threshold to prevent MOSFET burnout.
Some Lessons Cannot Be Repeated
This incident may seem like a small oversight, but the consequences were a complete shutdown of the warehouse, with dozens of devices rendered useless, and the costs of repairs and downtime far exceeded the code itself.
We later summarized the situation, and it can be boiled down to a simple statement:
Dead time is the “breathing space” that allows current to flow safely; it is not a trivial detail.
Often, system failures do not occur because of a lack of knowledge, but because one assumes that “this minor issue is not significant.”
Now, every time I see someone adjusting PWM, I remind them, “Have you configured the dead time?” It’s not a matter of distrust; it’s knowing that issues can arise too quickly, leaving no time for regret.