If you ask me what the most persistent issue in embedded development is, I would say just two words: interrupts. Especially in STM32, if the interrupt priorities are not configured correctly, the system may appear normal on the surface, but underneath it is a complete mess.
The loss we experienced was a vivid example.
It was a visual inspection project running on the STM32F427, using DCMI to connect a CMOS image sensor with low resolution, QVGA, and a low frame rate of about ten frames per second. We used DMA with double buffering to capture images, and then performed grayscale analysis and edge detection in the main loop to determine product quality.
In the initial phase of deployment, everything went smoothly; the image stream was intact, and the detection accuracy was close to 99%. But on the day the customer officially started production, problems arose.
The customer reported that the system sometimes missed detecting products, especially when the production line was running fast, leading to good products being misclassified as NG and directly removed by the robotic arm. In just one morning, hundreds of good items were packed and thrown away. The customer was furious, stating that if this continued, over a million orders would be lost.
We were also panicking. The camera was able to capture images, DMA was triggering normally, and the image processing algorithm had not changed; how could we start missing detections?
Initially, we suspected that the image processing speed was insufficient or that DMA was not configured for double buffering. However, after thorough investigation, we found that: DMA was enabled, double buffering was fine, and data was indeed written to both buffers.
But sometimes, the image data contained remnants of the previous frame in the latter half, while the first half was from the new frame. This is a typical case of inter-frame cross-contamination.
We then realized that the issue was not with DMA itself, but with the timing of the DCMI frame interrupt and the DMA transfer interrupt.
Specifically, both the DCMI frame start (VSYNC) interrupt and the DMA transfer complete interrupt were managed by NVIC for interrupt priority. However, we made a fundamental mistake:
We set the DCMI frame interrupt to a lower priority, while setting the interrupts for peripherals like UART, SPI, and USB to higher priorities.
The result was that on a high-speed production line, just as the system finished receiving a frame of image data and was ready to process it, the DCMI frame start interrupt had not yet cleared its flag and was suppressed by the USB and SPI interrupts. DMA then began writing the next frame of data, causing the previous frame to be partially overwritten before it was fully processed.
This is a classic case of interrupt priority preemption failure: the critical interrupt could not be processed, DMA continued blindly, data was contaminated, and images were missed.
We quickly adjusted the NVIC configuration, raising the priority of the DCMI frame start interrupt so that it could preempt other non-critical interrupts, and ensured that the DMA interrupt execution time was kept as short as possible.
HAL_NVIC_SetPriority(DCMI_IRQn, 0, 0); // Highest priority
HAL_NVIC_SetPriority(DMA2_Stream1_IRQn, 1, 0);
At the same time, we optimized the DMA configuration and enabled the double buffering mechanism:
hdma_dcmi.Init.DoubleBufferMode = ENABLE;
hdma_dcmi.Init.Memory0BaseAddr = (uint32_t)frameBuffer1;
hdma_dcmi.Init.Memory1BaseAddr = (uint32_t)frameBuffer2;
This way, while processing the previous frame, the next frame can be captured, eliminating the overwriting issue, and the main loop only processes frame data marked as “complete”.
Finally, we added frame number identification and CRC verification to ensure a quick validation of frame integrity before processing.
After reprogramming the system, it ran continuously for 48 hours without any missed detections.
The customer’s production line was restored, and we managed to save face. However, that weekend, I hardly slept for an hour, constantly replaying the interrupt priority configuration in my mind: who can preempt whom, who cannot be interrupted, and which sections of code must disable interrupts…
We did not incur financial losses from this incident, but we almost lost the customer. The lesson was very clear:
- The NVIC priority design in STM32 is very intricate; do not mess with default values.
- The data link of the visual system from acquisition to processing must be fully controllable and preemptible.
- DMA double buffering is not an “advanced feature”; it is a necessary safety mechanism.
Now, when I guide new colleagues in writing interrupt programs, the first question is not “Have you written your interrupt function?” but:
“Do you know if this code can be interrupted? Will there be issues if it is interrupted?”
Interrupts are not to be trifled with; if you do, you must take full responsibility. If you do not preempt it, it will take away the life of your entire production line.