A Clever Clarity Detection Solution on ESP32-S3

Introduction

Recently, while working on the development of an assistive device, the core requirement was to: real-time select the clearest image from a series of continuous shots to upload to the cloud for YOLO recognition. However, due to well-known reasons, traditional OV series cameras require a longer exposure time in slightly dim environments, leading to frequent complaints from customers about overly blurry images. Therefore, based on the ESP32-S3, I aimed to implement a simple algorithm for clarity selection.

Initially, I followed classic computer vision theories, using the Laplacian operator for edge detection in the YUV domain. Theoretically, this is perfect, but in practice, it was not usable. This article will document the evolution from “algorithm flow” to “engineering flow,” delving into the mathematical essence of the Laplacian operator, the performance bottlenecks of YUV to JPEG conversion, and how to achieve another clever optimization using information theory principles.

Part One: Review of the Laplacian Operator

Having forgotten the basics after studying so long ago, I revisited the fundamental principles.

1.1 How Computers See Images

In computer vision, unlike the human eye, which perceives images as a two-dimensional plane, computers view them as a three-dimensional terrain surface.

• The coordinates.

• The brightness value is the altitude at that coordinate.

The so-called “blurriness” and “clarity” understood by computers correspond to the terrain’s flatness and steepness.

1.2 How to Find Edges?

Edges are found by first calculating the slope using the first derivative, then the curvature using the second derivative.

• First Derivative (Gradient): Tells us the slope.

• : Indicates that there is a slope here.

• Second Derivative (Curvature): Tells us the degree of curvature (or the rate of change of the slope).

• : Indicates a sudden change occurred here.

For the human eye, clear image edges mean a sharp change in brightness. In fact, this is also true at the physical level, where such changes manifest as significant numerical oscillations (crossing zero) in the second derivative. Blurry images have edges that are “smoothed out” into gentle slopes, with the second derivative approaching 0.

1.3 Discretization Derivation

In the direction, the second-order difference formula is:

(i.e., left pixel + right pixel – 2 times the middle pixel)

Similarly, in the direction:

The Laplacian operator is the sum of the two directions:

UpDownLeftRightFourPointsSum

Extracting the coefficients gives us the classic convolution kernel:

1.4 Why is Clarity Scoring a Function of Light Intensity?

I was puzzled by this for a long time; I wasn’t attentive in class back then.

Intuitively, “clarity” should be a property describing the geometric boundaries of an object, independent of ambient light. However, in algorithms based on the Laplacian operator (and most gradient-based operators), clarity scoring is essentially the statistical magnitude of edge gradients.

In fact, it can be rigorously proven mathematically by establishing a discrete edge model: clarity scoring is positively linearly correlated with signal amplitude (i.e., light intensity), and even shows a squared correlation in variance calculations.

1.4.1 Ideal Step Edge Model

Let the image be a one-dimensional discrete signal . Define an ideal, absolutely clear edge, where the signal experiences an instantaneous jump at without transitional pixels.

BackgroundBrightnessObjectBrightnessBrightnessObjectBrightness

Where the local contrast of the image is defined as .

1.4.2 Laplacian Response Calculation

Using the standard one-dimensional discrete Laplacian operator to perform convolution on the signal:

We will examine the results under two lighting conditions.

Scene A: High Light Environment

Assuming background brightness , object brightness (high contrast, ).

The signal sequence is:

Calculating the Laplacian response at the edge:

• At (left side of the edge):

• At (right side of the edge):

• The response in the remaining flat areas is .

Absolute Clarity Score:

Scene B: Low Light Environment

Assuming the ambient light dims by a factor of 10, the sensor responds linearly, and the overall brightness drops to 1/10 of the original.

Background brightness , object brightness (low contrast, ).

Note: Geometric clarity remains unchanged; the edge still experiences an instantaneous jump without a blurred transition.

The signal sequence is:

Calculating the Laplacian response at the edge:

• At :

• At :

Absolute Clarity Score:

1.4.3 Linearity and Variance

By comparing and , we conclude:

Conclusion 1: Linearity of the Operator

Since the Laplacian operator is a linear operator, if the input signal amplitude is scaled by a factor of (), the output response is also strictly scaled by the same factor.

In this case, if the light intensity decreases by a factor of 10 (), the base clarity score also directly decreases by a factor of 10. This causes originally clear edges to appear numerically comparable to noise (Noise) amplitude under high light.

Conclusion 2: Squared Level Decay of Variance

In practical CV algorithms (e.g., cv2.Laplacian(img).var()), we typically use the variance of the Laplacian response to represent clarity.

According to the properties of variance:

If the light intensity decays to , the clarity score (variance) will decay to .

This explains why during dusk or nighttime, even if the camera is perfectly focused, the unnormalized Laplacian score will drop exponentially, causing the algorithm to fail.

Part Two: From Implementation to Abandonment

The initial design logic was typical: to obtain the most accurate clarity quantification index, pixel-level operations must be performed on uncompressed Raw images. Based on this, I devised the following data flow processing scheme:

1. Configure the CMOS sensor to output raw bitstream in YUV422 format, where every two pixels share a set of chromatic information, but the brightness information maintains full-resolution sampling, providing a grayscale data source for the Laplacian operator.
2. Directly perform improved Laplacian convolution operations on the Y channel in memory, locking in the frame with the highest score through a three-frame buffer comparison.
3. Call the software encoding library to re-encode this frame of YUV data into JPEG format.
4. Wrap the encoded binary stream in Base64 and upload it via a 4G module using the serial port.

To achieve real-time detection under limited clock frequency, I performed intense optimizations on the operator itself, compressing the single-frame computation time to an extreme level of 6ms. However, subsequent system logs did not directly allow me to revert to the old code version.

2.1 Log Data Sampling Analysis

Below is a snippet of the UART output log from the device under real load:

ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line// Continuous shooting and computation phase (sufficient computing power)I (14279) camera_driver: Frame[0]: Score=18676, Time=6214 usI (14489) camera_driver: Frame[1]: Score=20973, Time=6491 usI (14899) camera_driver: Frame[2]: Score=19897, Time=6492 us// Selected the best frame, entering encoding phase (pipeline blocked)I (14909) camera_driver: Selected Best Frame (Score: 20973). Converting to JPEG...I (15679) camera_driver: JPEG Converted. Size: 14232 bytes.

Interpretation of Data Dimensions:

• Detection phase time (~6.5ms): This is thanks to the successful spatial downsampling strategy.
• Encoding phase time (770ms): This means that in a simple “photo-upload” transaction, the CPU is blocked for over 99% of the time on format conversion rather than business logic.

2.2 ROI and Step Sampling

This is an intentional design based on the statistical characteristics of the image.

When processing images at SVGA (800×600) resolution, full pixel traversal involves 480,000 convolution operations and corresponding memory addressing. However, in the specific task of clarity evaluation, full pixel traversal is a significant waste of computing power.

First, I introduced an ROI mechanism, calculating only the central 50% of the image. This decision is based on the photographic optical principle of “center composition bias” and the MTF curve characteristics of lenses—most imaging systems have weaker edge resolution than the center, and the target object is likely located in the center of the field of view. Ignoring edge pixels not only reduces a large amount of data processing but also avoids noise interference caused by lens distortion and edge vignetting.

Secondly, a step sampling strategy was adopted, with a step size set to 4. From a signal processing perspective, natural images exhibit high spatial correlation. The brightness variation between adjacent pixels is usually smooth and continuous, meaning pixels and have a high degree of information redundancy in their second derivative features. By sampling at intervals below the Nyquist frequency but sufficient to maintain statistical significance, we reduced memory bandwidth usage by an order of magnitude without significantly sacrificing the accuracy of variance statistics. This strategy of trading spatial resolution for temporal resolution is a standard paradigm in embedded vision.

2.3 YUV to JPEG

Since the operator has been optimized to the extreme, why is the system still lagging? The core contradiction lies in the mismatch between the hardware architecture of the ESP32-S3 and the complexity of the JPEG encoding algorithm.

The JPEG encoding standard is not a simple memory copy; it is a pipeline involving multiple complex mathematical transformations. When we call <span>frame2jpg</span> on the ESP32, we are actually simulating the following process on a general-purpose CPU:

1. Color space conversion and blocking: Cutting the YUV planar data into pixel blocks (MCU, Minimum Coded Unit).
2. **Discrete Cosine Transform: This is the most time-consuming step. Each block requires a two-dimensional DCT transformation, converting spatial domain pixel intensities into frequency domain coefficients. This involves a large number of floating-point multiply-add operations (or fixed-point approximations).
3. Quantization: Using human visual psychology models, dividing high-frequency coefficients by the quantization table and rounding. This involves dense division operations.
4. Entropy encoding: Performing Zig-Zag scanning on the quantized coefficients and executing Huffman coding or arithmetic coding. This involves a large number of bit operations and table lookups.

Unlike application processors such as Allwinner T113 or Rockchip RK series, the ESP32-S3 is an MCU. It lacks hardware encoding and does not have dedicated cores to handle raw memory data.

Although the core has some DSP instruction extensions, relying on the CPU to serially execute the above complex mathematical transformations on 480,000 pixels per block is still too much pressure. Additionally, the raw YUV data at SVGA resolution is close to 1MB, far exceeding the internal SRAM capacity of the ESP32, forcing the system to frequently access external PSRAM. The bandwidth limitations of the SPI bus (even in Octal mode) lead to severe memory I/O bottlenecks—the CPU not only computes slowly but also consumes a lot of clock cycles while waiting for data transfers.

Conclusion: From implementation to abandonment.

Part Three: A Clever Approach and Why It Works

Since simulating a JPEG encoder on a general-purpose MCU architecture is the root cause of system latency, the optimal engineering solution is to directly utilize the built-in DSP hardware pipeline of the OV5640 sensor to output JPEG bitstreams. However, this architectural change brings a new mathematical challenge: we obtain a binary compressed stream after entropy encoding, rather than a spatial domain pixel matrix, meaning classic gradient-based convolution operators cannot be directly applied. To evaluate image quality in the compressed domain, we need to introduce a new metric dimension: information entropy.

3.1 Clarity and Compression Bitrate Correlation

In the context of digital image processing, “clarity” is mathematically equivalent to the proportion of high-frequency components in the signal. According to Shannon’s information theory, the information entropy of a system is defined as a measure of the uncertainty of that system:

where is the probability of occurrence of the symbol. In image compression, the data volume is a direct mapping of the image’s information entropy. By analyzing the internal mechanisms of the JPEG compression standard, we can prove that image sharpness is strictly positively correlated with the length of the compressed bitstream.

First, analyze from the perspective of frequency domain energy distribution.

The core of JPEG compression is the block-based Discrete Cosine Transform (DCT). DCT converts spatial domain pixel intensities into frequency domain coefficients.

According to signal processing principles, flat areas in an image (i.e., blurry or low-frequency areas) are primarily composed of DC components (DC) and low-frequency AC components (AC), with their energy concentrated in the upper left corner of the DCT matrix; sharp edges (i.e., clear areas) mathematically manifest as step signals. According to the Gibbs phenomenon and the convergence characteristics of Fourier series, an ideal step signal has infinite bandwidth in the frequency domain, meaning that clear images still have significant non-zero coefficient magnitudes in the lower right corner of the DCT matrix.

Secondly, analyze from the statistical characteristics of quantization and coding.

The subsequent step of the JPEG algorithm is quantization, which involves dividing DCT coefficients by the quantization table and rounding. Since the human eye is not sensitive to high-frequency details, the quantization table usually applies larger divisors to high-frequency coefficients.

• For Blurry Images: Due to physical low-pass filtering of object edges, the original high-frequency DCT coefficients are very low in magnitude. After quantization division, these high-frequency coefficients are almost entirely truncated to zero. In the subsequent Zig-Zag scanning, this will produce long sequences of consecutive zeros.

• For Clear Images: Because they retain sharp edges, high-frequency DCT coefficients have higher magnitudes, surviving the quantization process and maintaining non-zero values. This frequently interrupts the continuity of zero sequences.

Ultimately, the decisive factor causing file size differences is entropy encoding.

JPEG uses run-length encoding combined with Huffman coding to compress the quantized coefficients.

• Blurry images produce long strings of zero sequences (e.g., “50 consecutive 0s”) that can be represented very efficiently by RLE, occupying very few symbol bits.

• Clear images, due to the presence of high-frequency non-zero coefficients, disrupt the continuity of zero runs, forcing the encoder to record these non-zero coefficients and their magnitudes separately. According to Huffman coding principles, the more unique symbols that need to be recorded, and the more uniform the symbol probability distribution, the longer the required average code length.

In summary, the theorem holds:

Under the premise of a fixed quantization table and fixed resolution, the size of an image file (Size) is a monotonically increasing function of its high-frequency information content. That is:

3.2 Empirical Validation and Data Significance

To validate the effectiveness of this theoretical model in embedded engineering, we constructed controlled experiments on the ESP32-S3 platform. We configured the OV2640 sensor for hardware compression mode, locking the output resolution to SVGA (800×600), with a JPEG quality factor of . While keeping the field of view content essentially unchanged, we artificially introduced physical vibrations to simulate “blurry” samples and captured “clear” samples during still shots.

The experimental data exhibited high statistical significance:

In a sequence of continuously sampled images, frames with motion blur due to physical movement lost a significant amount of high-frequency information, resulting in an extremely sparse quantized DCT matrix, causing the final generated JPEG bitstream size to converge around 10KB.

Conversely, clear frames captured during motion gaps retained the step characteristics of edges, forcing the Huffman encoder to output more bits to record these details, with file sizes stabilizing above 14KB.

A data volume difference of up to 40% () provides a robust threshold space. This proves that under constrained computational resources, we can effectively use compressed bitrates as a proxy variable for spatial clarity, thus completing image quality discrimination with time complexity.

Part Four: Function Implementation

4.1 Scheme A: Initial Laplacian Implementation with ROI and Step Sampling Version

ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line/** * @brief Fast Laplacian Clarity Detection (for YUV422 format) * @note Utilizes Stride (step) and ROI (Region of Interest) for rapid computation */long calculate_sharpness_yuv_test(const uint8_t *img_data, int w, int h) {    long score = 0;    int stride = 4; // Downsampling factor to improve speed    // Only calculate the central 50% area    int start_y = h / 4;    int end_y = (h * 3) / 4;    int bytes_per_row = w * 2; // YUV422 each pixel 2 bytes    for (int y = start_y; y < end_y; y += stride) {        int row_offset = y * bytes_per_row;        int row_up = (y - 1) * bytes_per_row;        int row_down = (y + 1) * bytes_per_row;        // X direction also needs to multiply by 2 and must align to even (Y component)        for (int x = (w/4)*2; x < (w*3/4)*2; x += stride*2) {            // Get Y components from up, down, left, right (direct memory operation, no floating-point operations)            uint8_t center = img_data[row_offset + x];            uint8_t up     = img_data[row_up + x];            uint8_t down   = img_data[row_down + x];            uint8_t left   = img_data[row_offset + x - 2];            uint8_t right  = img_data[row_offset + x + 2];            // Laplacian convolution: 4*Center - (Up+Down+Left+Right)            int val = (4 * center) - (up + down + left + right);            score += (val > 0) ? val : -val; // abs()        }    }    return score;}

4.2 Clever Version

This is the final solution implemented on the ESP32. Zero computational cost, zero conversion delay.

ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line/** * @brief Complete test process: Simulate continuous shooting and selection */void test_clever_selection_logic() {    printf("--- [TEST] Starting JPEG Heuristic Selection ---\n");    // Simulate JPEG data of 3 continuous shots (only simulating size and pointers)    // Assume scenario: moving quickly, the first two shots are blurry, the third is clear    size_t sizes[] = {9800, 10200, 14500};     const char* descs[] = {"Blurry(Motion)", "Blurry(Focus)", "Sharp"};    int best_index = -1;    size_t max_size = 0;    // Simulate continuous shooting loop    for (int i = 0; i < 3; i++) {        // 1. Hardware directly outputs JPEG (time cost 0ms)        size_t current_len = sizes[i];         printf("Frame[%d] (%s): Size = %zu Bytes\n", i, descs[i], current_len);        // 2. Core algorithm: Compare sizes        if (current_len > max_size) {            max_size = current_len;            best_index = i;        }        // 3. Free memory of non-best frames (esp_camera_fb_return)    }    printf(">>> RESULT: Selected Frame[%d] (%zu Bytes)\n", best_index, max_size);    if (best_index == 2) {        printf(">>> TEST PASSED: Successfully selected the sharpest image based on entropy.\n");    } else {        printf(">>> TEST FAILED.\n");    }}

4.3 Theoretically Ideal Version

This is the most standard approach. To eliminate the influence of light, we must first perform histogram equalization, forcibly stretching the image’s contrast, and then perform edge detection. Of course, this is also the most resource-intensive episode for the ESP.

ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line/** * @note Step 1: Statistics histogram to build LUT (remove light influence) * @note Step 2: Apply LUT mapping and calculate Laplacian * @return Theoretically the most robust score, but the slowest computation */long calculate_sharpness_scheme_b_academic(const uint8_t *img_data, int w, int h) {    long score = 0;    int stride = 4; // Still retain step optimization, otherwise slower    int start_y = h / 4;    int end_y = (h * 3) / 4;    int start_x = (w / 4) * 2;    int end_x = (w * 3 / 4) * 2;    int bytes_per_row = w * 2;    // --- Stage 1: Statistics histogram (Histogram) ---    uint32_t hist[256] = {0};    int pixel_count = 0;    for (int y = start_y; y < end_y; y += stride) {        int row_offset = y * bytes_per_row;        for (int x = start_x; x < end_x; x += stride * 2) {            uint8_t val = img_data[row_offset + x];            hist[val]++;            pixel_count++;        }    }    // --- Stage 2: Build cumulative distribution function (CDF) mapping table ---    uint8_t lut[256];    uint32_t sum = 0;    float scale = 255.0f / pixel_count;    for (int i = 0; i < 256; i++) {        sum += hist[i];        lut[i] = (uint8_t)(sum * scale);    }    // --- Stage 3: Mapping + Convolution ---    for (int y = start_y; y < end_y; y += stride) {        int row_offset = y * bytes_per_row;        int row_up = (y - 1) * bytes_per_row;        int row_down = (y + 1) * bytes_per_row;        for (int x = start_x; x < end_x; x += stride * 2) {            // Note: Each pixel read here must go through LUT lookup            uint8_t center = lut[img_data[row_offset + x]];            uint8_t up     = lut[img_data[row_up + x]];            uint8_t down   = lut[img_data[row_down + x]];            uint8_t left   = lut[img_data[row_offset + x - 2]];            uint8_t right  = lut[img_data[row_offset + x + 2]];            int val = (4 * center) - (up + down + left + right);            score += (val > 0) ? val : -val;        }    }    return score;}

4.4 Benchmark Testing Function

ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line#include <stdio.h>#include <stdlib.h>#include <time.h>// Simulate getting the current time in microsecondslong get_time_us() {    return (long)clock(); // Should use esp_timer_get_time on ESP32}void run_benchmark() {    printf("========== ALGORITHM BENCHMARK ==========\n");    // 1. Prepare fake YUV data (SVGA 800x600)    int width = 800;    int height = 600;    // Allocate YUV422 memory    uint8_t *fake_yuv = (uint8_t*)malloc(width * height * 2);     // Fill with random noise to simulate image    for(int i=0; i<width*height*2; i++) fake_yuv[i] = rand() % 50;     // Draw an "edge" in the center    for(int i=300*width*2; i<305*width*2; i++) fake_yuv[i] = 200;    // --- Test Scheme A ---    long t0 = get_time_us();    long score_a = calculate_sharpness_scheme_a(fake_yuv, width, height);    long t1 = get_time_us();    printf("[Scheme A] Fast Laplacian\n");    printf("   Score: %ld\n", score_a);    printf("   Time : %ld ticks (Approx 6ms on ESP32)\n", t1 - t0);    // --- Test Scheme B ---    t0 = get_time_us();    long score_b = calculate_sharpness_scheme_b_academic(fake_yuv, width, height);    t1 = get_time_us();    printf("[Scheme B] Histogram Eq + Laplacian\n");    printf("   Score: %ld (Normalized & Robust)\n", score_b);    printf("   Time : %ld ticks (Approx 25ms+ on ESP32)\n", t1 - t0);    printf("   NOTE : High CPU load due to LUT lookup & 2-pass scan.\n");    // --- Test Scheme C ---    // Simulate sizes of 3 JPEG continuous shots    size_t jpeg_sizes[] = {10500, 9800, 14200};     t0 = get_time_us();    int best_idx = select_best_jpeg_heuristic(jpeg_sizes, 3);    t1 = get_time_us();    printf("[Scheme C] JPEG Entropy Heuristic\n");    printf("   Selected Frame Index: %d (Size: %zu)\n", best_idx, jpeg_sizes[best_idx]);    printf("   Time : %ld ticks (Instant)\n", t1 - t0);    printf("   NOTE : Zero computational cost. Requires Hardware JPEG Encoder.\n");    free(fake_yuv);    printf("=========================================
");}

Conclusion

In limited performance scenarios, it is necessary to make trade-offs based on actual business needs. There is no need to strive for perfection; it is better to invest in better chips.