This article is an original article from the WeChat public account “Talk Imaging” and the Zhihu column “All in Camera”. Please indicate the source when reprinting. Talk Imaging Reader QQ Group 2: 833282006 Talk Imaging Technical Forum: ww.dahuachengxiang.com Teaching videos on this site include “Image Sensor Technology and Applications”, “Imaging System Image Quality Debugging”, “Fundamentals of Imaging Algorithms (Python version)”, “Optics for Imaging Systems”, “New Version of Image Quality Testing and Measurement and International Standards”, “New Version of CMOS Sensor Testing and Measurement and International Standards”, “New Version of Digital Imaging System 42 Lectures” courses are available for sale at the official Taobao education store of Talk Imaging:https://shop322456667.taobao.com/
Previous articles on the public account introduced the automatic exposure algorithm of the Raspberry Pi, and this issue introduces the automatic white balance algorithm of the Raspberry Pi. The Raspberry Pi 3A is all placed in the Linux userspace and is open source. The camera driver is placed in the kernel layer, and all ISP-related drivers are managed by Broadcom, packaged together with the GPU code into a library that is not open to users. The driver files for the camera sensor are open, but since the code calling the sensor driver is not open, users cannot add new image sensor drivers themselves. The AWB code file is awb.cpp, which is completely open source.
The design of the Raspberry Pi’s AWB algorithm is based on a Bayesian assumption to solve the automatic white balance problem. That is, given several light sources, for any image input, select the light source from the given sequence that is most likely to produce appropriate colors as the result of automatic white balance. The mathematical essence of the automatic white balance problem is a statistical inference problem, solving for the most likely light source and applying white balance gain to this light source’s white point. This guiding principle is consistent with the guiding principles of the main automatic white balance algorithms popular today, whether it is the earlier maximum likelihood estimation, Bayesian estimation, or the current neural network model estimation methods, all of which are statistical inference solutions based on data.
The Raspberry Pi uses Bayesian estimation, a method of probabilistic inference based on Bayes’ theorem, which states:
Posterior Probability = (Likelihood * Prior Probability) / Normalization Constant
In other words, the posterior probability is proportional to the product of the prior probability and similarity.
The formula is:
In the context of AWB, the so-called prior data is the calibration data of AWB, which is obtained through laboratory collection under a series of standard light sources to get the distribution characteristics under the r/g and b/g coordinates, as well as the color temperature CCT values. The Raspberry Pi’s documentation records the Y-axis b/g as b and the X-axis r/g as r.
At the same time, assign a certain probability to each light source (i.e., prior data), i.e., prior probability. A simple example of a prior probability is the light source in outdoor conditions; under the current illumination (for example, 10000lux), the probability of high CCT light sources can be assigned a higher value, while low CCT light sources can be assigned a lower value.
The posterior probability refers to the probability of the new image data; the Raspberry Pi’s ISP divides the input image into 16×12 zones, and each zone will have the ISP calculate the R, G, B mean values, and then calculate the R/G, B/G values of the zone.
For example, if the image resolution is 1280×768, the image will be divided into 16×12 zones, with each zone containing 80×64 pixels, and these 80×64 pixels will be divided into 5×4 cells, each cell containing 16×16 pixels.
Each cell will have the ISP count how many pixels are close to saturation and how many pixels are close to black level. If too many saturated pixels or too many dark pixels are present, the data of this cell will be deemed unreliable and discarded.
The code defines that at least 80% of the non-saturated pixels in a cell must be present for it to be adopted.
255 x 90% is the threshold for determining saturation.
With these premises, AWB applies Bayes’ theory as follows:
I is the light source, P(I) is the prior probability of the light source
P(D) is the prior probability of pixel values
P(D|I) is the posterior probability of pixel data D given light source I
P(I|D) is the posterior probability of light source I given data D
Thus, the solution for AWB is:
To find the maximum P(I|D), that is, based on the input image (statistical) data, to determine which light source has the highest probability P.
P(D), the prior probability of pixels is a constant, meaning all pixel (statistical) data has equal prior probability. Therefore, the formula can be simplified to finding max P(D|I) * P(I)
How to determine P(I)? This becomes a very flexible technique; for example, we previously mentioned assigning probabilities to light sources based on current light intensity, but other conditions can also be used.
How to calculate P(D|I)?
The Raspberry Pi’s awb uses a color difference assessment algorithm, which means if the light source is chosen correctly, applying this WB Gain to the image will result in a smaller color difference. As shown in the image below, the original image (without white balance) is biased green, and after applying the correct WB gain, the image will be more neutral.
Focusing on the specific implementation, WB Gain is added to the near-gray area of the image, and then the color differences Δr, Δb are calculated.
The closer the color difference is to 0, the higher the probability that the light source is correct, meaning P(D|I) is higher.
Assuming that Δr, Δb follows a two-dimensional Gaussian distribution, there exists a proportional relationship as follows:
In actual calculations, to speed up computation, instead of calculating pixel by pixel, each zone can be treated as a ‘pixel’ D, allowing the summation of the product of color differences and P(I) across all zones.
Exponential calculations are inconvenient, so converting to log to seek likelihood (L) is more convenient. After taking log, (assuming σr = σb), the calculations can be further simplified to
L(i) is the prior log likelihood of that light source.
The Raspberry Pi’s algorithm design also leaves some control parameters, such as switching between bayers and gray world algorithms, how many frames (frame_period) to perform AWB, and the convergence speed of AWB (Speed), etc. Interested readers can refer to the code for understanding.
