1
About HDR
HDR images refer to high dynamic range images. This is a typical HDR image when the main subject of the image is set against a very bright background. The non-HDR image on the right side of the figure below (also known as SDR: standard dynamic range) shows overexposure in the background and underexposure inside the tunnel, making the image unclear, while the HDR image captured on the left retains all the details of the entire image. Capturing HDR images and videos is particularly important for outdoor camera applications such as in-vehicle cameras and outdoor security cameras.
2
HDR Technology
Standard dynamic range image sensors have a limited dynamic range and cannot capture images comparable to the capabilities of the human eye. However, these ordinary non-HDR image sensors can create HDR images by applying different exposure times across different image frames and then using software to synthesize the HDR image. This is the method used by ordinary smartphones to take HDR photos. However, this method is limited to capturing static HDR photos and cannot be used for HDR video. The photography process allows for significant delays when capturing several image frames and synthesizing these multiple images to generate the final HDR image, making such long delays unsuitable for HDR video.
We previously conducted a detailed algorithm analysis of tone mapping algorithms for multiple exposures (Bridging the Dynamic Range Gap (Part 1): Algorithm Principles and Practices for HDR Image Detail Recovery), and this time we will learn about the technological developments of HDR sensors.
3
LOFIC
The photos we capture come from the pixels on the sensor, which convert light into electrical signals and combine them to output an image. When the brightness of the captured object is too strong, a large amount of charge accumulates in a single pixel, resulting in overexposure of the image. The structure of LOFIC (Lateral Overflow Integration Capacitor) resembles a dual pot; when the charge exceeds the maximum limit that the pixel can hold (maximum well capacity), the excess charge flows into the adjacent “pot” instead of overflowing. This allows bright images to be displayed clearly. However, the LOFIC structure is complex and difficult to adapt to the trend of increasingly thin and light electronic devices. Therefore, LOFIC is now rarely found in cameras and smartphones.
4
Large and Small Pixels
Fujitsu has introduced the “Super CCD SR” with a large/small pixel separation structure, cleverly utilizing the differences in pixel properties to achieve HDR. Each “cell” of the sensor captures images simultaneously with one large and one small pixel, obtaining both high exposure and low exposure images. After synthesis, this forms an HDR image covering different dynamic ranges. However, due to the tendency for defects in the image, this technology has not been widely adopted.
5
Multi-Frame Uneven Exposure HDR
By continuously capturing multiple photos with different exposures and overlaying these photos, the best parts from each photo are extracted for synthesis. This method can cause “motion artifacts” when capturing moving objects or when there is camera shake during shooting.
6
Row Interleaved HDR
To address the issue of motion artifacts, the sensor is divided into rows, with long exposure and short exposure set at intervals between two rows. When the shutter is pressed, both long exposure and short exposure start simultaneously. This technology allows for the simultaneous capture of long exposure and short exposure images, ultimately obtaining an HDR image. However, the principle of row interleaving results in a loss of half the resolution, and the images often appear unnatural. Due to various shortcomings, this type of sensor has not been mass-produced.
7
ZigZag HDR
Sony proposed another possibility for pixel arrangement—Zigzag HDR. This is an upgraded version of row interleaved HDR. The sensor has both long exposure and short exposure settings distributed in a “Z” shape. Compared to row interleaving technology, the loss of resolution is effectively reduced, and the images appear more natural, making it applicable for video shooting.
8
HDR+
Google introduced HDR+ technology. It captures the same exposure image at least 8 times in succession with 3 sensors, overlaying them to distinguish details and noise in the dark areas, thereby utilizing algorithms to achieve a wide dynamic range. The “burst mode” of the Google Camera can automatically start pre-capturing before the shutter is pressed to achieve quick HDR effects. This is no longer comparable to the original multi-frame HDR.
9
QHDR
QHDR (Quad HDR) differs from Zigzag HDR not only in row alignment but also in the arrangement of pixels within the sensor. In QHDR, each 2×2 four-pixel area becomes a unit that works together with different exposure settings. This type of sensor not only meets the demands for high-resolution photography but also allows for video capture through 2×2 pixel merging, resulting in better low-light performance.
10
DOL Single-Frame Line-by-Line HDR
In the single-frame line-by-line HDR function, the shutter outputs long and short exposure data by rolling line-by-line from top to bottom. During the output of long exposure line data, short exposure line images are also generated simultaneously. This allows for a tight connection between the two line images, solving the unnatural appearance of iHDR images and effectively reducing motion artifacts.
This HDR sensor controls two, three, or even four sets of exposure pointers and readout pointers, with different pointers used to control different exposure times. The example shows an HDR with two sets of exposure pointers and readout pointers, one set for long (T1) exposure time flow and the other for short (T2) exposure time flow.
The basic working principle is that after the first line completes the required T1 exposure cycle, the image sensor reads out the first line image while allowing the first line to start the T2 exposure window. It can be seen that both T1 and T2 flows maintain full image resolution, with the T1 exposure window cascading with the T2 exposure window in each line of the image. The readout portions of T1 and T2 flows have a considerable degree of overlap. During the overlapping readout period, the sensor outputs T1 and T2 flows through the T1/T2 line interleaving process. This is why DOL-HDR is also referred to as interleaved HDR. Since the T1 exposure cycle and T2 exposure cycle in each line of the image are cascaded, as long as T1 + T2 < 33 milliseconds, the DOL-HDR sensor output can maintain 30 frames per second.