Banner
Abstract
The high dynamic range image processing algorithm is a key technology to enhance image quality, characterized by enhancing image details, improving light and shadow effects, and adapting to complex scenes. It has wide applications in photography and video production. Among them, promoting the development of HDR image processing algorithms for portrait scenes has become an important issue for achieving a balance between the realism and aesthetic quality of portrait images.
Recently, a team led by Professor Lin Zhixian from Fuzhou University and the Min Du Innovation Laboratory published a research article titled “HDR Image Processing Algorithm Based on Multi-Feature Fusion” in the Journal of Liquid Crystal and Display (ESCI, Scopus indexed, Chinese core journal), 2024, Issue 8, and it was selected as the cover article for that issue. This article proposes a U²HDRnet algorithm based on multi-feature fusion for HDR processing of images containing portraits.
Figure 1: Cover Image of Journal of Liquid Crystal and Display, 2024, Issue 8
U²HDRnet Structure
The U²HDRnet model mainly consists of three parts: skin feature extraction module, tri-branch feature extraction module, and color reconstruction module. The color and location information extracted by the skin feature extraction module is an important basis for subsequent calculations in the network, and its processing results will be input into the other two modules to ensure that the skin part does not undergo color shifting. The tri-branch feature extraction module is responsible for extracting various features. The color reconstruction module performs the minimal but critical workload: it learns a grayscale image and uses new slice nodes to upsample the affine coefficients grid back to full resolution. Then, local affine transformations of these pixels are applied to the full-resolution input while fully considering the weight of the skin part, producing the final output.
Figure 2: U²HDRnet Network Structure Diagram
Image source: Journal of Liquid Crystal and Display, 2024, 39(8):1024-1036. Fig.1
Tri-Branch Feature Extraction Module
To fully extract various feature information from images, this paper proposes a tri-branch feature extraction network, consisting of three branches: the local feature extraction branch operates at low resolution, while the semantic information extraction branch and global information extraction branch operate at high resolution, enhancing feature extraction effectiveness through the addition of attention modules during multi-feature fusion. Most of the calculations are performed in the tri-branch feature extraction module. In HDR processing tasks, not only local image features are relied upon but also global image features and semantic features, such as histograms, average intensity, and even scene categories. Therefore, the feature extraction part of the tri-branch network is further divided into local feature extraction, global feature extraction, and semantic feature extraction. The local feature extraction operates at a low resolution flow with an input resolution of 256×256, where the input first undergoes five convolution calculations to reduce the spatial resolution of the image. After the calculations, it undergoes two more convolution calculations to obtain its final local features. The high-resolution flow has an input resolution of 512×512, where the global feature extraction branch outputs the final global features after six convolution calculations followed by three fully connected calculations. The semantic feature extraction branch performs four down-sampling operations in the HSeg module and then one up-sampling to extract deep semantic information, which is processed through two convolution operations before outputting the final semantic information. After the calculations of each resolution flow are completed, the three parts of features and skin features are fused to produce the final affine coefficients.
Figure 3: HSeg Network Structure Diagram
Image source: Journal of Liquid Crystal and Display, 2024, 39(8):1024-1036. Fig.2
GAcmix Module
To enable ACmix to adapt to current HDR tasks and improve its perception of global information, this paper replaces the self-attention module with a global self-attention (GSA) module. Compared to self-attention, GSA pays more attention to the global relationships between all positions in the sequence, allowing each position to interact with the entire sequence, thereby better capturing global dependencies. This makes it more suitable for HDR tasks. Figures 4 and 5 show the structures of the GSA module and GAcmix, respectively.
Figure 4: GSA Module Structure Diagram
Image source: Journal of Liquid Crystal and Display, 2024, 39(8):1024-1036. Fig.3
Figure 5: GAcmix Module Structure Diagram
Image source: Journal of Liquid Crystal and Display, 2024, 39(8):1024-1036. Fig.4
Color Reconstruction Module
The color reconstruction module works at full resolution and performs the least amount of computation, but it plays a key role in capturing high-frequency effects and preserving edges. This paper adopts the slice nodes used by Gharbi et al., which perform data lookups in the affine coefficient grid based on learned guidance mapping. By obtaining high-resolution affine coefficients with a full-resolution slice grid, local color transformations are applied to each pixel to produce the final output.
Experimental Results Analysis
Through experimental analysis, this paper finds that U²HDRnet outperforms common algorithms in various scenes when the skin extraction module is not added. In vehicle scenes, its performance lags behind the FHDR algorithm. By examining the dataset, it was found that the vehicle scenes in the FiveK dataset have relatively uniform colors, mostly pure black or pure white vehicles. U²HDRnet excels at extracting deep features; however, in scenes with overly uniform colors, its performance lags behind FHDR, which improves saturation through continuous iterations, with a PSNR lag of 0.18, but the lag is only 0.5%. Overall, U²HDRnet shows the least fluctuation in PSNR values compared to other single-frame HDR algorithms across different scenes, and all PSNR values are above 30, indicating that the HDR image processing network constructed in this paper has strong stability.
To verify U²HDRnet’s HDR processing capability in portrait scenes and to compare the strengths of different algorithms in processing portrait scenes, this paper conducted comparative experiments on the PortraitHDR dataset. The experimental results indicate that in portrait scenes, it achieves the highest PSNR score, while its performance on SSIM exceeds that of other networks. SSIM reflects the similarity between the processed images and the ideal HDR images. Since this paper focuses on obtaining high-quality HDR images in portrait scenes while suppressing skin color shifts, the SSIM metric better reflects whether the suppression of skin color shifts has been achieved. Comparing the training and inference times of different networks, it is found that CNN-HDR has the shortest training and inference times, but its processing effect is also the worst; U²HDRnet’s training and inference times increased by 0.4 hours compared to the HDRnet algorithm using bilateral feature networks, but its PSNR improved by 10.7%, and SSIM improved by 8.9%. This is due to the U²net module in this algorithm adopting a transfer learning strategy, reducing the training time. Therefore, although U²HDRnet integrates more feature information, its training time only increased by 2%. Additionally, since this paper adopted an attention mechanism to filter out some irrelevant parameters, its inference speed ranks second among the compared algorithms. Considering all objective indicators, the strategy of using multi-feature fusion for HDR image processing is quite successful, and the comprehensive performance of the U²HDRnet algorithm is optimal among all algorithms.
Figure 6: Comparison of Different Algorithms on the PortraitHDR Dataset (a) Input Image; (b) Ideal Output Image; (c) FHSVNet Algorithm Output Image; (d) HDRnet Algorithm Output Image; (e) U²HDRnet Algorithm Output Image.
Image source: Journal of Liquid Crystal and Display, 2024, 39(8):1024-1036. Fig.10
Conclusion
This paper proposes a U²HDRnet algorithm based on multi-feature fusion for HDR image processing, aiming to address the issue of skin color shifts commonly encountered in existing HDR algorithms when processing portrait images. This algorithm utilizes U²net as the skin feature extraction module, after which the extracted skin features are input into the tri-branch feature extraction module and the color reconstruction module. Additionally, the proposed tri-branch feature extraction module performs excellently, where the local feature extraction branch first downsamples the image to reduce its resolution, extracting local information through multiple convolutions; the global feature extraction branch extracts global features after multiple convolutions; and the semantic information extraction branch extracts deep semantic information in the HSeg module. Finally, the three parts of output features are fused, and after calculations in the improved Acmix module, the skin features are fused to produce the final affine transformation coefficients. In the color reconstruction module, the slice nodes utilize learned guidance mapping. By obtaining high-resolution affine coefficients with a full-resolution slice grid, local color transformations are applied to each pixel to produce the final output, and the weight of the skin features is fused again in the final HDR reconstruction to retain skin information as much as possible. Experimental data indicate that the PSNR of the proposed algorithm reaches 31.42 dB and the SSIM reaches 0.985. Combining subjective evaluation results, it can be found that the U²HDRnet algorithm outperforms currently common HDR algorithms in HDR processing capability in portrait scenes.
Paper Information
Wu Chunlin, Zhang Yongai, Lin Zhixian, et al. HDR Image Processing Algorithm Based on Multi-Feature Fusion [J]. Journal of Liquid Crystal and Display, 2024, 39(8):1024-1036.
https://cjlcd.lightpublishing.cn/thesisDetails#10.37188/CJLCD.2023-0255
Corresponding Author Information
Lin Jianpu, PhD, Lecturer, obtained his PhD from Fuzhou University in 2019, mainly engaged in research on new display technologies, image processing technologies, electronic paper driving, and integration. E-mail: [email protected]
Supervised by: Zhao Yang
Edited by: Zhao Wei
Banner
👇 Follow Me 👇
Tap “Like” and “View”, so you won’t miss updates