Innovative ISP+NPU Fusion Architecture of Kirin 9000

The Kirin 9000, as the world’s first 5nm 5G SoC, represents a leap in technological innovation in imaging, helping the Huawei Mate 40 series smartphones achieve the No. 1 ranking on the DXO leaderboard.This upgrade in the imaging system mainly includes two aspects. First, the Kirin 9000 integrates Huawei’s most advanced ISP 6.0, achieving a 50% increase in throughput and a 48% improvement in noise reduction, enhancing image processing capabilities.More importantly, the Kirin 9000 realizes for the first time in the industry an ISP+NPU fusion architecture, bringing powerful detail restoration and noise reduction capabilities into the video domain. Videos shot in low-light environments are clearer, and details are vividly displayed, once again breaking the boundaries of smartphone imaging capabilities.

How does the ISP+NPU fusion architecture empower the smartphone imaging system? Let’s analyze the underlying technical principles:

Partnering with RYYB CFA

Enabling Stronger Night Photography and Color Restoration

In the digital imaging era, image sensors have become the light-sensitive materials replacing film, allowing thin devices like smartphones to achieve extraordinary image quality.Image sensors are often mentioned alongside color, but in fact, image sensors are “color blind”; the photos taken are black and white and require the assistance of a color filter array (CFA) to obtain color information.

Innovative ISP+NPU Fusion Architecture of Kirin 9000

Bayer Array RGB Color Filter

The image above shows a typical RGB CFA, invented by Kodak’s Bayer in 1976. The gray blocks below are the light-sensitive PD sections, where photons are converted into electrical signals; the red/green/blue (R/G/B) color blocks above are the color filters that filter out red, green, and blue light from white light, which are then absorbed by the gray PD sections.

After understanding the Sencor and RGB CFA, let’s learn about three important modules in smartphone photography:

① Three Important Modules Determine Color and Detail Restoration of Photos

First, unlike film cameras that form images physically, digital cameras require multiple algorithms to support the direct output of images, deeply processing the information obtained from Sencor and filters. The three most important modules in this process are the Demosaic interpolation algorithm, automatic white balance, and color correction.

Demosaic Interpolation Algorithm: Under the RGB Bayer Pattern, each pixel can only capture information from one color channel, while the other two color information is missing. This information needs to be calculated using interpolation algorithms, utilizing adjacent pixels’ color information to compute the missing two color information, so that each pixel has complete color information. This “de-mosaicing” process is called the Demosaic interpolation algorithm and is one of the most important modules in the entire ISP pathway.

Automatic White Balance (AWB): Due to color temperature effects, white appears differently under different color temperatures, appearing yellowish at low temperatures and bluish at high temperatures. The function of white balance is to ensure that the RGB ratio of white objects remains 1:1:1 under any color temperature, presenting a white appearance. Common AWB algorithms include gray world and perfect reflection methods.

Color Correction (CCM): The light sensitivity curve of the camera differs from that of humans; after AWB calibrates the white, the accuracy of other colors still needs to be calibrated using a CCM matrix.

② The ISP+NPU Fusion Architecture Perfectly Supports the New RYYB CFA for More Accurate Color Restoration

To maintain a leading position in the smartphone imaging industry, continuous innovation and deep research and development are essential. In 2019, to enhance image quality while maintaining limited pixel sizes on smartphones, the Huawei P30 series first adopted a new type of ultra-sensitive image sensor with RYYB CFA. This sensor’s Y (yellow) spectral response is broader, allowing it to sense more photons, increasing the total light intake by approximately 30%-40%. This significantly enhances the luminance signal-to-noise ratio in low-light scenes, resulting in superior night photography.

However, this more powerful sensor is not easy to handle. For most digital cameras using RGB filters, the traditional ISP’s Demosaic interpolation algorithm, automatic white balance, and color correction can achieve relatively accurate color restoration. But facing the RYYB CFA, the various modules in the traditional ISP pipeline cannot cope with the rich color information contained in the Y pixels, nor can they accurately interpolate and restore object details from the RYYB sensor.

In simple terms, the RYYB CFA can be understood as having one additional color channel on the R channel compared to the traditional RGB CFA, where the Y channel can simultaneously absorb green and red light. Two colors enter the same channel, and G needs to be derived from Y-R, making interpolation calculations more complex. Therefore, the traditional ISP cannot handle it and must customize new technology for the RYYB CFA.

To better leverage the performance of the ultra-sensitive sensor, Huawei adopted an AI neural network Demosaic algorithm, a new AI color AWB, and a color algorithm CCM management module, integrating them into the Kirin ISP Pipeline image processing pathway. The neural model networks of each module, trained on massive RYYB sensor RAW data, can effectively find the complex mapping relationships between object details and color components, achieving breakthrough support for the new RYYB CFA using computational photography methods within the traditional ISP processing architecture. This time, with the robust AI performance support of the Kirin chip, the innovative upgraded fusion architecture of ISP and NPU enables real-time processing of videos with the new RYYB CFA, significantly enhancing the detail and color restoration effects of 4K video in low-light conditions.

Data and Information Direct Access

Achieving Pixel-Level Processing of Real-Time Video

The Kirin 9000 pioneered the ISP+NPU fusion architecture, effectively combining the ISP processing pipeline and NPU matrix computation through a precisely designed fusion architecture, achieving pixel-level processing of real-time video. The greatest advantage of the ISP+NPU fusion architecture lies in the direct access to data and information. By directly connecting the NPU computations into the ISP’s Pipeline through hardware, utilizing the large capacity and bandwidth capabilities of SmartCache 2.0, it creates a ping-pong processing of data streams, where both input and output data streams are continuous without any interruption, completing seamless buffering and processing of data.

Through this hardware’s intra-frame and inter-frame information direct access mechanism, the Kirin 9000 can efficiently complete the reconfiguration of control information between ISP and NPU in multiple scenes. Even within a single frame, multiple algorithms can seamlessly switch, fully utilizing the powerful computing power of the upgraded Huawei Davinci 2.0 architecture NPU.

In simple terms, the videos we see are composed of continuously changing static frames. When a phone processes a 24 frames per second video, it means it needs to process 24 static images every second, making video processing far more complex than processing static photos. Pixel-level real-time video processing requires collaboration between the ISP and NPU, which also raises higher demands for collaborative efficiency.

In the traditional intelligent processing flow of video streams, the data exchange unit between ISP and NPU is frames. Under the ISP+NPU fusion architecture, the intra-frame and inter-frame information is directly accessed, with slice-level data interaction between ISP and NPU. The contents of the same frame can be split into smaller units, allowing for rapid data transfer between ISP and NPU, with hardware responding instantly without waiting, completing more complex tasks in a shorter time.

For example, if two people collaborate to decorate a cake, A places strawberries, and B places blueberries. A takes 1 minute to place strawberries, and B takes 1 minute to place blueberries, making the total decoration time 2 minutes. If the cake is cut into 4 pieces, A can place strawberries on 1/4 of the cake and directly pass it to B, who starts placing blueberries, and so on, reducing the entire process to 1 minute and 15 seconds, thus improving efficiency. (Note: Cutting the cake is just a metaphor; in video processing, each cut requires overlap, so each piece will be slightly larger than 1/4). In actual applications, the Kirin 9000 can complete massive computing tasks in 4K video scenarios within intervals of 33ms or even less, achieving precise color restoration with rich details in complex environments like night scenes, precisely because of the higher processing efficiency of the ISP+NPU fusion architecture, fully leveraging the strong performance of the ISP and NPU.

Efficient Joint Computing Optimization

Better Performance and Energy Efficiency Gains

The ISP+NPU fusion architecture’s other advantage lies in adopting a more efficient joint computing optimization, achieving higher performance and energy efficiency gains. It supports more efficient network integration for AI Demosaic, AI AWB, and other algorithms, allowing for faster and more efficient computing.

Despite the powerful performance upgrades, the Kirin 9000 maintains good energy efficiency. On one hand, it performs intelligent slice processing for large input data scenarios, significantly reducing the memory requirements for intermediate layer computations in the network, effectively controlling power consumption. Meanwhile, the Kirin 9000 can also intelligently search for data for low-bit mixed quantization through quantization tools while ensuring computational accuracy with FP16 high-precision calculations, further reducing power consumption by up to 20%. Additionally, slice-level data interaction effectively controls algorithm latency and reduces the input size of NN computations, benefiting from the low-power capabilities of SmartCache, resulting in better power efficiency in video scenarios. Overall, the Kirin 9000 enables an upgrade in smartphone imaging system capabilities while keeping power consumption manageable, allowing users to unleash their imaging creativity without worrying about battery life.

Huawei smartphones have always demonstrated strong performance in imaging technology, thanks to the robust support of Kirin chips. In 2015, Huawei completed its first self-developed ISP design, applied to the Kirin 950, establishing Huawei’s leadership in smartphone photography. In 2019, the Huawei P30 series innovatively supported the ultra-sensitive RYYB sensor, delivering astonishing image quality and night shooting capabilities, a technology that still stands out in the industry today. Now, with years of R&D experience in ISP and NPU, Huawei has a deeper understanding of end-to-end imaging technology. From the perspective of chip architecture, it has a comprehensive overview of the entire system, cleverly embedding NPU capabilities into the tightly integrated ISP pipeline, pioneering the ISP+NPU fusion architecture, and reaching new heights in smartphone imaging capabilities.

Scan the QR code to follow Huawei Kirin

Give a thumbs up for the clever little Kirin

Related posts

Leave a Comment Cancel reply