As one of the core modules of camera phones, the adjustment of camera sensor effects involves numerous parameters. A deep understanding of basic optical principles and the principles of sensor software/hardware in image processing can significantly enhance our work efficiency. Otherwise, lacking theoretical guidance, one can only rely on intuition and experience, often failing to grasp the key issues accurately and unable to master the core technologies of sensor debugging, thus unable to fundamentally solve problems. Therefore, in this article, the author combines some image processing principles learned from a personal interest in photography to analyze factors related to sensor image processing, sharing some insights for mutual discussion and learning.
ISP (Image Signal Processor) is used to process the image signals output by the image signal sensor. It plays a core and dominant role in the camera system and is an essential component of the camera.
Main Internal Components
As shown in the figure below, the ISP contains devices such as CPU, SUB IP, and IF. In fact, the ISP can be considered a SoC that can run various algorithm programs to process image signals in real-time.
ISP Architecture
-
CPU
The CPU, or Central Processing Unit, can run various image processing algorithms such as AF and LSC, and control peripheral devices. Modern ISPs typically use ARM Cortex-A series CPUs, such as Cortex-A5 and Cortex-A7.
-
SUB IP
SUB IP refers to various functional modules that perform specialized processing on images. Common SUB IPs include DIS, CSC, and VRA.
-
Image Transmission Interface
The image transmission interface is mainly divided into two types: parallel ITU and serial CSI. CSI stands for MIPI CSI, and due to its many advantages, MIPI-CSI interfaces are widely used in mobile camera applications for transmitting image data and various custom data. External ISPs generally include both MIPI-CSIS and MIPI-CSIM interfaces, while internal ISPs typically only require the MIPI-CSIS interface.
-
General Peripheral Devices
General peripheral devices refer to I2C, SPI, PWM, UART, WATCHDOG, etc. The ISP includes an I2C controller for reading OTP information and controlling VCM, among other functions. For external ISPs, the ISP itself is an I2C slave device. The AP can control the ISP’s operating mode and obtain its working status via I2C.
Main Functional Features
As the core component of image processing, the ISP has very important functions. The following figure illustrates the basic process of how the ISP processes image data.
ISP Processing Flow
Below are the main functional features of the ISP.
-
DEMOSAIC
DEMOSAIC is one of the main functions of the ISP. The pixels of the SENSOR are covered with a CFA, and light passes through the CFA to illuminate the pixels. The CFA consists of color filters for R, G, and B, allowing only one color to pass through each filter. Therefore, the signal output from each pixel contains information for only one of the R, G, or B colors. The data output by the SENSOR is known as BAYER data, commonly referred to as RAW data. It is evident that RAW data does not reflect the true color information. DEMOSAIC uses interpolation algorithms to calculate the true color represented by each pixel.
-
FOCUS
According to optical knowledge, the clearest image of an object is formed on the sensor when it is at the focus plane. The ISP’s FOCUS function is responsible for adjusting the position of the LENS to ensure a clear image of the object on the sensor. FOCUS can be manual or automatic. The ISP can run algorithms such as CONTRAST AF, PDAF, and LASER AF to achieve automatic focusing.
-
EXPOSURE
Exposure primarily affects the brightness of the image. The ISP needs to implement AE functionality to control the exposure level, ensuring the image brightness is appropriate.
-
WB
White balance is related to color temperature and is used to measure the color accuracy and authenticity of the image. The ISP needs to implement AWB functionality to accurately restore the true colors of objects in various complex scenes.
-
LSC
LSC is used to eliminate inconsistencies between the edges and the center of the image, including both brightness and chromaticity. The ISP needs to utilize calibration data from OTP to complete the LSC function.
-
GAMMA CORRECTION
Gamma correction adjusts the image to align with human visual perception, as the sensor’s response to light differs from that of the human eye.
-
CROP/RESIZE
Image cropping refers to changing the size of the image, which can be used to output images of different resolutions.
-
VRA
Visual recognition is used to identify specific objects, such as face recognition and license plate recognition. The ISP accurately identifies specific objects through various VRA algorithms.
-
DRC
Dynamic range correction refers to the range of brightness in an image. DRC ensures that dark areas are not underexposed and bright areas are not overexposed. The ISP needs to support DRC functionality.
-
CSC
Color space conversion, for example, the ISP converts RGB signals to YUV signals for output.
-
IS
Image stabilization (IS) primarily prevents images from becoming blurry due to slight hand movements. There are various types of IS, such as OIS, DIS, and EIS. The ISP can implement DIS and EIS.
In fact, in addition to the main functions mentioned above, the ISP also needs to support adjustment functions such as DENOISE, CONTRAST, SATURATION, and SHARPNESS.
Control Methods
The control methods discussed here refer to how the AP controls the ISP.
-
I2C/SPI
This is generally the approach for external ISPs. SPI is typically used for downloading firmware, while I2C is generally used for register control. In the kernel’s ISP driver, external ISPs are usually implemented as I2C devices and then encapsulated as V4L2-SUBDEV.
-
MEM MAP
This is generally the approach for internal ISPs, where the ISP’s internal register address space is mapped to the kernel address space.
-
MEM SHARE
This is also a method used for internal ISPs. The AP allocates memory and then passes the memory address to the ISP, effectively sharing the same memory. Therefore, operations on this shared memory by the AP will be reflected in real-time on the ISP side.
ISP Architecture Solutions
As mentioned earlier, external ISPs and internal ISPs refer to different ISP architecture solutions.
External ISP Architecture
The external ISP architecture refers to a separate ISP chip placed outside the AP for image signal processing. The architecture diagram of an external ISP is generally as follows:
External ISP Architecture Diagram
The advantages of the external ISP architecture include:
-
It can provide superior image quality.
In a fiercely competitive market, external ISP manufacturers that have survived typically possess deep expertise in this field and have accumulated rich experience in image quality tuning, enabling them to offer better performance and results than internal ISPs. Therefore, selecting a high-quality external ISP can provide professional and excellent image quality.
-
It can support more diverse design plans.
The selection of external ISPs is generally not influenced by the AP, allowing many mobile manufacturers (such as Meitu, Meizu, Huawei, etc.) to choose the most suitable components from a wide range of excellent ISP chip suppliers, leading to the design of more outstanding products.
-
It can achieve product differentiation.
Internal ISPs are packaged within the AP and are closely linked to it. If the AP is the same, the ISP will also be the same. Therefore, smartphones produced with the same AP will have identical ISP performance and fixed tuning conditions, which is not conducive to achieving product differentiation. However, by choosing an external ISP, the same AP can be paired with different models of ISPs, allowing for product differentiation and providing users with a richer and higher-quality product.
The disadvantages of the external ISP architecture include:
-
Higher costs.
External ISPs need to be purchased separately, often at a high price, and certain special functions may incur additional costs. Using external ISPs requires additional schematic design and layout, as well as the use of extra components.
-
Longer development cycles.
Designing external ISP drivers requires considerable effort and time. When using external ISPs, the ISP drivers provided by the AP supplier cannot be used, necessitating the design and writing of additional external ISP drivers. Furthermore, to achieve optimal compatibility with the AP, more debugging effort is often required. As mentioned earlier, using external ISPs requires additional schematic design and layout, as well as the use of extra components, which also takes time to address.
Internal ISP Architecture
The internal ISP architecture refers to the ISP IP embedded within the AP, directly using the internal ISP of the AP for image signal processing. The architecture diagram of an internal ISP is generally as follows:
Internal ISP Architecture Diagram
The advantages of the internal ISP architecture include:
-
It can reduce costs.
Since the internal ISP is embedded within the AP, there is no need for additional purchases like external ISPs, and it does not occupy PCB space or require separate peripheral circuit design, thus saving BOM costs and reducing overall expenses. Given that most users prioritize price when purchasing smartphones, lowering costs can effectively reduce the final product price, aiding in market capture.
-
It can accelerate product time-to-market.
Internal ISPs are closely integrated with the AP, eliminating the need for schematic and layout design, thereby shortening the development cycle and speeding up product launch.
-
It can lower development difficulty.
If using an internal ISP, the AP supplier can provide relevant documentation in advance, allowing driver developers ample time to familiarize themselves with the materials, and there will be no software version compatibility issues or platform architecture compatibility problems. However, if using an external ISP, the ISP supplier often cannot provide code/materials for a specific platform, and there are generally software version compatibility issues, requiring driver developers to invest additional effort and time.
Of course, using internal ISPs also has its drawbacks, which have been discussed earlier and will not be repeated here.
In fact, due to the importance of ISPs, to promote their APs and enhance their competitiveness, the ISPs embedded in APs are becoming increasingly powerful, with performance sufficient to meet the demands of the smartphone market. Coupled with a series of advantages, the number of smartphones using internal ISP solutions is on the rise.
Camera Effect Evaluation
What are the evaluations of camera effects?
Conclusion
The ISP is an important component of the camera system, and understanding the camera system requires familiarity with the ISP. This article introduced various aspects of the ISP from the perspectives of its internal structure, functional features, and control methods. Additionally, it provided a detailed analysis of the advantages and disadvantages of the two ISP architecture solutions, hoping to be beneficial.