Click belowCard to follow the “New Machine Vision” public account
Heavyweight content delivered at the first time
1. Camera Structure and Working Principle.
The scene is captured through the lens, projecting the generated optical image onto the sensor. The optical image is then converted into an electrical signal, which is further converted into a digital signal through analog-to-digital conversion. The digital signal is processed by a DSP and then sent to the computer for processing, ultimately converting into an image visible on the mobile phone screen.
The function of the digital signal processing chip DSP (DIGITAL SIGNAL PROCESSING): Mainly optimizes the parameters of the digital image signal through a series of complex mathematical algorithms, and transmits the processed signals to devices such as PCs through interfaces like USB. The DSP structure framework:
1. ISP (image signal processor)
2. JPEG encoder
3. USB device controller
The common types of camera sensors are mainly two,
one is CCD sensor (Charge Coupled Device), which is a charge coupler.
The other is CMOS sensor (Complementary Metal-Oxide Semiconductor).
The advantage of CCD is good imaging quality, but the manufacturing process is complex, cost is high, and power consumption is high. At the same resolution, CMOS is cheaper than CCD, but the image quality is slightly lower than that of CCD. The CMOS image sensor has the advantage of low power consumption compared to CCD, and with the advancement of process technology, the image quality of CMOS is continuously improving. Therefore, currently, mobile phone cameras mostly use CMOS sensors.
The Simple Structure of Mobile Phone Cameras
The filter has two main functions:
1. Filter out infrared rays. Filter out infrared light that interferes with visible light, making the imaging effect clearer.
2. Adjust incoming light. The photosensitive chip is composed of photosensitive cells. The best light is direct, but to avoid interference with adjacent photosensitive cells, the light needs to be adjusted. Therefore, the filter is not glass but quartz, utilizing the physical polarization characteristics of quartz to retain the direct part of incoming light and reflect the oblique part, avoiding interference with adjacent photosensitive points.
2. Relevant Parameters and Terms
1. Common Image Formats
1.1 RGB Format:
Traditional red, green, and blue format, such as RGB565, RGB888, its 16-bit data format is 5-bit R + 6-bit G + 5-bit B. The extra bit for G is because the human eye is more sensitive to green.
1.2 YUV Format:
Luma (Y) + Chroma (UV) format. YUV refers to the pixel format that separates luminance and chrominance parameters, and the benefit of this separation is that it can avoid mutual interference and reduce the chrominance sampling rate without significantly affecting image quality. YUV is a general term, and its specific arrangement can be divided into many specific formats.
Chrominance (UV) defines two aspects of color - hue and saturation, represented by Cb and Cr respectively. Among them, Cr reflects the difference between the red part of the RGB input signal and the luminance value of the RGB signal. Cb reflects the difference between the blue part of the RGB input signal and the luminance value of the RGB signal.
The main sampling formats are YCbCr 4:2:0, YCbCr 4:2:2, YCbCr 4:1:1, and YCbCr 4:4:4.
1.3 RAW Data Format:
RAW images are the original data captured by the CMOS or CCD image sensor converting light source signals into digital signals. RAW files record the original information of the digital camera sensor, as well as some metadata generated by the camera (such as ISO settings, shutter speed, aperture value, white balance, etc.). RAW is an unprocessed and uncompressed format, which can be conceptualized as "raw image encoding data" or more vividly as "digital negatives." Each pixel of the sensor corresponds to a color filter, and the filter is distributed according to the Bayer pattern. Each pixel's data is directly output, i.e., RAW RGB data.
RAW data (Raw RGB) becomes RGB after color interpolation.
RAW Format Image Example
2. Related Technical Indicators
2.1 Image Resolution:
SXGA (1280 x 1024) also known as 1.3 million pixels
XGA (1024 x 768) also known as 800 thousand pixels
SVGA (800 x 600) also known as 500 thousand pixels
VGA (640 x 480) also known as 300 thousand pixels (350 thousand refers to 648X488)
CIF (352 x 288) also known as 100 thousand pixels
SIF/QVGA (320 x 240)
QCIF (176 x 144)
QSIF/QQVGA (160 x 120)
2.2 Color Depth:
256 grayscale, with 256 shades of gray (including black and white).
15 or 16-bit color (high color): 65,536 colors.
24-bit color (true color): Each primary color has 256 levels, and their combination yields 256*256*256 colors.
32-bit color: In addition to the 24-bit color, an additional 8-bit is used to store graphic data for overlapping layers (alpha channel).
2.3 Optical Zoom and Digital Zoom:
Optical zoom: Adjusting the lens to zoom in or out on the subject while maintaining pixel and image quality.
Digital zoom: Actually no zoom, just cropping from the original image and enlarging it. You see it enlarged on the LCD screen, but the image quality does not improve fundamentally, and the pixel count is lower than the maximum the camera can capture. In terms of image quality, it is basically a gimmick but can provide some convenience.
2.4 Image Compression Methods:
JPEG/M-JPEG
H.261/H.263
MPEG
H.264
2.5 Image Noise:
Refers to the noise in the image. Manifested as fixed colored noise points in the image.
2.6 Auto White Balance Processing Technology:
Simply put: the camera restores white objects. Related concepts: color temperature.
2.7 Viewing Angle:
The imaging principle is similar to that of the human eye, simply put, it is the imaging range.
2.8 Auto Focus:
Auto focus can be divided into two main categories: one is distance measurement auto focus based on the distance between the lens and the subject, and the other is focus detection auto focus based on clear imaging on the focus screen (clarity algorithm).
Note: Zooming is bringing distant objects closer. Focusing is making the image clear.
2.9 Auto Exposure and Gamma:
It is the combination of aperture and shutter speed. Aperture, shutter speed, ISO. Gamma is the response curve of human eyes to brightness.
3. Qualcomm's CAMERA Hardware Architecture
CAMERA Hardware Architecture
VFE: VIDEO front-end
VPE: Video preprocessing
The camera module has an ISP (image signal processor), so the functions related to image effect processing of VFE and VPE are disabled.
1. VFE Functions:
1.1 Improve image quality through algorithms.
1.2 Provide high-resolution image AWB (auto white balance)/AE (auto exposure)/AF (auto focus) algorithm processing.
1.3 Image attenuation correction.
1.4 Noise filtering in low light.
1.5 Image color effect optimization.
1.6 Skin color effect optimization.
1.7 Image shake calculation.
1.8 Brightness adaptation algorithm.
2. VPE Functions:
2.1 Image stability.
2.2 Digital focus.
2.3 Image rotation.
2.4 Overlay.
3. Basic Architecture of the Android System Camera
1. Application Layer
The application layer of Camera on Android is represented by a Camera application APK package developed by directly calling SDK API. The code is located at /android/packages/apps/Camera. It mainly calls the android.hardware.Camera class (in Framework) and implements the business logic and UI display of the Camera application. If an Android application wants to use this android.hardware.Camera class, it needs to declare Camera permission in the Manifest file, and also needs to add some elements to declare Camera features in the application, such as auto focus, etc. The specific approach can be as follows:
<uses-permission android:name = "android.permission.CAMERA" />
<uses-feature android:name = "android.hardware.camera" />
<uses-feature android:name = "android.hardware.camera.autofocus" />
2. Framework Layer
2.1 android.hardware.Camera: Code location /android/frameworks/base/core/java/android/hardware/Camera.java
This part targets framework.jar. This is the Java interface provided by Android for app layer calls. This class is used to connect or disconnect a Camera service, set shooting parameters, start and stop previews, take photos, etc.
2.2 The android.hardware.Camera class is the same as the class defined in JNI, some methods are called by JNI to obtain local code, and some methods are implemented by themselves.
The JAVA native call part of Camera (JNI): /android/frameworks/base/core/jni/android_hardware_Camera.cpp. Camera.java serves as a bridge from JAVA code to C++ code. It is compiled into libandroid_runtime.so. The libandroid_runtime.so library is shared, and besides Camera, it has functionalities in other areas.
2.3 Client part of the Camera framework:
Code location: /android/frameworks/base/libs/camera/ five files under.
Camera.cpp
CameraParameters.cpp
ICamera.cpp
ICameraClient.cpp
ICameraService.cpp
Their header files are located in /android/frameworks/base/include/camera directory.
This part is compiled into libcamera_client.so. In the various libraries of the Camera module, libcamera_client.so is core, serving as the client part of the Camera framework, communicating with the other part of the service side libcameraservice.so through inter-process communication (i.e., Binder mechanism).
2.4 Service part of the Camera framework:
Code location: /android/frameworks/base/services/camera/libcameraservice.
This part is compiled into the library libcameraservice.so. CameraService is the Camera service, the middle layer of the Camera framework, used to link CameraHardwareInterface and the Client part, implementing functions by calling the actual Camera hardware interface, i.e., the lower HAL layer.
4. Basic Data Flow and Processing Process for Camera Preview, Shooting, and Video Recording, as well as Driver Debugging
HAL layer related code: (frameworks/base/services/camera/libcameraservice/CameraService.cpp) vendor/qcom/android-open/libcamera2/QualcommCameraHardware.cpp vendor/qcom/proprietary/mm-camera/apps/appslib/mm_camera_interface.c vendor/qcom/proprietary/mm-camera/apps/appslib/camframe.c vendor/qcom/proprietary/mm-camera/apps/appslib/snapshot.c vendor/qcom/proprietary/mm-camera/apps/appslib/jpeg_encoder.c vendor/qcom/proprietary/mm-camera/apps/appslib/cam_frame_q.c vendor/qcom/proprietary/mm-camera/apps/appslib/cam_display.c vendor/qcom/proprietary/mm-camera/targets/vfe31/8x60/vendor/qcom/proprietary/mm-camera/targets/vfe31/common/vpe1/QualcommCameraHardware.cpp is mainly divided into three parts: preview, snapshot, video. Each is processed by a pthread. In addition, the auto focus function is also processed in a pthread. The data frames obtained from previewing, shooting, or video threads are callback to the upper layer CameraService.cpp through datacallback for storage or preview operations. The following is a rough structure of the calling flow of the HAL layer code.
The entire module mainly runs three main threads: control, config, and frame.
Control is for executing overall control, which is the upper layer control interface.
Config mainly performs some configurations; this thread mainly handles 3A work and some effect-related settings;
Frame thread is mainly for frame queue cyclic retrieval processing. All event or status feedback is returned to QualcommCameraHardware.cpp via callback functions.
2. The driver part starts from the device driver s5k8aa.c. After creating the platform device, when executing the entry function probe, it calls the camera device creation function
int msm_camera_drv_start(struct platform_device *dev,
int (*sensor_probe)(const struct msm_camera_sensor_info *,
struct msm_sensor_ctrl *))
and passes the device information structure and the camera device calling entry sensor_probe. The msm_camera_drv_start(xxx) function is implemented in msm_camera.c. It creates four device nodes for upper layer calls:
/dev/msm_camera/frame%d
/dev/msm_camera/control%d
/dev/msm_camera/config%d
/dev/msm_camera/pic%d
It implements the control calling interface for the upper layer library to the VFE module, VPE module, jpeg_encoder module, and camera sensor module. The corresponding functions in file_operations implement the initialization and IOCTL function call interfaces for these devices.
Then this function also creates four work queues:
struct msm_device_queue event_q;
struct msm_device_queue frame_q;
struct msm_device_queue pict_q;
struct msm_device_queue vpe_q;
event_q includes the control signal queue passed in by /dev/msm_camera/control%d, used to pass control commands from the upper layer to the config thread.
frame_q is used for managing image frame operations; during preview or recording, frames will be passed to DSP for processing.
pict_q contains photos taken, used for image encoding processing by jpeg_encoder.
vpe_q is the VPE control command queue.
s5k8aa.c is the driver part of the corresponding camera device. Its function is simple, mainly implementing the creation, initialization, and control of the sensor module. It mainly implements the following three functions:
s->s_init = ov2685_sensor_init;
s->s_release = ov2685_sensor_release;
s->s_config = ov2685_sensor_config;
ov2685_sensor_init function:
Mainly implements power on, clock control (MCLK), and device initialization functions. Power on is divided into DOVDD, DVDD, AVDD, reset, PWDN, etc. It needs to be operated in order according to device requirements; generally, the clock control order is also included. The device initialization process initializes all registers of the sensor device, sending the initialization register addresses and values to the sensor via IIC. After completion, the camera module can work normally and transmit images to the CPU via the MIPI line.
ov2685_sensor_config function:
Mainly implements various configuration interfaces for the sensor, including frame rate configuration, white balance effect settings, exposure settings, special effect settings, etc. The corresponding interface sends the configured register list to the sensor via IIC.
3. Several issues in camera debugging:
1.1 Whether powering on is correct, whether there is clock waveform output. Detect the output voltage value to see if it meets the power-on timing and whether the MCLK meets the sensor's requirements. This part can be measured with an oscilloscope and multimeter. Measure whether the voltage values and power-on timing and MCLK frequency are correct.
1.2 Whether IIC read and write are normal. Debugging the I2C communication between CPU and ISP. Check if the IIC address is correct and if the protocol matches. This part can also be measured with an oscilloscope to check the peak and waveform logic of IIC's SDA and CLK.
1.3 After correct power-on and initialization, whether the sensor module works normally. This part mainly checks the data and clock PIN of the MIPI line with an oscilloscope to see if it contains data, whether the waveform is standard, and whether the peak values meet the requirements.
1.4 If all of the above are correct, the MIPI controller will receive an interrupt and start processing the image signal. If an error occurs at this point, you can check the error status through the error value of the interrupt signal. Besides checking whether the CPU side is working normally, pay attention to whether the image format and size set on the module match the default image format and size received by the CPU. The image format and size in the module can be checked through register values. The image format and size received by the CPU are set in the HAL part of s5k8aa, and the source image size for shooting and previewing needs to be set separately.
After completing the above part, the camera can preview correctly.
Source: OSCHINA Community Author: luoshi129
Declaration: Some content comes from the internet and is for readers' learning and communication purposes only. The copyright of the article belongs to the original author. If there are any inappropriate aspects, please contact for deletion.
