Overview
1. YUV
Luma (Y) + Chroma (UV) format, generally, the sensor supports YUV422 format, which means the data format is output in the order of Y-U-Y-V. Each pixel in YUV occupies 2B, and if the pixel count is too high, the baseband chip may not handle it at high clock speeds. The data volume for JPEG is much smaller, so some mobile baseband chips can only support YUV sensors below 2M, while those above 3M can only use JPEG sensors.
2. RGB
Traditional RGB format, such as RGB565, has a 16-bit data format of 5-bit R + 6-bit G + 5-bit B. The extra bit for G is due to the human eye’s sensitivity to green. Outputting RGB is most convenient for LCD displays, so RGB565 is generally used in low-end baseband chips, directly rendering to the screen. Most baseband chips require the output to be in YUV format for further processing, as YUV outputs brightness signals without any loss, while the human eye is not particularly sensitive to color deviation. The output format of RGB565 is R5G3G3B5, which loses a lot of original information, so the image quality and stability of YUV are much better than RGB565.
3. RAW RGB
Each pixel of the sensor corresponds to a color filter, distributed according to the Bayer pattern. The data for each pixel is output directly as RAW RGB data. Each pixel of RAW data occupies 1B, resulting in a much smaller data volume. Generally, sensors above 5M only output RAW data to ensure faster output speeds, with a DSP handling the output data.
4. JPEG
Some sensors, especially low-resolution ones, come with a built-in JPEG engine, allowing them to output compressed jpg format data directly.
Differences Between RAW and JPEG
RAW is a format commonly used by professional photographers because it preserves information in its original form, allowing users to make significant adjustments to photos in post-production, such as adjusting white balance, exposure, color contrast, etc. It is also particularly suitable for beginners to salvage failed shots, and regardless of any changes made in post-production, the photo can be restored to its original state without fear of accidental loss.Another advantage of RAW is that software like Canon’s DPP can correct lens vignetting, distortion, etc.
a) Because RAW files need to retain all details and information, they are much larger than JPEG, which means storing or transferring photos takes longer and requires more storage capacity;
b) RAW files require special software to open, and if the computer does not have the software installed, the files cannot be opened. If, after 10 years, that specific software cannot be installed, previously taken photos will be inaccessible;
c) Opening RAW files with software takes longer; fast software may take 8 to 9 seconds, while slower software may take up to 20 seconds or more;
d) Different software has different ways of interpreting RAW files, so a RAW file may look different in Photoshop and Nikon Capture NX, and proprietary software sold by manufacturers can be quite expensive.
Regarding the JPEG format, the camera first processes the image based on user settings, then compresses it (the degree of compression depends on the camera’s internal photo quality settings) and saves the photo. The JPEG format is a very popular photo format, almost all modern digital cameras can use this format, and the vast majority of computers can open JPEG files. Users can also freely set the compression level to retain image quality (the best JPEG quality is very close to that of RAW), making it a very convenient format.
In summary, if you need to take a large number of photos, you should consider using JPEG, as it requires less storage and allows for post-processing and conversion to JPEG quickly. If you are doing commercial photography or enjoy post-production, you should use RAW, as it offers more post-production flexibility. If you are doing travel photography, you might consider using RAW or RAW+JPEG, as you may not be able to revisit the locations frequently, and using RAW gives you a better chance to salvage failed shots.
Data Output Formats
Data output formats are generally divided into CCIR601, CCIR656, RAW RGB, etc. The RGB format mentioned here should be either CCIR601 or CCIR656 format. The RAW RGB format differs from the general RGB format. The sensor’s light-sensing principle samples and quantizes light through individual light-sensitive points, but in the sensor, each light-sensitive point can only sense one color from RGB. Therefore, when we refer to 300,000 pixels or 1.3 million pixels, it indicates that there are 300,000 or 1.3 million light-sensitive points, each of which can only sense one color.
To restore a true image, each point must have all three RGB colors. Therefore, for CCIR601 or 656 formats, there is an ISP module inside the sensor module that interpolates and processes the data collected by the sensor. For example, if a light-sensitive point senses the color R, the ISP module will calculate the G and B values based on the values of the surrounding light-sensitive points, thus restoring the RGB value of that point, which is then encoded into 601 or 656 format for transmission to the Host.
RAW RGB format sensors directly transmit the RGB values sensed by each light-sensitive point to the Host, which performs interpolation and effect processing. Each pixel of RAW RGB has only one color (either R, G, or B), while each pixel of RGB has all three colors, with each value ranging from 0 to 255. During the testing of mobile camera sensors, the data output from the sensor is Raw data ( RAW RGB), which is then converted to RGB through color interpolation.
The data output format of the sensor is mainly divided into two types: YUV (more popular) and RGB, which is the data output of the sensor. Among these, GRB refers to RAW RGB, which is the data obtained from the sensor’s Bayer array (each type of sensor obtains the corresponding color brightness). However, the output data does not equal the actual image data; during module testing, software must be written to complete data collection (obtaining Raw data) – Color interpolation (to obtain RGB format for image display) – Image display.
This allows for the detection of whether the entire module is functioning normally, whether there are any dead pixels, dirt, etc., and to identify defective products (during the software processing, to achieve better image quality, white balance, gamma correction, and color correction are also required). In mobile applications, the phone provides an ISP (mainly for RGB format) in conjunction with software to enable the camera function. For SENSOR, both Bayer RGB and RGB Raw have the same image structure of BG/GR.
The Bayer pattern refers to the structure of the COLOR FILTER, which is divided into two types: STD Bayer pattern and Pair pattern, where the structure of the STD Bayer pattern is BG/GR, while the Pair Pattern refers to BGBG/GRGR, which is structured in four rows as a unit, with the first two rows being BG and the last two rows being GR, a structure patented by Micron, mainly used in outputting TV modes ( NTSC/PAL system).
Due to the backend application, the decoding of RAW DATA images is done according to the default structure, such as BG/GR, therefore, both Bayer RGB and RGB Raw must have the image structure of BG/GR, and if the output image structure is BGBG/GRGR, it cannot be displayed or decoded directly.The main difference between Bayer RGB and RGB Raw is that the processing before output is different; Bayer RGB outputs from ADC and has only undergone LENSSHADING CORRECTION, GAMMA, and other module processing before direct output, while RGB Raw undergoes the entire ISP module processing, ultimately being converted from YUV422 data.
CCIR601 Encoding Standard
1. Sampling Frequency
To ensure signal synchronization, the sampling frequency must be a multiple of the television signal line frequency. CCIR is a common television image sampling standard established for NTSC, PAL, and SECAM systems, where fs = 13.5MHz. This sampling frequency is exactly 864 times the line frequency of PAL and SECAM systems, and 858 times the line frequency of NTSC, ensuring synchronization between the sampling clock and the line sync signal. For 4: 2: 2 sampling format, the brightness signal is sampled at fs frequency, while the two chroma signals are sampled at fs/ 2 = 6.75MHz. Thus, the minimum sampling rate for chroma components is 3.375MHz.
2. Resolution
Based on the sampling frequency, it can be calculated that for PAL and SECAM systems, each scan line samples 864 sample points. For NTSC, it is 858 sample points. Since each line of the television signal includes certain sync and return sweep signals, the effective number of image signal sample points is not that many. CCIR601 specifies that for all systems, the number of effective sample points per line is 720 points. Due to the different effective line counts per frame for different systems ( PAL and SECAM systems have 576 lines, while NTSC has 484 lines), CCIR defines 720×484 as the basic standard for high-definition television HDTV ( High Definition TV). When displaying digital video on a computer, it is usually done using NTSC/640×480/30fps or PAL/768×576/25.
3. Data Volume
CCIR601 specifies that each sample point is digitized at 8 bits, which means there are 256 levels. However, in practice, the brightness signal occupies 220 levels, while the chroma signals occupy 225 levels, with the remaining bits used for sync, encoding, and other control purposes. If we calculate based on the sampling rate of fs, the data volume for digital video in the 4: 2: 2 sampling format is 13.5 ( MHz) × 8 ( bit) + 2 × 6.75 ( MHz) × 8 ( bit) = 27Mbyte/s. Similarly, if we calculate based on the 4: 4: 4 sampling method, the data volume for digital video is 40 megabytes per second. Based on a data rate of 27 megabytes per second, a 10-second segment of digital video would occupy 270 megabytes of storage space.
At this data rate, a 680 megabyte capacity disc can only record about 25 seconds of digital video data, and even with current high-speed optical drives, their data transfer rates do not meet the requirement of 27 megabytes per second, making real-time playback of video data impossible. This uncompressed digital video data volume is impractical for current computers and networks in terms of storage or transmission; therefore, the key issue in applying digital video in multimedia is the compression technology for digital video.
4. Differences Between CCIR601 and CCIR656
ITU-R BT 601, 16 bit data transmission ( 8 bit DATA + CLK + HSYNC + VSYNC, and also 16 bit DATA + CLK + HSYNC + VSYNC), 21 pins, Y, U and V signals are transmitted simultaneously. The transmission is parallel data, with separate outputs for line and field synchronization. The establishment of CCIR601 is a step towards the unification and standardization of digital television broadcasting system parameters. This recommendation specifies the basic parameter values for digital encoding in television studios for both the 625 and 525 line systems. Recommendation 601 specifically defines the encoding standards for television studios. It clearly specifies the encoding methods, sampling frequencies, and sampling structures for color television signals. It states that color television signals use component encoding. Component encoding means that the full color television signal is separated into brightness and chroma signals before being converted into digital form, and then each is encoded separately. The component signals ( Y, B-Y, R-Y) are encoded separately and then combined into a digital signal. It specifies the sampling frequency and sampling structure.
For example, in the 4: 2: 2 encoding, the sampling frequencies for brightness and chroma signals are specified as 13.5MHz and 6.75MHz, with the sampling structure being orthogonal, meaning it repeats by line, field, and frame, with the sampling of R-Y and B-Y sampled at odd positions ( 1, 3, 5……), while the sampling of Y is at the same position, meaning the sampling structure is fixed, and the relative position of the sampling points on the television screen remains unchanged. It specifies the encoding method. The brightness signal and the two chroma signals are linearly PCM encoded, with each sampling point quantized at 8 bits.
At the same time, it specifies that during digital encoding, the entire dynamic range of A/D conversion is not used; only 220 quantization levels are allocated to the brightness signal, with the black level corresponding to quantization level 16 and the white level corresponding to quantization level 235. For each chroma signal, 224 quantization levels are allocated, with the zero level of the chroma signal corresponding to quantization level 128. In summary, we know that the data stream for component signal encoding is very high. For example, in the 4: 2: 2 encoding standard, the bitstream is 13.5×8 + 6.75×8×2 = 216Mb/S. If using 4: 4: 4 encoding method, which directly encodes the composite signal, the sampling frequency is 13.3×8 = 106.4Mb/S.
ITU-R BT 656, 9 pins, does not require sync signals, and it achieves “soft” synchronization through 8 bit data lines ( DATA + CLK) for serial video transmission, with a transmission rate of 601 being 2 times, first transmitting Y, then transmitting UV. CCIR656 = CCIR601 + HSYNC + VSYNC656 outputs serial data, with line and field sync signals embedded in the data stream. 656 is merely a data transmission interface, which can be said to be a transmission method for 601. In simple terms, ITU-R BT 601 is the standard for “studio digital television encoding parameters”, while ITU-R BT 656 is the digital interface standard in ITU-R BT 601 Annex A, used for digital transmission standards between major digital video devices (including chips) using 27Mhz/s parallel or 243Mb/s serial interfaces.