DIY Camera Part 1: Using the Libcamera Library

To build a DIY camera, the first step is to determine the camera model to use. For compatibility with Raspberry Pi and better image quality, there are currently two main options: the Raspberry Pi Camera Module 3 and the Raspberry Pi HQ Camera.

The Raspberry Pi Camera Module 3 uses the Sony IMX708 sensor, which has been used in flagship phones like the OPPO Find X2.

The parameters of the IMX708 sensor are shown in the table below:

Let’s talk about the rolling shutter. A Rolling Shutter is a way that camera sensors operate, in contrast to the Global Shutter.

In a Rolling Shutter, the camera sensor exposes each row from top to bottom sequentially, like a rolling curtain. This means that during a single exposure, different rows of pixels may have a slight time difference.

This method usually does not cause issues when capturing slowly moving subjects or when the camera is stationary. However, when capturing fast-moving objects or when the camera itself is moving, the Rolling Shutter may cause strange distortion effects in the image, known as the “rolling shutter effect”.

In contrast, a Global Shutter exposes the entire image sensor instantaneously at the same time, rather than row by row. This means that when capturing moving objects or when the camera is in motion, a Global Shutter will not cause distortion effects, but the camera may be more expensive because sensors that implement global shutter technology are typically more complex.

Therefore, when selecting a camera, consider the shooting conditions. If you need to capture fast-moving objects, you may need to prioritize cameras that support Global Shutter.

The second option, the Raspberry Pi HQ Camera, has the advantage of separating the CMOS from the lens, allowing for different lenses to be attached as needed, such as telephoto or zoom lenses, and it uses the IMX477 sensor. We will provide detailed information about this camera when needed (mainly because it is quite expensive, the CMOS and lens cost over 1k).

In addition to considering the camera, compatibility with Raspberry Pi must also be taken into account. The table below shows the correspondence between different sensors and Raspberry Pi boards and drivers.

From the table, we can see that the IMX708 sensor only supports the libcamera driver, not the older raspicam driver.

Using the libcamera driver means that when installing the Raspberry Pi system, the corresponding Debian version must be higher than _Bullseye_, for example, I am using Debian version: 12 (bookworm).

When using the bookworm system, the libcamera library is directly usable after the system installation is complete.

The Raspberry Pi will also automatically connect to the camera. However, first, confirm that the content of the /boot/config.txt file is as shown below, especially the camera_auto_detect=1 item.

For the Raspberry Pi Camera Module 3, the Raspberry Pi can automatically detect it without needing us to manually specify the dtoverlay. When we use the Raspberry Pi HQ camera later, which uses the IMX477 sensor, we will need to manually specify the dtoverlay model.

# For more options and information see# http://rptl.io/configtxt# Some settings may impact device functionality. See link above for details
# Uncomment some or all of these to enable the optional hardware interfaces#dtparam=i2c_arm=on#dtparam=i2s=on#dtparam=spi=on
# Enable audio (loads snd_bcm2835)dtparam=audio=on
# Additional overlays and parameters are documented# /boot/firmware/overlays/README
# Automatically load overlays for detected camerascamera_auto_detect=1
# Automatically load overlays for detected DSI displaysdisplay_auto_detect=1
# Automatically load initramfs files, if foundauto_initramfs=1
# Enable DRM VC4 V3D driverdtoverlay=vc4-kms-v3dmax_framebuffers=2
# Don't have the firmware create an initial video= setting in cmdline.txt.# Use the kernel's default instead.disable_fw_kms_setup=1
# Run in 64-bit modearm_64bit=1
# Disable compensation for displays with overscandisable_overscan=1
# Run as fast as firmware / board allowsarm_boost=1
[cm4]# Enable host mode on the 2711 built-in XHCI USB controller.# This line should be removed if the legacy DWC2 controller is required# (e.g. for USB device mode) or if USB support is not required.otg_mode=1
[all]

sudo apt updatesudo apt upgrade

sudo libcamera-hello -t 0

libcamera-jpeg -o test.jpg

This capture command will display a preview on the serial port for about 5 seconds, then take a full-resolution JPEG image and save it as test.jpg.

Additionally, the Raspberry Pi’s libcamera driver will call a tuning file for different camera modules, which provides various parameters. When calling the camera, libcamera will use the parameters from the tuning file and combine them with algorithms to process the image and output it as a preview.

Since the libcamera driver can only automatically sense the chip signal, the final display effect of the camera will also be affected by the entire module. The use of tuning files is to flexibly handle different camera modules and adjust to improve image quality.

Here, we will call the tuning file provided in the system for the IMX708.

libcamera-jpeg -o test1.jpg --tuning-file /usr/share/libcamera/ipa/rpi/vc4/imx708.json

However, by using FastStone software to compare, the imaging of the two images does not show much difference. The tuning file is also applicable to other libcamera commands.

You can also specify the output resolution and preview window preview time. -t 2000 indicates a 2s preview time.

libcamera-jpeg -o test.jpg -t 2000 --width 640 --height 480

libcamera-jpeg --ev -0.5 -o darker.jpglibcamera-jpeg --ev 0 -o normal.jpglibcamera-jpeg --ev 0.5 -o brighter.jpg

We will explain the following terms:

AEC (Automatic Exposure Control) and AGX (Automatic Gain Control) are two commonly used automatic control algorithms in image processing, typically used in camera or camcorder systems to adjust the exposure and gain of images to ensure appropriate image quality under different lighting conditions.

AEC (Automatic Exposure Control): The AEC algorithm controls the image brightness by adjusting the shutter speed (exposure time). In low light conditions, it increases the exposure time to enhance brightness, while in high light conditions, it reduces the exposure time to avoid overexposure.

AGX (Automatic Gain Control): The AGX algorithm controls brightness by adjusting the amplification of the image signal. It can increase the amplification level of the image signal in low light conditions to improve brightness.

EV represents Exposure Value, a camera setting used to adjust the exposure level of photos. Changes in EV values will affect the camera’s shutter speed, aperture, and ISO settings, thus changing the brightness and details of the photo.

The EV value is a logarithmic indicator, typically expressed as “EV +/- value”, such as +1 EV or -2 EV. Adjustments to EV values are generally made in increments of 1/3 or 1/2 EV.

Positive EV values indicate that the camera will increase exposure, making the image brighter. Negative EV values indicate that the camera will decrease exposure, making the image darker.

Exposure compensation is commonly used in complex situations with changing lighting conditions, or when the photographer wishes to make adjustments outside of the camera’s automatic settings. For example, when photographing high-contrast scenes, you might use exposure compensation to avoid overexposure or underexposure.

The three photos above were taken by adjusting the EV values to achieve different exposure levels. The following table shows the EXIF information of the three images.

From the shutter and ISO data of these three photos, we can see that the different exposure levels obtained by adjusting the EV values were achieved by adjusting the shutter and ISO data.

Next, let’s look at how libcamera achieves different brightness images by controlling gain.

libcamera-jpeg -o normal.jpg -t 2000 --shutter 25000 --gain 0libcamera-jpeg -o brighter.jpg -t 2000 --shutter 25000 --gain -1libcamera-jpeg -o darker.jpg -t 2000 --shutter 25000 --gain 1

The Raspberry Pi Camera Module 3 has a fixed aperture size, and the shutter speed is also fixed at 25ms. Currently, only ISO is adjustable.

It can be seen that the –gain parameter is also used to obtain different exposure levels of photos by controlling the ISO value.

libcamera-still -o test.jpg

This will yield an image similar to that obtained by libcamera-jpeg, with the same storage space occupied.

libcamera-still can specify different encoders through the -e parameter, allowing saving in different formats. It supports PNG and BMP encoding, and also directly dumps RGB or YUV pixel binary data to a file without encoding.

If saving RGB or YUV data directly, the program must understand the pixel arrangement of the file when reading such files.

libcamera-still -e png -o test.pnglibcamera-still -e bmp -o test.bmplibcamera-still -e rgb -o test.datalibcamera-still -e yuv420 -o test_yuv.data

PNG and JPEG are two common formats, and here we introduce their differences:

PNG (Portable Network Graphics) and JPEG (Joint Photographic Experts Group) are two common image file formats, and they have significant differences in several aspects.

1. Compression Algorithm:

○ PNG uses a lossless compression algorithm, preserving all image details without losing image quality. This makes the PNG format suitable for scenarios that require high-quality detail, such as graphic design and line art.

○ JPEG uses a lossy compression algorithm, reducing file size by eliminating some details and color information from the image, thus lowering image quality. This makes JPEG suitable for photos and other situations where smaller file sizes are needed.

2. Color Depth:

○ PNG supports various color modes such as indexed color, grayscale, RGB, and RGBA, and supports both 16-bit and 8-bit color depth, suitable for images with rich colors.

○ JPEG is mainly used for saving photos, typically stored in 8-bit RGB mode.

3. Transparency:

○ PNG supports full and semi-transparency, allowing for complex transparency effects in images, suitable for graphic design and scenarios that require transparent backgrounds.

○ JPEG does not support transparency and can only display solid color backgrounds.

4. File Size:

○ JPEG files are usually much smaller than PNG files of the same resolution and quality because they are lossy compressed, which can reduce file size to some extent.

○ PNG files are relatively larger because they are losslessly compressed and retain all image details.

5. Applicable Scenarios:

○ PNG is suitable for images that require high-quality detail, transparency, or lossless compression, such as graphic design, icons, and line art.

○ JPEG is suitable for photos and other scenarios where smaller file sizes are needed.

In summary, the choice between PNG and JPEG depends on the specific use case and requirements of the image.

If high-quality detail, transparency support, or lossless compression is needed, PNG is a better choice. If the goal is mainly to save photos or achieve smaller file sizes, JPEG may be more suitable.

It can be seen that the size of the jpg file is 1.2M, while the png file size is 10.7M. However, from the comparison images, there is no significant quality advantage of png over jpg.

A raw image is the direct output from the image sensor, without any processing by ISP or CPU. In post-processing photography, using raw images is particularly important for the following reasons:

Using RAW format images has advantages over JPEG format, especially in post-processing:

1. Higher Dynamic Range: RAW images contain the original data captured by the camera sensor, so they typically have a higher dynamic range than JPEG. This means you can better adjust highlights and shadows during post-processing to retain more detail.

2. More Color Depth: RAW images are typically stored with higher color depth (such as 14-bit or 16-bit), while JPEG is usually 8-bit. This allows for finer adjustments to color and tone during post-processing.

3. Greater Flexibility: Since RAW images are uncompressed or minimally compressed, you can make a wider range of adjustments during post-processing, such as white balance, exposure, contrast, etc.

4. Lossless Compression: RAW images typically use lossless compression, meaning the image does not lose any detail or quality, even after multiple saves.

5. Avoids Camera Internal Processing: JPEG images often undergo internal processing by the camera, including sharpening, noise reduction, etc. Using RAW images allows for better control over the appearance of the image in post-processing.

6. Better White Balance Control: RAW images allow for more precise adjustments to white balance during post-processing, while JPEG typically locks in a single white balance.

However, using RAW format also has some drawbacks:

1. File Size: RAW files are typically larger than JPEG files because they contain uncompressed original data.

2. Requires Post-Processing: Compared to JPEG, RAW images usually require more post-processing work because they have not undergone automatic processing by the camera.

3. Requires Special Software: Specialized software (such as Adobe Camera Raw, Lightroom, etc.) is often needed to process RAW format images.

4. Greater Storage Requirements: Since RAW files are usually larger, you may need more storage space to save these files.

In summary, if you value flexibility and control in post-processing photography, using RAW format may be a good choice. However, if you want to quickly capture and share photos, JPEG format may be more convenient.

For color camera sensors, the raw image output format is generally Bayer. Note that raw images are different from RGB and YUV images mentioned earlier, as RGB and YUV are processed images after ISP.

To capture a raw image, use the following command:

libcamera-still -r -o test.jpg

Raw images are typically saved in DNG (Adobe Digital Negative) format, which is compatible with most standard programs, such as dcraw, RawTherapee, or HoneyView. The raw image will be saved with the same name as the .dng suffix, for example, if you run the command above, it will be saved as test.dng, and a jpeg file will be generated simultaneously.

The DNG file contains metadata related to the image acquisition, such as white balance data, ISP color matrix, etc. Below is the metadata encoding information displayed using the exiftool tool:

---- ExifTool ----ExifTool Version Number         : 12.69---- File ----File Name                       : test.dngDirectory                       : .File Size                       : 24 MBFile Modification Date/Time     : 2023:10:29 17:18:03+08:00File Access Date/Time           : 2023:10:29 17:18:03+08:00File Creation Date/Time         : 2023:10:29 17:18:03+08:00File Permissions                : -rw-rw-rw-File Type                       : DNGFile Type Extension             : dngMIME Type                       : image/x-adobe-dngExif Byte Order                 : Little-endian (Intel, II)---- EXIF ----Subfile Type                    : Reduced-resolution imageImage Width                     : 288Image Height                    : 162Bits Per Sample                 : 8 8 8Compression                     : UncompressedPhotometric Interpretation      : RGBMake                            : Raspberry PiCamera Model Name               : imx708Strip Offsets                   : 8Orientation                     : Horizontal (normal)Samples Per Pixel               : 3Strip Byte Counts               : 139968Planar Configuration            : ChunkySoftware                        : libcamera-stillSubfile Type                    : Full-resolution imageImage Width                     : 4608Image Height                    : 2592Bits Per Sample                 : 16Compression                     : UncompressedPhotometric Interpretation      : Color Filter ArrayStrip Offsets                   : 140624Samples Per Pixel               : 1Strip Byte Counts               : 23887872Planar Configuration            : ChunkyCFA Repeat Pattern Dim          : 2 2CFA Pattern 2                   : 2 1 1 0Black Level Repeat Dim          : 2 2Black Level                     : 64 64 64 64White Level                     : 1023Compression                     : UncompressedStrip Offsets                   : 0Strip Byte Counts               : 0Exposure Time                   : 1/16ISO                             : 400Date/Time Original              : 2023:10:29 17:18:03Subject Distance                : 0.8847590685 mDNG Version                     : 1.1.0.0DNG Backward Version            : 1.0.0.0Unique Camera Model             : Raspberry Pi imx708Color Matrix 1                  : 0.9413505197 -0.32803303 -0.09434454143 -0.2564420402 1.072353721 0.1573984027 -0.02698263898 0.1290469915 0.4360769093As Shot Neutral                 : 0.4639694691 1 0.5782194734Calibration Illuminant 1        : D65---- Composite ----CFA Pattern                     : [Blue,Green][Green,Red]Image Size                      : 4608x2592Megapixels                      : 11.9Shutter Speed                   : 1/16

This is the extraction result of the Exif information for an image named test.dng. The Exif information contains detailed metadata about the photo.

Here are explanations for some key pieces of information:

1. ExifTool Version Information:

ExifTool Version Number: 12.69

2. File Information:

File Name: test.dng

File Size: 24 MB

File Modification Date: 2023-10-29 17:18:03 (UTC+8)

File Access Date: 2023-10-29 17:18:03 (UTC+8)

File Creation Date: 2023-10-29 17:18:03 (UTC+8)

File Permissions: -rw-rw-rw-

File Type: DNG (Digital Negative Format)

MIME Type: image/x-adobe-dng

Exif Byte Order: Little-endian (Intel, II)

3. EXIF Information

Full-resolution Image:

Image Width: 4608 pixels

Image Height: 2592 pixels

Bits Per Sample: 16 bits

Compression: Uncompressed

Photometric Interpretation: Color Filter Array

Exposure Time: 1/16 seconds

ISO: 400

Date/Time Original: 2023-10-29 17:18:03

Subject Distance: 0.8847590685 meters

DNG Version: 1.1.0.0

Camera Model: Raspberry Pi imx708

Reduced-resolution Image:

Image Width: 288 pixels

Image Height: 162 pixels

Bits Per Sample: 8 bits

Compression: Uncompressed

Photometric Interpretation: RGB

Camera Manufacturer: Raspberry Pi

Camera Model: imx708

…

(There are many other EXIF information items, such as color correction matrix, exposure settings, etc.)

4. Composite Information:

CFA Pattern: [Blue,Green][Green,Red]

Image Size: 4608×2592 pixels

Megapixels: 11.9 MP (megapixels)

This EXIF information provides detailed information about the image, including the capturing device, shooting conditions, image resolution, etc.

Long exposure is generally used in scenarios where we want to capture flowing light shutter effects. By long exposure, we can record the changes in the scene over a period of time.

However, in the Raspberry Pi Camera Module 3 we are using, long exposure is not very useful.

Including in professional modes on phones without variable apertures, setting long exposure will not yield flowing light shutter effects.

To achieve flowing light shutter effects, not only must the shutter setting be long enough, but the aperture must also be set particularly small, and ND1000, ND64 or similar neutral density filters must be added, along with setting ISO to the minimum, to achieve a flowing light shutter effect.

Here is a brief introduction to the three elements of exposure:

The three elements of exposure refer to the three key factors in photography that control photo exposure, which are:

1. Aperture:

The aperture refers to the size of the camera lens opening. It is represented by a number called the aperture value (f).

The smaller the aperture value, the larger the aperture, allowing the camera to receive more light, resulting in a brighter photo.

The larger the aperture value, the smaller the aperture, allowing less light to enter the camera, resulting in a darker photo.

The aperture also affects depth of field; a larger aperture (smaller aperture value) produces a greater background blur effect.

2. Shutter Speed:

Shutter speed refers to the duration of time the camera sensor or film is exposed to light.

The faster the shutter speed, the shorter the time the camera receives light, resulting in a darker photo, but it can freeze motion.

The slower the shutter speed, the longer the time the camera receives light, resulting in a brighter photo, but it may cause motion blur.

3. ISO Sensitivity:

ISO value represents the sensitivity of the camera sensor to light. A higher ISO value means the camera can use a faster shutter speed under the same lighting conditions but may introduce image noise.

A lower ISO value reduces noise but may require slower shutter speeds or larger apertures to maintain proper exposure.

These three elements influence each other, known as the “exposure triangle”. Adjusting one parameter will affect the other two. For example, if you increase the aperture value (reduce the aperture), you can choose a faster shutter speed to maintain the same exposure or lower the ISO to reduce noise.

Understanding and mastering these three elements of exposure will allow you to better control the exposure level of your photos, resulting in more satisfying images.

When using mirrorless or DSLR cameras, experienced photographers often use M mode (manual mode) for shooting, where they choose appropriate aperture, shutter, and ISO values based on the scene.

For example, when shooting fast-moving sports, a higher shutter speed, such as 1/400s, is needed to ensure that the resulting photos are not blurry, using the largest aperture to ensure enough light enters, such as using a large aperture of F1.8.

At this time, to ensure normal exposure levels, you can let the machine decide the ISO value or specify the ISO value yourself.

If using a mirrorless or DSLR camera to achieve flowing light shutter effects, common parameter settings are: shutter speed 30s, aperture F16 or the minimum aperture supported by the lens, and ISO set to the lowest supported by the machine, adding ND1000 or ND64 neutral density filters.

When using a phone to capture flowing light shutter effects, it is not the same as the pure optical imaging effect of a DSLR; it is a result of algorithmic calculations.

Therefore, similar to using a phone, when using the Raspberry Pi Camera Module 3, since the aperture size is fixed, even if long exposure is enabled, flowing light shutter effects cannot be achieved because a large aperture would lead to excessive light intake and overexposure. Post-processing algorithms are needed to achieve flowing light shutter effects.

Here, we will briefly introduce the long exposure API for libcamera.

To take a long exposure image, we need to disable AEC/AGC and white balance; otherwise, these algorithms will cause the image to wait for many frames during convergence.

Disabling these algorithms requires setting explicit values; additionally, users can use the –immediate setting to skip the preview process.

libcamera-still -o long_exposure.jpg --shutter 100000000 --gain 1 --awbgains 1,1 --immediate

Note: Reference table for the longest exposure times of several official cameras.

This article mainly explains the camera selection and basic usage of the underlying libcamera when using Raspberry Pi to DIY a camera.

The features provided by libcamera are far more than this; all parameters that can be set when shooting in M mode with a DSLR/mirrorless camera can be set through the libcamera library.

However, libcamera is only a command used in the terminal; to build a camera, we need our own camera app and gallery app, which requires us to call it in a higher-level language. In the next section, we will introduce the picamera2 library that wraps libcamera.