Introduction

The visual camera is key to machine intelligence/automation. NVIDIA Jetson supports cameras from multiple partners, including USB, Ethernet, and MIPI interfaces. Jetson provides a complete Jetpack SDK for these cameras, allowing users to quickly integrate these cameras for their designs.

The list of cameras supported by Jetson can be found on the official website, as shown in the link below:

By selecting the CSI/MIPI interface and the Orin kit, a list of cameras supported by Jetson Partner will appear on the webpage, mainly: Sony’s IMX290, IMX334, IMX179, IMX335, IMX415, and Aptina’s AR0234, AR0144, AR0822, etc. Therefore, for example, if a user uses NVIDIA’s Jetson AGX Orin developer kit, the SDK of the kit natively supports the sensors listed above for CSI integration, allowing users to quickly develop their solutions.

If you choose the cameras from the above list and pair them with the corresponding CSI adapter board, since the Camera Partner has already connected with various models of NVIDIA Jetson in advance, the drivers in the original factory image are already supported, allowing for plug-and-play functionality.

However, if we select sensor models outside the list, i.e., non-Camera Partner products, they cannot be used directly. For example, if I want to use Sony’s IMX472, IMX252, or STMicroelectronics’ SC2210, SC235HGS, etc., then hardware matching and driver software upgrades are necessary. For this reason, Aowei provides a complete solution that can carry any camera and can also integrate ISP using FPGA.

FPGA Simulation of MIPI Camera Integration into Jetson Solution

As shown in the figure below: The solution uses the Efinix Titanium 16nm FPGA Ti60F225 to collect image sensor data through MIPI CSI RX. After a series of complex ISP calculations inside the FPGA, the image data is sent to the backend via MIPI CSI TX.

Among them, VC-MIPI-AGX is Aowei’s self-developed MIPI access sub-card for the NVIDIA Jetson AGX Orin developer kit. As shown in the figure below, it is the MIPI slot of AGX and the access sub-card we designed. We use a high-speed and stable Type-C 3.0 data cable to transmit MIPI signals. The current solution supports 2 MIPI CSI channels, with a single channel supporting a maximum bandwidth of 6Gbps. The current solution has successfully integrated with AGX, is stable and reliable, and has passed burn-in testing.

Efinix MIPI Introduction

The low power consumption, high bandwidth, and small size features of Efinix’s 16nm Titanium series FPGA empower terminal products to be innovative and competitive. The Titanium series Ti60 can support multiple camera inputs with MIPI/LVDS interfaces, drive various types of screens, and high-performance RISC-V soft cores, suitable for various camera and sensor system applications.

Therefore, we chose Efinix’s FPGA as the main control to fully utilize its high-speed and low-power features, completing the MIPI CSI collection, ISP image processing, and MIPI CSI transmission. The MIPI PHY of Efinix Ti60 FPGA supports 1.5Gbps single channel and 4 lanes support 6Gbps. At 85% effective bandwidth, the effective transmission resolution/frame rate is evaluated as follows:

Using RGB888 or Bayer 8bit transmission has its pros and cons:

• RGB888 8bit transmission allows AGX to directly acquire image data, which is beneficial as there is no additional source data processing overhead; the downside is due to the 3x bandwidth of Bayer, the current FPGA solution can only support 1080P60 solutions.

• Bayer 8bit transmission has a bandwidth of 1/3 of RGB888, thus it can support higher resolutions (4K60). At the same resolution, it can support longer transmission distances compared to RGB888. The downside is that AGX’s CPU (Neon) needs to process the Bayer2RGB algorithm. Currently, testing at 1080P120 consumes 43% of one CPU’s resources (out of a total of 12 CPUs, single-core resource consumption is shown in the figure below).

In summary, since currently only one CPU of AGX is used, and the user’s algorithm mainly runs on the GPU using CUDA, the advantages of using Bayer transmission are more significant.

Aowei ISP Introduction

NVIDIA can also perform complete ISP image processing, and even the ISP functions and performance implemented in ASIC are stronger than FPGA. However, in this solution, we have to use FPGA to perform complete ISP processing for the following main reasons:

• For the selected sensor, NVIDIA still requires a lot of Tuning work;

• NVIDIA’s ISP does not support 8bit RAW input mode;

• Some application engineers are not familiar with the ISP field;

• Some ISP algorithms are not included by NVIDIA;

• Low latency requirements necessitate pipeline completion of ISP.

In view of this, we have used FPGA to complete a complete ISP processing flow. The MIPI CSI directly outputs the data processed by the FPGA, and once NVIDIA AGX receives the RGB data, users can directly proceed with subsequent application algorithms. Based on Efinix FPGA’s ISP image processing pipeline, as shown below:

Aowei’s self-developed pipeline ISP and related Features are shown in the table below:

AGX Orin Access Driver

In terms of hardware, the VC-MIPI-AGX sub-card designed by Aowei is used to realize the user’s MIPI CSI to AGX Samtec input. In terms of software, specific sensor driver adaptation needs to be developed based on the hardware board and module design. If there are special custom control requirements, such as controlling ISP parameters, the Camera framework needs to be modified for deep custom development.

Due to the protection of system security, the kernel compiled at the factory of Jetson has the driver signature option enabled. If the signature certificate of the newly developed driver does not match, the driver ko file cannot be loaded. However, general users cannot obtain the signature certificate when NVIDIA compiles the factory image, making it impossible to resolve the ko signature issue. If new sensor modules are needed, it is necessary to recompile the kernel, generate a complete system image, and flash the system, which is a considerable workload.

Aowei solves the above software and hardware problems for customers and creatively uses Type-C 3.0 cables as the data transmission medium, ensuring reliable connections and stable communication. Based on the NVIDIA Jetson AGX Orin developer platform, the self-developed FPGA MIPI ISP camera connection environment is shown below:

Effect Display