Key Configuration for RK3588 MIPI/DVP Camera Debugging

Introduction:

The previous article introduced how to light up a sensor on RK3588, using the MIPI camera IMX577 as an example. From previous articles, we know that RK3588 supports up to 6 MIPI cameras and 1 DVP camera. Therefore, this article serves as a supplement, introducing the key configurations for both MIPI and DVP cameras.

1. Common Camera Interfaces:

Cameras can be categorized based on different interfaces, commonly including: USB camera, MIPI camera, and DVP camera.

① USB Camera:

Common data formats for USB Cameras include: YUYV/MJPEG/H.264
YUYV: Transmits YUV data, which is large in volume with lower resolution and frame rate, generally reprocessed by the backend for special purposes (iris recognition, infrared facial recognition, etc.). MJPEG: Generally used for relatively lower resolution USB Cameras (720P, VGA), with larger data transmission. H.264: Generally used for higher resolution USB Cameras (1080p, 720p), with smaller data transmission.

② MIPI Camera:

MIPI is a low-voltage differential signaling (LVDS) interface that offers fast transmission speeds and strong anti-interference capabilities, generally supporting cameras with 8 million pixels and above. Currently, mainstream mobile phone camera modules use MIPI for transmission, utilizing 4 pairs of differential data signals and 1 pair of differential clock signals during transmission.

③ DVP Camera:

DVP Camera, also known as Parallel Camera, has the interface shown in the figure below and generally supports the transmission of BT601/BT656/BT1120 data. DVP is a parallel transmission method with relatively slower speeds and lower bandwidth, typically used for cameras with less than 5 million pixels. It requires PCLK clock, VSYNC field synchronization, HSYNC line synchronization, and D[0:11] parallel data, which can be 8/10/12bit/16bit data sizes.

Key Configuration for RK3588 MIPI/DVP Camera Debugging

2. RK3588 Debugging:

① Overview:

The connection block diagram is shown below, which briefly introduces the key points of RK3588 camera configuration.

According to the above structure diagram, we can roughly see how MIPI and DVP are connected, with the key points as follows:

RK3588 supports two dcphy nodes, named csi2_dcphy0/csi2_dcphy1. Each dcphy hardware supports RX/TX simultaneously, with RX used for camera input. It supports DPHY/CPHY protocol multiplexing; note that the same dcphy’s TX/RX can only use either DPHY or CPHY at the same time.
RK3588 supports 2 dphy hardware, referred to as dphy0_hw/dphy1_hw, both of which can operate in full mode and split mode. 1. dphy0_hw and dphy1_hw

Full mode: Node names use csi2_dphy0 and csi2_dphy3, supporting up to 4 lanes.
Split mode: Split into 2 phy for use, namely csi2_dphy1 (using 0/1 lane), csi2_dphy2 (using 2/3 lane), dphy1_hw is split into csi2_dphy4 and csi2_dphy5, with each phy supporting a maximum of 2 lanes.
When dphy0_hw is used in full mode, the link needs to be configured according to the csi2_dphy1 link, but the node name csi2_dphy1 needs to be changed to csi2_dphy0; the software distinguishes the phy usage mode through the phy’s serial number. The same applies to dphy1_hw.

Using the above MIPI phy nodes requires configuring the corresponding physical nodes: csi2_dcphy0_hw/csi2_dcphy1_hw/csi2_dphy0_hw/csi2_dphy1_hw.

Each MIPI phy requires a csi2 module to parse the MIPI protocol, with node names mipi0_csi2~mipi5_csi2.

All camera data on RK3588 must go through vicap and then connect to isp. RK3588 supports only one vicap hardware, which can simultaneously input 6 MIPI phy and 1 DVP data, so we differentiate vicap into rkcif_mipi_lvds~rkcif_mipi_lvds5, rkcif_dvp, etc., with seven nodes; the binding relationship of each node must be strictly configured according to the block diagram’s node sequence number.

Each vicap node’s connection to isp is indicated through the corresponding virtual XXX_sditf to specify the connection relationship.

RK3588 supports 2 isp hardware, and each isp device can virtualize multiple virtual nodes; the software reads image data from ddr for processing in isp sequentially. For multiple camera schemes, it is recommended to evenly distribute the data flow across two isps.

Bypass and Readback Mode:

Bypass: Refers to data collected by vicap, sent directly to isp for processing without being stored in ddr. Note that in HDR bypass mode, only the short frame is truly bypassed, while the long frame must exist in ddr for isp to read from ddr.
Readback: Refers to data collected by vicap into ddr; after the application retrieves the data, the buffer address is pushed to isp, which then obtains image data from ddr.
In dts configuration, if an isp hardware is configured with only one virtual node, it defaults to bypass mode; if multiple virtual nodes are configured, it defaults to readback mode.

② RK3588 MIPI Configuration:

RK3588 can use up to 4 2-lane and 2 4-lane MIPI interface cameras; key configuration points are as follows:

Driver interface configuration:

Pay attention to the configuration of the g_mbus_config interface, which needs to be configured to use either DVP interface or MIPI interface, and specify whether it is MIPI DPHY or MIPI CPHY; for example, IMX577 is configured as V4L2_MBUS_CSI2_DPHY. Next, if it is a MIPI interface, the lane count must also be configured; IMX577 is configured as 4 lanes. Finally, the number of virtual channels must also be configured; for IMX577, a single channel is configured in linear mode, while HDR mode requires two channels if multiple vc is used.

static int imx577_g_mbus_config(struct v4l2_subdev *sd, unsigned int pad,
    struct v4l2_mbus_config *config)
{
 struct imx577 *imx577 = to_imx577(sd);
 const struct imx577_mode *mode = imx577->cur_mode;
 u32 lane_num = imx577->bus_cfg.bus.mipi_csi2.num_data_lanes;
 u32 val = 0;
 
 val = 1 << (lane_num - 1) |
  V4L2_MBUS_CSI2_CHANNEL_0 |
  V4L2_MBUS_CSI2_CONTINUOUS_CLOCK;
 if (mode->hdr_mode != NO_HDR)
  val |= V4L2_MBUS_CSI2_CHANNEL_1;
 
 config->type = V4L2_MBUS_CSI2_DPHY;
 config->flags = val;
 
 return 0;
}

dts configuration

Refer to the previous article’s IMX577 configuration; here we emphasize a few key points:

data-lanes: Configure the lane count; if it is 4 lanes, it is <1 2 3 4>, for 2 lanes it is <1 2>;

camera-module-name and camera-module-lens-name: Used when the RAW sensor runs AIQ;

RAW sensor configuration: sensor->csi2_dphy->mipi_csi->rkcif_mipi_lvds … rkcif_mipi_lvds_stdif->rkisp_vir

YUV/RGB888 sensor configuration: sensor->csi2_dphy->mipi_csi->rkcif_mipi_lvds

YUV or RGB888 sensors typically come with an ISP, directly outputting YUV422 images without needing to go through the RK3588 ISP, thus storing directly at the vicap level into ddr.

③ RK3588 DVP Camera:

RK3588 has a DVP interface that supports BT601/BT656/BT1120, similarly, if it is a RAW sensor, it needs to be configured to ISP; if it is YUV, it does not need to go through ISP, with key configurations as follows:

BT601 interface:

Key points for BT601 interface configuration:

hsync-active/vsync-active must be configured for the v4l2 framework to recognize the BT601 interface asynchronously; otherwise, it will be recognized as a BT656 interface;

pclk-sample/bus-width are optional;

It is essential to indicate the effective polarity of the current sensor’s hsync-active/vsync-active/pclk-active in the g_mbus_config interface of the sensor driver through flags; otherwise, it may lead to data not being received;

The pinctrl needs to reference the appropriate GPIO for BT601-related GPIO to perform the necessary IOMUX; otherwise, it may result in data not being received;

Driver interface configuration is as follows:

static int gc2145_g_mbus_config(struct v4l2_subdev *sd, unsigned int pad_id,
    struct v4l2_mbus_config *config)
{
 struct gc2145 *gc2145 = to_gc2145(sd);
 
  config->type = V4L2_MBUS_PARALLEL;
  config->flags = V4L2_MBUS_HSYNC_ACTIVE_HIGH |
    V4L2_MBUS_VSYNC_ACTIVE_LOW |
    V4L2_MBUS_PCLK_SAMPLE_RISING;
 
 return 0;
}

The DTS configuration for BT656 and BT1120 is basically the same, so it will not be repeated; refer to the following BT656/BT1120 for details;

BT656/BT1120:

Key points for configuration are as follows:

hsync-active/vsync-active should not be configured; otherwise, the v4l2 framework will recognize it as BT601 during asynchronous registration;

pclk-sample/bus-width are optional;

It is crucial to indicate the effective polarity of the current sensor’s pclk-active in the g_mbus_config interface through flag variables; otherwise, it may lead to data not being received;

It is necessary to implement the querystd interface in the v4l2_subdev_video_ops to indicate that the current interface is an ATSC interface; otherwise, it may lead to data not being received;

It is necessary to implement RKMODULE_GET_BT656_MBUS_INFO, as both BT656 and BT1120 call this ioctl, with interface compatibility. Implementation reference can be found in drivers/media/i2c/nvp6158_drv/nvp6158_v4l2.c;

The pinctrl needs to reference the appropriate GPIO for BT656/BT1120-related GPIO to perform the necessary IOMUX; otherwise, it may result in data not being received.

Example code for the g_mbus_config interface is as follows:

static int lt8619c_g_mbus_config(struct v4l2_subdev *sd, unsigned int pad,
        struct v4l2_mbus_config *cfg)
{
 struct lt8619c *lt8619c = to_lt8619c(sd);
 
 cfg->type = V4L2_MBUS_BT656;
 if (lt8619c->clk_ddrmode_en) {
  cfg->flags = RKMODULE_CAMERA_BT656_CHANNELS |
   V4L2_MBUS_PCLK_SAMPLE_RISING |
   V4L2_MBUS_PCLK_SAMPLE_FALLING;
 } else {
  cfg->flags = RKMODULE_CAMERA_BT656_CHANNELS |
   V4L2_MBUS_PCLK_SAMPLE_RISING;
 }
 
 return 0;
}

Example code for the querystd interface is as follows:

static int lt8619c_querystd(struct v4l2_subdev *sd, v4l2_std_id *std)
{
 struct lt8619c *lt8619c = to_lt8619c(sd);
 
 if (lt8619c->yuv_output_mode == BT656_OUTPUT)
  *std = V4L2_STD_PAL;
 else
  *std = V4L2_STD_ATSC;
 
 return 0;
}

DTS configuration reference is as follows:

&amp;i2c2 {
 status = "okay";
 pinctrl-names = "default";
 pinctrl-0 = &lt;&amp;i2c2m4_xfer&gt;;
 
 nvp6158: nvp6158@30 {
  compatible = "nvp6158-v4l2";
  status = "okay";
  reg = &lt;0x30&gt;;
  clocks = &lt;&amp;cru CLK_CIFOUT_OUT&gt;;
  clock-names = "xvclk";
  power-domains = &lt;&amp;power RK3588_PD_VI&gt;;
  pinctrl-names = "default";
  pinctrl-0 = &lt;&amp;cif_clk &amp;cif_dvp_clk &amp;cif_dvp_bus8 &amp;cif_dvp_bus16&gt;;
  // pwr-gpios = &lt;&amp;gpio1 RK_PA6 GPIO_ACTIVE_HIGH&gt;;
  pwr2-gpios = &lt;&amp;gpio1 RK_PA5 GPIO_ACTIVE_HIGH&gt;;
  rst-gpios = &lt;&amp;gpio1 RK_PA3 GPIO_ACTIVE_HIGH&gt;;
  // rst2-gpios = &lt;&amp;gpio3 RK_PC1 GPIO_ACTIVE_HIGH&gt;;
  // pwdn-gpios = &lt;&amp;gpio1 RK_PA4 GPIO_ACTIVE_HIGH&gt;;
  // pwdn2-gpios = &lt;&amp;gpio4 RK_PA6 GPIO_ACTIVE_HIGH&gt;;
  rockchip,camera-module-index = &lt;0&gt;;
  rockchip,camera-module-facing = "back";
  rockchip,camera-module-name = "default";
  rockchip,camera-module-lens-name = "default";
  rockchip,dvp_mode = "BT1120"; //BT656 or BT1120 or BT656_TEST
  rockchip,channel_nums = &lt;4&gt;; //channel nums, 1/2/4
  rockchip,dual_edge = &lt;1&gt;; // pclk dual edge, 0/1
  rockchip,default_rect= &lt;1920 1080&gt;; // default resolution
  port {
   nvp6158_out: endpoint {
    remote-endpoint = &lt;&amp;dvp_in_bcam1&gt;;
   };
  };
 };
};
 
&amp;rkcif {
 status = "okay";
};
 
&amp;rkcif_dvp {
 status = "okay";
 ports {
  #address-cells = &lt;1&gt;;
  #size-cells = &lt;0&gt;;
 
  port@0 {
   #address-cells = &lt;1&gt;;
   #size-cells = &lt;0&gt;;
   /* Parallel bus endpoint */
   dvp_in_bcam1: endpoint@1 {
    reg = &lt;1&gt;;
    remote-endpoint = &lt;&amp;nvp6158_out&gt;;
    bus-width = &lt;16&gt;;
   };
  };
 };
};
 
&amp;rkcif_mmu {
 status = "okay";
};

Similarly, in the dts configuration, if it is a YUV sensor, only the above configurations are needed, connected to the rkcif_dvp node; if it is a RAW sensor, it is necessary to add the rkcif_dvp_sditf node and connect it to the rkisp_vir node.

Reference: http://t.csdnimg.cn/r7kNF