Click the blue "Set as Favorite" in the top left corner.
Rockchip | Camera OV13850 Porting Notes
Camera | 1. Basics of Camera
Camera | 2. Basics of MIPI, CSI
Camera | 3. Common Debugging Commands for MIPI Cameras on Rockchip Platform
Camera | 4. Writing Applications for MIPI Cameras on Rockchip Platform
Camera | 5. Linux V4L2 Architecture (Based on RK3568)
Camera | 6. V4L2 Topology Architecture (Based on RK3568)
Due to the demand for taking photos, recording videos, and live streaming, the cameras in mobile phones are basically high definition, and SoCs that support high definition cameras all support MIPI-CSI.
The implementation of MIPI-CSI varies among different SoCs, and even chips designed and produced by the same manufacturer are not entirely the same.
This article takes the Rockchip RK3568 platform EVB1 board as an example to explain the MIPI-CSI/DPHY driver in detail.
Before reading this article, it is recommended that everyone carefully study the previous articles.
11. RK3568 Hardware Module Part
Research on drivers often starts with hardware, so let’s first look at the circuit of the RK3568 board.
1) Circuit Diagram
From the circuit diagram, we can see that the camera and SoC’s MIPI-CSI interface can be x4lane or x2lane, with data and clk being differential signals.
If you don’t understand, it is recommended to ask a hardware engineer.
The camera control interface is I2C and is located on I2C channel 4.
2) Internal MIPI Related Module Diagram of RK3568
The circuit diagram only shows the interface relationship between the SoC’s MIPI controller and the camera; now let’s take a look at the internal modules related to MIPI in the RK3568.
Complaining about Rockchip’s documentation is a long story; I seriously suspect that the manufacturer does not want others to truly understand their SDK, which is good for collecting technical support fees every year. Qualcomm’s dirty trick is to make people understand it completely.
Since the official manuals either have errors or some images are incorrect, I won’t take screenshots.
The following diagram is organized based on the official manual description of the internal module diagram. 
-
Sensor output data stream is transmitted to the RK3568 DPHY controller through MIPI lanes. -
CSI controller extracts image data from hardware. -
VICAP reads data from the MIPI interface. -
Then the data is passed to the ISP, which outputs the image after a series of image processing algorithms. -
MP is used for image preview. -
SP is used for scaling.
-
VICAP Video Capture reads data from the camera through DVP/MIPI interface and transmits it to main memory via AXI bus.
VICAP features:
Supports BT601 YCbCr 422 8bit, RAW 8/10/12bit input
Supports BT656 YCbCr 422 8bit input
Supports BT1120 YCbCr 422 8bit input, single/double edge sampling
Supports 2/4 mixed BT656/BT1120 YCbCr 422 8bit input
Supports YUYV sequence configuration
Supports configurable polarity of pixel_clk, hsync, and vsync
Supports receiving CSI2 protocol data (up to 4 IDs)
Supports receiving DSI protocol data (Video mode/Command mode)
Supports window cropping
Supports virtual stride when writing to DDR
Supports NV16/NV12 format YUV data output
Supports compact/non-compact output for RAW data
Supports MMU
-
ISP (Image Signal Processing)
ISP is a complete video and still image input device. This module supports integrated YCbCr processing image sensors and simple CMOS sensors, submitting without any comprehensive image processing Bayer RGB mode images.
RK3568 uses ISP version 21.
ISP21 includes a series of image processing algorithm modules, mainly including: dark current correction, bad pixel correction, 3A, HDR, lens shading correction, lens distortion correction, 3DLUT, denoising (including RAW domain denoising, multi-frame denoising, color denoising, etc.), sharpening, etc.
ISP21 includes hardware algorithm implementation and software logic control parts, RkAiq is the implementation of the software logic control part. 
RkAiq continually obtains statistical data from ISP HW and generates new parameters through algorithms such as 3A to feedback to various hardware modules.
The main function of the RkAiq software module is to obtain image statistics from the ISP driver, combine IQ Tuning parameters, and use a series of algorithms to calculate new hardware parameters for ISP, Sensor, etc., continuously iterating this process to ultimately achieve optimal image quality.
3) CSI_RX, VICAP, ISP Register Base Address
《Rockchip RK3568 TRM Part1 V1.1-20210301.pdf》
22. Analysis of Rockchip MIPI-CSI Device Tree
The RK3568 mainly includes 4 devices:
-
isp-subdev: Image processing controller, such as 3A processing, and feeds back the processed parameters to the sensor. -
csi-subdev: MIPI data parsing controller. -
cis2-dphy: MIPI data hardware receiving controller. -
sensor: External sensor supporting MIPI output.
Now let’s see how Rockchip MIPI-CSI is described using the device tree.
1. Related MIPI Device Tree Documentation in the Kernel
Rockchip MIPI-CSI device tree node attribute descriptions refer to the kernel documentation:
[kernel\Documentation\devicetree\bindings\media]
video-interfaces.txt Description of sensor node attributes, interface types,
rockchip-isp1.txt Description of ISP module attributes
rockchip-mipi-dphy.txt Description of DPHY module
kernel\Documentation\devicetree\bindings\media\i2c\ovxxxxxx.txt Description of OV series camera device tree
2. Device Tree Node Description
All device tree nodes used by RK3568’s MIPI-CSI:
a) rockchip,rkisp-vir
rkisp_vir0: rkisp-vir0 {
compatible = "rockchip,rkisp-vir";
rockchip,hw = <&rkisp>;
status = "disabled";
};
This device tree information corresponds to the initialization function
[kernel\drivers\media\platform\rockchip\isp\dev.c]
struct platform_driver rkisp_plat_drv = {
.driver = {
.name = DRIVER_NAME,
.of_match_table = of_match_ptr(rkisp_plat_of_match),
.pm = &rkisp_plat_pm_ops,
},
.probe = rkisp_plat_probe,
.remove = rkisp_plat_remove,
};
This node is used to initialize ISP-related components,
The driver will create the components in the topology diagram: rkisp-isp-subdev, rkisp-csi-subdev, rkisp_mainpath, rkisp_selfpath, rkisp_rawwr0, rkisp_rawwr2, rkisp_rawwr3, rkisp_rawrd0_m, rkisp_rawrd2_s, rkisp-statistics, rkisp-input-params
ISP hardware-related information is described in the parent node **rockchip,hw = <&rkisp>;**.
b) rkisp
rkisp: rkisp@fdff0000 {
compatible = "rockchip,rk3568-rkisp";
reg = <0x0 0xfdff0000 0x0 0x10000>;
interrupts = <GIC_SPI 57 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 58 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 60 IRQ_TYPE_LEVEL_HIGH>; //Interrupt GPIO used, triggered by high level
interrupt-names = "mipi_irq", "mi_irq", "isp_irq"; //Interrupt names
clocks = <&cru ACLK_ISP>, <&cru HCLK_ISP>, <&cru CLK_ISP>; //Clocks
clock-names = "aclk_isp", "hclk_isp", "clk_isp"; //Clock names
resets = <&cru SRST_ISP>, <&cru SRST_H_ISP>;
reset-names = "isp", "isp-h";
rockchip,grf = <&grf>;
power-domains = <&power RK3568_PD_VI>; //ISP VICAP power and clock
iommus = <&rkisp_mmu>; //MMU attributes
rockchip,iq-feature = /bits/ 64 <0x3FBFFFE67FF>;
status = "disabled";
};
rkisp_mmu: iommu@fdff1a00 {
compatible = "rockchip,iommu-v2";
reg = <0x0 0xfdff1a00 0x0 0x100>;
interrupts = <GIC_SPI 59 IRQ_TYPE_LEVEL_HIGH>;
interrupt-names = "isp_mmu";
clocks = <&cru ACLK_ISP>, <&cru HCLK_ISP>;
clock-names = "aclk", "iface";
power-domains = <&power RK3568_PD_VI>;
#iommu-cells = <0>;
rockchip,disable-mmu-reset;
status = "disabled";
};
pmu: power-management@fdd90000 {
pd_vi@RK3568_PD_VI {
reg = <RK3568_PD_VI>;
clocks = <&cru HCLK_VI>,
<&cru PCLK_VI>;
pm_qos = <&qos_isp>,
<&qos_vicap0>,
<&qos_vicap1>;
};
};
This device tree node is used to describe ISP hardware information: Base address 0xfdff0000, interrupt source, clock, reset pin, iommus, etc.
The driver extracts corresponding hardware information and fills it into the struct rkisp_hw_dev structure variable.
Corresponding driver entry:
[kernel\drivers\media\platform\rockchip\isp\hw.c]
static struct platform_driver rkisp_hw_drv = {
.driver = {
.name = "rkisp_hw",
.of_match_table = of_match_ptr(rkisp_hw_of_match),
.pm = &rkisp_hw_pm_ops,
},
.probe = rkisp_hw_probe,
.remove = rkisp_hw_remove,
.shutdown = rkisp_hw_shutdown,
};
c) CSI2 Protocol Related Device Tree
-
csi2_dphy0 Topology Structure Related Information -
csi2_dphy_hw CSI2 Driver Related Hardware Information
The following describes the topology information of csi2_dphy0, actual camera information needs to be filled in by the user:
[rk3568-evb1-ddr4-v10.dtsi]
<csi2_dphy0 {
status = "okay";
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
#address-cells = <1>;
#size-cells = <0>;
mipi_in_ucam0: endpoint@1 {
reg = <1>;
remote-endpoint = <&0v13850_out>;
data-lanes = <1 2 3 4>;
};
};
port@1 {
reg = <1>;
#address-cells = <1>;
#size-cells = <0>;
csidphy_out: endpoint@0 {
reg = <0>;
remote-endpoint = <&isp0_in>;
};
};
};
};
This node describes the content:
- Parent node csi2_dphy0
- port@n : Indicates the pad number is n
- mipi_in_ucam0 : Sink Pad (in indicates entering this entity, the upstream connected device is given by remote-endpoint, i.e., camera 0v13850_out)
- data-lanes : Number of MIPI channels: 4
- csidphy_out : Source Pad, the downstream connected device is given by remote-endpoint, i.e., isp0_in
The following is the hardware information related to the csi2_dphy controller, located in the Rockchip 3568 platform device tree file rk3568.dtsi
[rk3568.dtsi]
aliases {
csi2dphy0 = &csi2_dphy0;
……
}
csi2_dphy0: csi2-dphy0 {
compatible = "rockchip,rk3568-csi2-dphy";
rockchip,hw = <&csi2_dphy_hw>;
status = "disabled";
};
csi2_dphy_hw: csi2-dphy-hw@fe870000 {
compatible = "rockchip,rk3568-csi2-dphy-hw";
reg = <0x0 0xfe870000 0x0 0x1000>;
clocks = <&cru PCLK_MIPICSIPHY>;
clock-names = "pclk";
rockchip,grf = <&grf>;
status = "disabled";
};
csi2dphy0 corresponds to the driver entry:
[kernel\drivers\phy\rockchip\phy-rockchip-csi2-dphy-hw.c]
static struct platform_driver rockchip_csi2_dphy_hw_driver = {
.probe = rockchip_csi2_dphy_hw_probe,
.remove = rockchip_csi2_dphy_hw_remove,
.driver = {
.name = "rockchip-csi2-dphy-hw",
.of_match_table = rockchip_csi2_dphy_hw_match_id,
},
};
In the function rockchip_csi2_dphy_hw_probe(), the structure variable rockchip_csi2_dphy_driver will also be registered.
630 platform_driver_register(&rockchip_csi2_dphy_driver);
rockchip_csi2_dphy_driver is defined as follows:
[kernel\drivers\phy\rockchip\phy-rockchip-csi2-dphy.c]
struct platform_driver rockchip_csi2_dphy_driver = {
.probe = rockchip_csi2_dphy_probe,
.remove = rockchip_csi2_dphy_remove,
.driver = {
.name = "rockchip-csi2-dphy",
.pm = &rockchip_csi2_dphy_pm_ops,
.of_match_table = rockchip_csi2_dphy_match_id,
},
};
Analyzing the driver starts from these entry functions probe.
33. Driver Initialization
1. Driver File Directory
kernel
├── arch/arm64/boot/dts/rockchip DTS configuration files
├── drivers/phy/rockchip/
├── phy-rockchip-csi2-dphy.c
└── phy-rockchip-csi2-dphy-hw.c MIPI DPHY driver
├── drivers/media|
├── platform/rockchip/isp rkisp ISP driver
│ ├── capture_v21.c Includes configurations of MP/SP and vb2, frame interrupt processing
│ ├── dev.c Includes probe, asynchronous registration, clock, pipeline, iommu and
media/v4l2 framework
│ ├── isp_params_v21.c 3A related parameter settings
│ ├── isp_stats_v21.c 3A related statistics
│ ├── regs.c Register related read and write operations
│ └── rkisp.c Corresponds to isp_sd entity node,
│ Includes receiving data from MIPI and has crop functionality
├── v4l2-core V4L2 core code
└── i2c/
└── ov13850.c CIS (CMOS Image Sensor) driver
Note: The ISP version of 3568 is v21, only files ending with v21 need to be viewed.
1. Character Device Number Application: videodev_init()
This function is mainly used to apply for device numbers:
Major device number: 81
Device name: video4linux
Applied class: video4linux
#define VIDEO_MAJOR 81
#define VIDEO_NUM_DEVICES 256
#define VIDEO_NAME "video4linux"
static struct class video_class = {
.name = VIDEO_NAME,
.dev_groups = video_device_groups,
};
static int __init videodev_init(void)
{
dev_t dev = MKDEV(VIDEO_MAJOR, 0);
ret = register_chrdev_region(dev, VIDEO_NUM_DEVICES, VIDEO_NAME);
ret = class_register(&video_class);
}
static void __exit videodev_exit(void)
{
dev_t dev = MKDEV(VIDEO_MAJOR, 0);
class_unregister(&video_class);
unregister_chrdev_region(dev, VIDEO_NUM_DEVICES);
}
Note: For simplification, only the most important parts of all code are listed, and the rest will follow.
2. ISP Architecture Initialization: rkisp_plat_probe()
This function is the most important initialization function. Apart from rkisp_csi2_dphy(entity67), other functional components are initialized in this function.
Register rkisp-vir0 parent device, isp-dubdev child device, csi2-dev child device, etc. Since RK3568 supports multiple sensor inputs, that is, ISP supports multi-channel processing, multiple virtual channels isp-virx will be created.
The main work of this function:
-
Apply memory for isp_dev and initialize it; this structure is used for all information of the camera controller. -
Register the v4l2_device structure. -
Initialize media-related resources. -
The function rkisp_register_platform_subdevs() is used to register various modules in the topology structure; the corresponding entity is detailed in the diagram. -
Entity 1, 7 are registered as sub_device, initializing struct v4l2_subdev_ops, and initializing media sub-modules require relevant information; entity 1 will also set the default image format. -
Entities 13, 19, 25, 31, 37, 43, 49, 55, 61 are registered as video_device, filling struct v4l2_file_operations (struct video_device->fops), struct v4l2_ioctl_ops (struct video_device->ioctl_ops), struct vb2_ops (struct video_device->vb2_queue->ops).
These callback functions are related in the architecture as shown in the following diagram: 
where the resources related to entity 67 are registered in the function rockchip_csi2_dphy_probe().
3. ISP Driver Initialization: rkisp_hw_probe()
This function mainly initializes the ISP driver.