Video Encoding Development with Luckfox Pico Ultra W RV1106

Video encoding is the process of converting raw video data into a compressed format to reduce file size or transmission bandwidth requirements while maintaining video quality as much as possible.

The Luckfox Pico Ultra W’s main control chip is the Rockchip RV1106, a visual processor positioned for smart IPC, supporting dual MIPI CSI video signal acquisition and multiple video stream encoding capabilities. It also features an NPU with a maximum of 1 TOPs, enabling it to handle simple intelligent recognition tasks. Video encoding is the core function of the IPC processor; AI is merely an auxiliary feature. Without video encoding, AI is like a tree without roots or water without a source. This article will detail the video encoding capabilities of the RV1106 chip and ultimately develop a small program in C language to save camera footage as an MP4 video file.

Since video encoding requires video acquisition, the Luckfox Pico Ultra W has a video acquisition FPC interface that can connect to MIPI-CSI cameras and a USB port for external USB cameras. Although it is possible to connect a USB camera or HDMI-USB video capture card via the USB port, due to the limitations of USB 2.0 transfer speeds, it only supports MJPEG and other compressed formats. Therefore, if you want to acquire YUV RAW data for video encoding, the MIPI-CSI interface is the only option for this board. The video acquisition solution used in this article is the SC3336 3MP Camera (A) recommended by Luckfox, as shown in the image below:

1. ‘Core’ Capabilities of Supported Video Encoding

According to the RV1106 datasheet, the chip supports the following video hardware encoding:

The data in the manual shows that mainstream video encoding formats (such as H265, H264, and JPEG) are all supported, especially the support for H265 video encoding, which can significantly reduce the data volume of high-definition video transmission over networks.

2. Choosing a Video Encoding Development Solution

Based on the hardware resources used for encoding, there are two categories: software encoding and hardware encoding. In embedded development, software encoding typically uses FFmpeg combined with libraries like x264 and x265 to implement encoding through software algorithms. Conversely, hardware encoding utilizes the chip’s specific hardware encoding units to encode data. Compared to software encoding, hardware encoding offers higher efficiency and lower CPU resource consumption. However, it requires the chip manufacturer to provide a relevant development SDK, and developers must use the APIs provided by the SDK to access hardware resources. As such, hardware encoding has higher development requirements and is less portable to other platforms due to the tight coupling between software and hardware. The developer of the RV1106 chip is Fuzhou Rockchip Microelectronics, which has launched various chip models for different application fields. Due to historical reasons, Rockchip’s SDKs for different chips are often different, leading to some confusion. For developers new to Rockchip, seeing various versions of documentation can be overwhelming. Therefore, it is essential to clarify the media libraries available in Rockchip’s SDK before starting.

1. RKMedia was the primary media library promoted in the early days, mainly used during the RK3399 era, so there is a lot of information available online about this library on the RK3399 platform. It can still be seen in the RV1126, but it has been removed from the newly released RV1106. The author has not used this library, but the API interface looks very similar to the currently promoted Rockit library, so it is unlikely to be updated in the future and is not recommended for use.

2. MPP (Media Process Platform) is a general media processing software platform for Rockchip’s chip series, mainly focusing on video encoding and decoding. It serves as the foundation for the later Rockit library. This is currently the only open-source media library from Rockchip, which should support all platforms and mainstream chips. However, due to the specialized knowledge involved in encoding and decoding, it has a steep learning curve. Nevertheless, it is the preferred choice for capable developers due to its open-source nature and strong community support. It is recommended to use on standard operating system platforms (such as Android, Ubuntu, Buildroot, etc.).

3. Rockit is a newly launched media library with many similarities to the HiSilicon platform’s API. It appears to be a wrapper modeled after HiSilicon’s MPI based on MPP and RGA, likely to attract HiSilicon platform developers. Like HiSilicon’s MPI, it only provides header files and library files. Compared to the open-source freedom of MPP, Rockit resembles a black box, as its internal implementation is not visible. This can make debugging challenging when issues arise, especially since this library is not robust and does not provide precise error messages. Experience becomes crucial in such situations. It is recommended for use with IPC series chips.

4. RKADK is officially described as: “RKADK is a further encapsulation based on RKMedia and Rockit, providing basic common components such as recording, taking photos, playback, and preview, simplifying application development difficulty and supporting rapid application software development.” It can be seen that this adds another layer of encapsulation, simplifying application development difficulty, but at the same time losing a lot of flexibility, making it harder to troubleshoot issues. It is recommended for beginners.

5. RGA (Raster Graphic Acceleration Unit) is an independent 2D hardware accelerator used to accelerate point/line drawing and perform common 2D graphics operations such as image scaling, rotation, bitBlt, and alpha blending. This library is used to call hardware resources for image data conversion.

Based on the above discussion, the RV1106 is an IPC chip, so the Rockit media library is chosen for development.

3. Using the Rockit Media Library to Develop a Video Encoding Program

The main process for using the Rockit media library to encode camera footage in H264 format is as follows:

1. Initialize the RK MPI system

 ret = RK_MPI_SYS_Init();
    if (RK_SUCCESS != ret) {
        RK_LOGE("rk mpi sys init failed!");
        goto __FAILED;
    }

1. Set video input VI parameters and enable the VI channel

 memset(&stChnAttr, 0, sizeof stChnAttr);
    stChnAttr.stIspOpt.u32BufCount = 3;
    stChnAttr.stIspOpt.enMemoryType = VI_V4L2_MEMORY_TYPE_DMABUF;
    stChnAttr.stIspOpt.bNoUseLibV4L2 = RK_TRUE;
    stChnAttr.u32Depth = 0;
    stChnAttr.enPixelFormat = RK_FMT_YUV420SP;
    stChnAttr.stFrameRate.s32DstFrameRate = -1;
    stChnAttr.stFrameRate.s32SrcFrameRate = -1;

    stChnAttr.stSize.u32Width = 1920;
    stChnAttr.stSize.u32Height = 1080;
    stChnAttr.enCompressMode = COMPRESS_MODE_NONE;
    stChnAttr.stIspOpt.stMaxSize.u32Width = 1920;
    stChnAttr.stIspOpt.stMaxSize.u32Height = 1080;

    memcpy(stChnAttr.stIspOpt.aEntityName, "/dev/video11", strlen("/dev/video11"));
    // vi init begin
    ret = RK_MPI_VI_QueryDevStatus(viId, &stDevStatus);
    if (RK_SUCCESS == ret) {
        if(!stDevStatus.bProbeOk){
            RK_LOGE("sensor probe failed!");
            goto __FAILED;
        }
    }

    ret = RK_MPI_VI_GetDevAttr(viId, &stDevAttr);
    if (RK_ERR_VI_NOT_CONFIG == ret) {
        ret = RK_MPI_VI_SetDevAttr(viId, &stDevAttr);

        if (RK_SUCCESS != ret) {
            RK_LOGE("rk mpi vi set dev attr failed 0x%x!",ret);
            goto __FAILED;
        }
    }else{
        RK_LOGE("rk mpi vi set dev attr already!");
    }

    ret = RK_MPI_VI_GetDevIsEnable(viId);
    if (RK_SUCCESS != ret) {
        ret = RK_MPI_VI_EnableDev(viId);
        if (RK_SUCCESS != ret) {
            RK_LOGE("rk mpi vi enable dev failed 0x%x!",ret);
            goto __FAILED;
        }

        stBindPipe.u32Num = 0;
        stBindPipe.PipeId[0] = 0;
        ret = RK_MPI_VI_SetDevBindPipe(viId, &stBindPipe);
        if (ret != RK_SUCCESS) {
            RK_LOGE("rk mpi vi set dev bind pipe failed 0x%x!",ret);
            goto __FAILED;
        }
    }else{
        RK_LOGE("rk mpi vi enable  dev  already!");
    }

    ret = RK_MPI_VI_SetChnAttr(0, channelId, &stChnAttr);
    if (RK_SUCCESS != ret) {
        RK_LOGE("rk mpi vi set chn attr failed 0x%x!",ret);
        goto __FAILED;
    }

    ret = RK_MPI_VI_EnableChn(0, channelId);
    if (RK_SUCCESS != ret) {
        RK_LOGE("rk mpi vi enable chn failed 0x%x!",ret);
        goto __FAILED;
    }

1. Set encoding parameters, create the encoding channel, and start encoding

 // venc chn attr init
 VENC_RECV_PIC_PARAM_S stRecvParam;
    memset(&stEncAttr, 0, sizeof stEncAttr);
    stEncAttr.stVencAttr.enType = 8;// 8:H264, 12:H265
    stEncAttr.stVencAttr.enPixelFormat = stChnAttr.enPixelFormat;
    stEncAttr.stRcAttr.enRcMode = VENC_RC_MODE_H264CBR;
    stEncAttr.stRcAttr.stH264Cbr.u32BitRate = 3000;
    stEncAttr.stRcAttr.stH264Cbr.u32Gop = gopSize;
    stEncAttr.stVencAttr.u32Profile = 100;
    stEncAttr.stVencAttr.u32MaxPicWidth = maxWidth;
    stEncAttr.stVencAttr.u32MaxPicHeight = maxHeight;
    stEncAttr.stVencAttr.u32PicWidth = width;
    stEncAttr.stVencAttr.u32PicHeight = height;
    stEncAttr.stVencAttr.u32VirWidth = width;
    stEncAttr.stVencAttr.u32VirHeight = height;
    stEncAttr.stVencAttr.u32StreamBufCnt = 5;
    stEncAttr.stVencAttr.u32BufSize = width * height  / 2;
    stEncAttr.stGopAttr.u32MaxLtrCount = 1;
    // venc chn init
    stRecvParam.s32RecvPicNum = loopCountSet;
    printf("create venc\n");
    ret = RK_MPI_VENC_CreateChn(0, &stEncAttr);
    if (ret != RK_SUCCESS) {
    RK_LOGE("rk mpi venc create chn failed 0x%x!\n",ret);
    goto __FAILED;
    }
    ret = RK_MPI_VENC_StartRecvFrame(0, &stRecvParam);
    if (ret != RK_SUCCESS) {
    RK_LOGE("rk mpi venc start recv frame failed 0x%x!\n",ret);
    goto __FAILED;
    }

1. Bind the video input VI and encoding channel to establish the data flow

 MPP_CHN_S stSrcChn, stDestChn;
 // bind
    stSrcChn.enModId = RK_ID_VI;
    stSrcChn.s32ChnId = viId;
    stSrcChn.s32ChnId = channelId;
    stDestChn.enModId = RK_ID_VENC;
    stDestChn.s32DevId = 0;
    stDestChn.s32ChnId = 0;
    ret = RK_MPI_SYS_Bind(&stSrcChn, &stDestChn);
    if (ret != RK_SUCCESS) {
        RK_LOGE("rk mpi sys bind failed 0x%x!\n",ret);
        goto __FAILED;
    }

1. Continuously read data from the encoding channel and process the H264 data (the following code implements saving the read data from the encoding channel as an H264 file)

 stFrame.pstPack = (VENC_PACK_S *) malloc(sizeof(VENC_PACK_S));
    fp = fopen("rockit_output.h264", "w+b");
    printf("begin loop\n");
    while (run_) {
        printf("read ...\n");
        
        ret = RK_MPI_VENC_GetStream(0, &stFrame,1000);
        if (ret == RK_SUCCESS) {
            pData = RK_MPI_MB_Handle2VirAddr(stFrame.pstPack->pMbBlk);
            if (pData) {
                fwrite(pData, 1, stFrame.pstPack->u32Len, fp);
                fflush(fp);
            }
            usleep(1 * 1000);
            ret = RK_MPI_VENC_ReleaseStream(0, &stFrame);
            if (ret != RK_SUCCESS) {
                RK_LOGE("RK_MPI_VENC_ReleaseStream fail 0x%x", ret);
            }
        } else {
            RK_LOGE("RK_MPI_VI_GetChnFrame fail 0x%x", ret);
        }
    }

1. Exit the program and release hardware resources

    if(fp) fclose(fp);
    RK_MPI_SYS_UnBind(&stSrcChn, &stDestChn);
    RK_MPI_VI_DisableChn(viId,channelId );
    RK_MPI_VENC_StopRecvFrame(0);
    RK_MPI_VENC_DestroyChn(0);
    RK_MPI_VI_DisableDev(viId);
    if (stFrame.pstPack) {
        free(stFrame.pstPack);
    }
 RK_MPI_SYS_Exit();

During step 5—reading data from the encoding channel and processing the encoded data, you can call FFmpeg-related APIs to encapsulate the H264 video data into an MP4 file for saving. The above code reads data from the encoding channel using a polling method. You can obtain the device file handle corresponding to the encoding channel using RK_MPI_VENC_GetFd(chnid), and then use the return value of this interface to query the data status of the corresponding channel using select/poll/epoll. The complete source code for the project has been uploaded to my Gitee repository (please see the end of the article). The project has implemented EPOLL methods for querying encoding channel data and calling FFmpeg APIs to encapsulate H264 into MP4 files. Interested parties are welcome to provide feedback and engage in discussions.

4. Conclusion

1. While the Rockit media development library is indeed useful, it only provides library files without source code. Debugging internal issues with such libraries can be quite troublesome, especially since Rockit lacks robust error mechanisms for parameter validity checks (at least not as strong as HiSilicon’s MPI). It is crucial to zero out structure parameters before setting them during the program development process. For example, the following error can be quite frustrating.

Thread 1 "rockit_test" received signal SIGSEGV, Segmentation fault.
0xa4fe8b54 in strlen () from /lib/libc.so.0
(gdb) bt
#0  0xa4fe8b54 in strlen () from /lib/libc.so.0
#1  0xa5842734 in ?? () from /oem/usr/lib/librockit_full.so
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) q
A debugging session is active.

 Inferior 1 [process 3999] will be killed.

Image with green tint

Finally, thank you for reading this entire article. I hope this technical sharing has been helpful to you! You are also welcome to like, comment, and follow this public account. Your support will be my motivation to continue updating!

The source code for the project mentioned in this article can be found at:

https://gitee.com/lyphotoes2022/rockit_test