Due to factors such as network structure and training methods, most neural network models have format restrictions on input data. In the field of computer vision, these restrictions are mostly reflected in image size, color space, normalization parameters, etc. If the source image or video does not meet the requirements of the network model in terms of size or format, we need to preprocess the data.
AscendAI hardware is equipped with a dedicated hardware unit for image preprocessing, allowing developers to leverage its powerful media processing acceleration capabilities through the upper software stackCANN. The Orange Pi AIpro development board is a high-performance AI development board jointly created by Orange Pi and Ascend, enabling developers to preprocess data that does not meet the input requirements of neural network models, thereby better completingAI inference calculations.
02
Ascend CANN Data Preprocessing Methods
CANN provides two dedicated methods for data preprocessing: AIPP and DVPP.
AIPP: Artificial Intelligence Pre-ProcessingAIPP completes data preprocessing on AI Core, with main functions including changing image size (cropping, padding, etc.), color space conversion (changing image format), mean subtraction/scaling (changing image pixels), etc.AIPP is divided into static AIPP and dynamic AIPP; you can only choose one of them, and simultaneous configuration of both is not supported.– Static AIPP: Set the AIPP mode to static during model conversion and configure AIPP parameters. After the model is generated, the AIPP parameter values are saved in the offline model (*.om), and the fixed AIPP preprocessing parameters are used in each model inference process, which cannot be modified.– Dynamic AIPP: Set the AIPP mode to dynamic during model conversion and configure dynamic AIPP parameter values before each model inference according to requirements, allowing different AIPP parameters to be used during model execution.
DVPP: Digital Vision Pre-Processing
DVPP is an image processing unit built into Ascend AI processors, providing powerful hardware acceleration capabilities for media processing through the AscendCL media data processing interface, with main functions including scaling, cropping, color space conversion, image encoding/decoding, and video encoding/decoding.
In summary, although both are data preprocessing methods, AIPP and DVPP have different functional scopes (for example, DVPP can perform image and video encoding/decoding, while AIPP can perform normalization configuration), and the computational units for processing data are also different. AIPP uses the AI Core computational acceleration unit, while DVPP uses a dedicated image processing unit.
AIPP and DVPP can be used independently or in combination. In combined usage scenarios, DVPP is generally used first for basic processing such as decoding, cropping, and scaling images/videos, followed by AIPP for color space conversion and normalization.
03How to Use AIPP Functionality
Taking this as an example: The test image resolution is 250*250, image format is YUV420SP, and the model requires the image to be 224*224 resolution and RGB format. Therefore, we need to use AIPP to achieve cropping and image format conversion. Various format conversion color space conversion coefficients have templates available from the “ATC Tool User Guide”, which can be accessed at the end of this document in the “Ascend Documentation Center”.
Static AIPP
1. Construct the AIPP configuration file *.cfg.Crop: The effective data area starts from the top left corner (0, 0) pixel, with crop width*height of 224*224.Image format conversion: Input image format is YUV420SP_U8, and output image format is controlled by color space conversion coefficients.
2. Enable Static AIPP.When using the ATC tool to convert the model, you can pass the AIPP configuration file through the insert_op_conf parameter, saving its configuration parameters in the model file.
The parameter explanations are as follows:– framework: The original network model framework type, 3 represents the TensorFlow framework.– soc_version: Specify the version of the Ascend AI processor during model conversion. You can execute the npu-smi info command to query it, and add Ascend information before the queried “Name”, for example, if the “Name” corresponds to the value xxxyy.– model: The file path of the original network model, including the file name.– insert_op_conf: AIPP preprocessing configuration file path, including the file name.– output: The file path of the converted *.om model file, including the file name; after successful conversion, the file name automatically ends with the .om suffix.
3. Call the AscendCL interface to load the model and perform inference; refer to how to develop AI inference applications based on Orange Pi AIpro.
Dynamic AIPP
1. Construct the AIPP configuration file *.cfg.
2. Enable Dynamic AIPP.When using the ATC tool to convert the model, you can pass the AIPP configuration file through the insert_op_conf parameter, saving its configuration parameters in the model file.
The parameter explanations are as follows:
– framework: The original network model framework type, 3 represents the TensorFlow framework.– soc_version: Specify the version of the Ascend AI processor during model conversion.– model: The file path of the original network model, including the file name.– insert_op_conf: AIPP preprocessing configuration file path, including the file name.– output: The file path of the converted *.om model file, including the file name; after successful conversion, the file name automatically ends with the .om suffix.3. Call the AscendCL interface to load the model, set the AIPP parameters, and then perform inference; refer to how to develop AI inference applications based on Orange Pi AIpro. The code example for calling the AscendCL interface to set AIPP parameters is as follows:04How to Use DVPP FunctionalityThe Ascend AI processor has a built-in image processing unit DVPP, providing powerful hardware acceleration capabilities for media processing. Meanwhile, the heterogeneous computing architecture CANN provides an entry point for utilizing the computational power of image processing hardware: the AscendCL interface, allowing developers to perform image processing through this interface to leverage the computational power of the Ascend AI processor.
The functional modules within DVPP are as follows.
– VPC（Vision Preprocessing Core）Processes images in formats such as YUV and RGB, including scaling, cropping, color space conversion, histogram statistics, etc.– JPEGD（JPEG Decoder）Decodes JPEG compressed format images to YUV format.– JPEGE（JPEG Encoder） Encodes YUV format images to JPEG compressed format.– VDEC（Video Decoder） Decodes H264/H265 format video streams to YUV/RGB format.– VENC（Video Encoder） Encodes YUV420SP format video streams to H264/H265 format.– PNGD（PNG decoder） Decodes PNG format images to RGB format.
Here we take JPEGD image decoding + VPC image scaling as an example to illustrate how to use DVPP functionality. First, we overview the interface calling process, including resource initialization & de-initialization, channel creation and destruction, decoding, scaling, waiting for task completion, and releasing memory resources.
After overviewing the interface calling process, we will now showcase the key code logic for JPEGD image decoding + VPC image scaling functionality in a way that developers are more familiar with.
05More Learning ResourcesAscend Community Documentation CenterOne-stop Learning Resources for Orange Pi AIpro Development Board
Previous Recommendations
1、Beginner’s Guide | Quick Start with Orange Pi AIpro Development Board
2、Comprehensive Sample Collection of Orange Pi AIpro Peripheral Interfaces (with Source Code)
3、How to Develop AI Inference Applications Based on Orange Pi AIpro
4、How to Upgrade CANN Software Package on Orange Pi AIpro Development Board
5、How to Convert Open Source Framework Models to Ascend Models Based on Orange Pi AIpro
