AI Systems: From NPU Scheduler to AI Inference Engine

In the article AI System – 7 Pytorch Digital Recognition Practice and Operator Introduction, it mentions AI algorithms implemented using Pytorch on PC, and in AI System – 17 NPU Architecture Design Introduction, it discusses hardware implementation in NPU. Now, there is a question:

How can AI algorithms be deployed and adapted on hardware?

This article will unveil this confusion. The answer is that first, the NPU scheduler splits the algorithm into tasks for parallel computation and scheduling, and secondly, the tasks generated by the AI compiler are executed in parallel on the NPU hardware.

1. Introduction to NPU Scheduler

Here, we take Huawei’s Ascend as an example, reference: zhuanlan.zhihu.com/p/707141458…

We can see that the chip integrates a hardware task scheduler module. What is this module mainly for? Let’s first explain from the AI APP perspective:

The APP will be converted into different expression forms at different levels of programming models, namely Stream, Task, and Block. All these expressions are executed in parallel in different levels of schedulers, detailed as follows:

As shown in the figure, DNN development frameworks such as Pytorch, Tensorflow, and MindSpore are at the top of the entire development software stack.

• Application Layer: Multiple applications can be executed in parallel on the same Ascend SoC.
• Stream/Task Layer: The graph compiler will compile each application into multiple Streams, each containing multiple Tasks. Tasks from multiple Streams can be executed in parallel under the scheduling of the TS (Task Scheduler), while Tasks within the same Stream are executed serially.
• Block Layer: Each Task can be divided into several Blocks (which need to be explicitly described by the developer or compiler), and different Blocks can be dispatched to different Cores for execution.

Ascend provides a scheduling solution that combines software and hardware. The Ascend development software stack supports native differentiable programming paradigms and native execution modes, striking a balance between development efficiency and computational efficiency. MindSpore is a dedicated framework for Ascend core, which can fully utilize Ascend’s computing resources to achieve high computational efficiency.

1. On one hand, developers can easily deploy DNN models to various Ascend devices, which can also be equipped with different Ascend cores. Developers can use high-level programming models to achieve one-time development and on-demand deployment.

2. On the other hand, experts with more architectural knowledge can use low-level programming models and libraries to further improve computational efficiency.

The output of the DNN development framework is called a Graph, which represents the coarse-grained dependencies in the algorithm. Then, as shown in the figure above, the Graph Engine will split the “Graph”, dividing operators suitable for CPU logical processing into CPU subgraphs and computationally intensive operators into dedicated accelerator subgraphs, thus converting them into Streams. Among them, a Stream consists of several ordered Tasks.

2. Scheduler Design

The above figure shows that the A core in the AI chip is responsible for task distribution, while the Accelerators are responsible for NPU task execution, which has a large number of heterogeneous accelerators in its hardware and software system. Since accelerators may need to use data produced by the CPU, the design of memory coherence and consistency is quite challenging. Moreover, the operating system stack, compiler, and scheduler all need to be redesigned, especially for parallel computing, so additional execution parallelism needs to be added between the A core and NPU: the scheduler.

2.1 Scheduling Model

Academia and industry have been searching for different levels of parallel programming models to improve computational performance. Common parallel programming models include ILP (Instruction Level Parallelism), TLP (Thread Level Parallelism), and DLP (Data Level Parallelism), as shown in Figure 2. These three parallel models correspond to the development of computer architecture. Out-of-order scheduling in processors has greatly increased ILP capabilities; multi-core designs have made TLP particularly important; and the increase in vector computing units, such as GPGPU, has effectively utilized DLP.

NPU chips have a large number of dedicated heterogeneous accelerators. ILP, TLP, and DLP cannot fully exploit the computational potential of heterogeneous accelerators, so it is urgent to choose a new programming model as the interface between AI algorithms and chips. The use of heterogeneous accelerators naturally paves the way for tasks as computational units, and TLP (Task Level Parallelism) is logically introduced. Here, a task is an abstract level that defines a set of instructions as primitives that can be reused in the program. A user’s thread can consist of multiple tasks, and typically a task is executed by one type of accelerator in the chip.

As shown in the figure above, multiple tasks can form a DAG based on different algorithms and application scenarios to achieve fine-grained parallelism, with edges between tasks representing dependencies.

2.2 Software Scheduler

Typically, a task-based parallel computing environment consists of two components:

(1) Task parallel API

(2) Task runtime system.

The former defines how developers describe parallelism, dependencies, data distribution, etc., while the latter defines the efficiency and capabilities of the environment. The runtime determines the supported architecture, scheduling goals, scheduling strategies, and exception handling, etc. However, this software-based scheduler has inherent drawbacks:

1. Running the scheduler instance on the CPU incurs overhead, and when task granularity is small and the number of accelerators is large, software scheduling consumes a lot of CPU resources, significantly reducing end-to-end computational efficiency;
2. When tasks finish execution on the accelerators, they must notify the CPU through interrupts with long latency, further increasing scheduling latency;
3. The software overhead and latency of scheduling lead to the inability to compile fine-grained tasks, resulting in the inability to utilize fine-grained parallelism;

2.3 Hardware Scheduler

If the NPU system includes a SoC-level hardware task scheduler, it can greatly alleviate the above drawbacks, allowing the compiler to compile fine-grained tasks to fully utilize fine-grained parallelism. As shown in the figure, adding HTS (Hardware Task Scheduler) to a system containing CPUs and various accelerators.

The CPU can push tasks and their dependencies to the HTS, the CPU only needs to poll the task completion queue and handle exceptions. The HTS will maintain several task queues, similar to the instructions executed in an OoO CPU. The HTS will maintain the busy and idle states of each accelerator in the system and send tasks to idle accelerators. When a task finishes execution, the accelerator writes a task completion descriptor to notify the HTS, which is several orders of magnitude faster than interrupt handling. The HTS can handle task dependencies, and when a dependency for a task is released, the HTS will schedule that task in an unordered manner, which can bring significant speed improvements in heterogeneous ADS systems.

The hardware scheduler TS is essentially a CPU core running firmware on it,

1. The CPU can be ARM or a high-performance RISCV, multi-core is of course better, scheduling tasks in the stream, directly binding a stream to a fixed core maximizes performance, which is quite luxurious.
2. The firmware is basically RTOS sufficient, such as FreeRTOS, and should not be too complex.

Additionally, the hardware submodule needs to have its own peripheral devices:

1. Inter-core communication mailbox hardware
2. SRAM storage usage
3. WDT watchdog
4. UART serial port
5. CRU clock reset power hardware
6. DMA support to improve data exchange speed
7. NoC and other buses for communication with NPU
8. Other communication hardware

For the startup of the NPU scheduler corresponding to the SoC chip, the firmware is generally stored in the flash fip package, which can be moved to the SRAM or TCM of the NPU scheduler core by ATF BL2, and then triggered for execution.

As the big brother in the SoC chip, the A core can also control the NPU scheduler, for example, when the system goes to sleep or energy-saving mode, it needs to stop AI services, then the A core will have a driver to manage the lifecycle of the NPU scheduler.

3. AI Inference Engine

Reference: zhuanlan.zhihu.com/p/687336210… In the figure, it can be seen that there is an AI Runtime above the scheduler, which is the runtime executor of the inference engine. On the surface, it is an organized Task graph generated to complete an AI task. In actual operation, it runs in the A core, and if those Tasks need to be executed, they are sent to the NPU scheduler for further scheduling.

These Tasks are stored in pattern files in file systems like UFS, and at runtime, they will be loaded into DDR and form a queue. But where do these Tasks come from? This involves a lot of behind-the-scenes work for AI algorithm deployment.

3.1 AI Inference Engine Architecture

From the perspective of the AI inference engine, let’s look at where this AI Runtime is positioned, as shown in the figure below:

It can be seen that this AI Runtime only organizes AI tasks at runtime, and these Tasks are determined at compile time, which will be reflected during NPU computation. Therefore, there is still a lot of work to be done above the AI Runtime, which will be specifically introduced in section 3.6 of AI Runtime.

All these functions involve the deployment of models in actual hardware, leading to the concept of the AI inference engine: The inference engine is a key component in the AI system, responsible for deploying trained models into practical applications, executing inference tasks, thereby achieving intelligent decision-making and automated processing.

Optimization Phase focuses on converting and optimizing trained models into deployable forms:

• Model Conversion Tools are responsible for converting models from research-stage formats to efficient execution formats, optimizing graphs to reduce computational burdens.
• Model Compression reduces model size through techniques such as pruning, quantization, and knowledge distillation, making it more suitable for resource-constrained environments.
• Edge Learning allows models to continue learning and adapting to new data after deployment without returning to the server for retraining, enhancing model performance under specific scenarios or user personalization needs.
• Other Components include benchmarking tools for performance evaluation and tuning guidance, as well as application demos for showcasing model capabilities and collecting practical feedback, collectively aiding in efficient deployment and continuous optimization of models.

Execution Phase ensures efficient execution of models on target devices:

• Scheduling Layer manages model loading, resource allocation, and task scheduling, flexibly arranging computational tasks based on device conditions.
• Execution Layer directly executes model computations, optimizing computation logic for different hardware, effectively utilizing CPU, GPU, and other resources.

A good inference engine can bring substantial benefits to user services, as shown in the bar chart above, where each bar represents the performance comparison of different inference engines on different models of mobile phones. The inference engines mentioned include NCNN, MACE, TF-Lite, CoreML, and MNN.

• NCNN: NCNN is a lightweight, high-performance neural network inference framework developed by Tencent Youtu Lab, designed to achieve extreme inference speed on mobile and embedded devices. It supports offline model conversion and can directly load and execute models trained from frameworks like Caffe, TensorFlow, and PyTorch. A notable feature of NCNN is that it has no third-party dependencies and is fully optimized for mobile, achieving high-performance computation by directly utilizing the CPU’s SIMD instruction set while also supporting GPU acceleration. Its design allows developers to achieve fast inference speeds even without a GPU.
• MACE: MACE is a mobile AI computing engine launched by Xiaomi, which stands for Mobile AI Compute Engine. MACE is designed to optimize the inference efficiency of neural network models on mobile devices, supporting various hardware accelerations such as CPU, GPU, and DSP. It provides a complete model conversion toolchain that can convert models trained from Caffe, TensorFlow, and PyTorch into MACE’s running format. MACE achieves efficient operation of models on mobile devices through highly optimized kernel libraries and hardware acceleration, with a particular focus on balancing power consumption and performance.
• TF-Lite: TF-Lite is a lightweight solution launched by Google’s TensorFlow team, specifically designed for mobile and embedded devices. It is a subset of the TensorFlow framework, focusing on the inference part of models, aiming to provide low-latency and low-power machine learning inference. TF-Lite supports model quantization and custom operators, significantly reducing model size and improving running efficiency. It supports multiple backends such as CPU, GPU, and NNAPI (Android Neural Networks API), allowing developers to choose the best inference strategy based on device characteristics.
• CoreML: CoreML is a machine learning framework designed by Apple for iOS devices, supporting the execution of pre-trained machine learning models on iPhones and iPads without an internet connection. CoreML is not limited to neural networks but also supports various machine learning models, such as support vector machines and decision trees. It is tightly integrated into the iOS, macOS, watchOS, and tvOS ecosystems, providing very low latency and efficient inference performance, making it particularly suitable for application development within the Apple ecosystem.
• MNN: MNN is a lightweight deep learning inference engine developed by Alibaba Damo Academy, aimed at providing high-performance inference capabilities for mobile and edge devices. MNN is particularly focused on optimizing internal business models, such as deep optimizations for tasks like face detection, enabling fast inference even on older hardware (like iPhone 6). It supports multi-platform deployment, including iOS, Android, and Linux, as well as various hardware accelerations such as CPU and GPU. MNN optimizes model execution through semi-automatic search, achieving efficient support for model and device diversity while maintaining flexibility in model updates.

3.2 Model Conversion Tools

Model conversion first faces the challenge of format crossing. Imagine a model initially trained in research-friendly frameworks like TensorFlow, PyTorch, or MindSpore; its original form is not directly applicable to inference engines in production environments. Therefore, conversion tools take on the role of “translators”, converting models from their originating framework language into one or more widely accepted standard formats, such as ONNX, which not only enhances the model’s portability but also provides a universal interface for subsequent processing and deployment.

• Cross-Framework Compatibility: Supports converting models from one framework (like TensorFlow, PyTorch) to another (like ONNX, TensorRT), enabling models to execute on different inference engines, enhancing flexibility in application development and platform universality.
• Version Adaptability: Addresses model compatibility issues caused by framework upgrades, ensuring that old models can run correctly on new versions of inference engines, or that new models can backtrack to support older version systems.
• Standardized Output: Converted models are typically formatted into one or more industry-standard formats, such as ONNX, which promotes interoperability of tools and services within the ecosystem.

Format conversion is just the tip of the iceberg. The real challenge lies in how to deeply optimize the computation graph, which is key to determining whether the model can execute efficiently. The computation graph, as a mathematical abstraction of the neural network structure, involves meticulous analysis and reshaping of the graph structure. Among them, operator fusion, layout transformation, operator replacement, and memory optimization constitute the core four steps of optimization, each step being a profound exploration and careful refinement of the model’s performance limits. Detailed explanations are not provided here; please refer to: zhuanlan.zhihu.com/p/687336210…

3.3 Model Compression

Model compression is a core technology in the AI field and an indispensable part of the inference engine architecture. It aims to reduce the size of the model through a series of clever strategies while keeping its predictive performance as unchanged as possible, and in some cases even accelerating training and inference processes. Achieving this goal relies on the comprehensive use of key technologies such as quantization, knowledge distillation, pruning, and binarization, each uniquely slimming the model while minimizing the sacrifice of its expressiveness. These contents will be detailed in subsequent articles, so only a brief introduction is provided here.

1. Quantization

The core idea of quantization technology is to convert the weights and activation functions in the model from high-precision floating-point numbers to low-precision data types, such as 8-bit integers or even binary forms. This conversion not only significantly reduces the model’s storage requirements but also accelerates the inference process due to the efficient implementation of low-precision operations on modern hardware. Of course, careful design of the quantization scheme is needed during the quantization process, such as selecting appropriate quantization ranges, quantization strategies, and error compensation methods, to ensure that precision loss is controlled within acceptable limits, achieving a balance between performance and accuracy.

2. Knowledge Distillation

Knowledge distillation, a vivid metaphor, refers to using a large and complex “teacher” model (usually a model with high accuracy) to guide a smaller “student” model to learn, allowing it to significantly reduce its size while maintaining relatively high accuracy. This process is achieved by having the student model mimic the output distribution of the teacher model or directly using the soft labels of the teacher model for training, thus facilitating knowledge transfer. Knowledge distillation is not limited to reducing model size; it also opens up new ideas for lightweight model design, especially when deploying models on resource-constrained devices.

3. Pruning Technology

Pruning technology, as its name suggests, aims to remove weights and connections in the model that contribute little to predictions or are redundant, simplifying the model structure. This includes but is not limited to weight pruning, channel pruning, and structured pruning strategies. By setting certain pruning thresholds or using sparsity constraints, “useless branches” in the model are identified and removed one by one, leaving a more refined core structure. It is worth noting that the pruning process is often accompanied by retraining or fine-tuning steps to recover any accuracy loss that may occur due to pruning, ensuring that model performance is not affected.

4. Binarization

Binarization, as the name implies, restricts the weights and even activation values in the model to only two discrete values (usually +1 and -1). This extreme quantization method further compresses the model size and simplifies computational complexity, as binary operations can be efficiently implemented at the bit level. Although binarized models are theoretically very attractive, they face challenges of accuracy loss in practice, requiring carefully designed training strategies and advanced optimization techniques to compensate.

3.4 Edge Learning

Edge learning, as a cutting-edge branch in the AI field, aims to overcome the limitations of traditional cloud-centric model training by directly endowing edge devices, such as mobile phones and IoT sensors, with learning capabilities, achieving localized and instantaneous data processing. The two core concepts of this paradigm—incremental learning and federated learning—are redefining the training and application methods of AI models, providing innovative solutions to issues such as data privacy, network latency, and computational resource allocation.

To support efficient edge learning, a complete inference engine is not just a platform for model execution; it also needs to integrate core modules such as data preprocessing, model training (Trainer), optimizer (Opt), and loss function (Loss), forming a closed-loop end-to-end solution.

Federated learning provides an innovative path to address data privacy and cross-device model training. In this framework, users’ personal data does not need to be uploaded to the cloud; instead, model parameters are updated on local devices, and only these updates (not the original data) are shared with a central server for aggregation, forming a global model. This process is repeated until the model converges. Federated learning not only protects user privacy, reduces the burden and risks of data transmission but also allows models to learn richer and more diverse features from distributed data, enhancing the model’s general applicability. The technical challenges lie in designing efficient and secure parameter aggregation algorithms and addressing issues arising from device heterogeneity and communication instability.

3.5 Intermediate Representation (IR)

In the design and implementation of modern inference engines, “Intermediate Representation” (IR) plays a crucial role, serving as a bridge between model training and actual inference execution. The core goal of intermediate representation is to provide a unified and efficient model description method, allowing models from different sources and architectures to be standardized for processing, thereby optimizing execution efficiency and enhancing compatibility across platforms. This concept penetrates every aspect of model optimization, compilation, and execution, and its importance is self-evident.

Intermediate representation provides rich optimization space for models. Most optimization work on computation graphs focuses on models after intermediate representation, where operations such as trimming, fusion, and quantization can be performed to reduce computational load and memory usage, thereby improving inference speed. This process is akin to compiling high-level programming languages into machine code, but it is aimed at neural network models.

A unified intermediate representation ensures that models can be freely deployed on various hardware types, such as cloud, edge, and terminal, achieving the goal of “one conversion, run everywhere”. It simplifies the adaptation work for specific hardware, allowing models to seamlessly migrate between different environments to meet diverse application needs.

A complete ecosystem can form around intermediate representation, including toolchains, library functions, and community support. Developers can leverage these resources to quickly implement model debugging, performance monitoring, and continuous optimization, accelerating the entire cycle from prototype to production.

3.6 AI Runtime

Runtime, the execution engine of the inference engine, is responsible for converting models in intermediate representation into executable instruction sequences and deploying them to target devices for execution. The execution engine involves not only the basic steps of model loading and execution but also encompasses various strategies and techniques to optimize resource utilization, enhance operational efficiency, and ensure high-performance performance across diverse hardware platforms. Taking autonomous driving as an example, we will introduce the role of Runtime technology in model inference.

Dynamic Batch Processing

Dynamic batch processing technology brings unprecedented flexibility to the inference engine, allowing the system to dynamically adjust batch sizes based on real-time system load conditions. During lighter load periods, such as early morning or late at night, when there are fewer vehicles and the number of image frames received by the system decreases, the inference engine can intelligently merge multiple image frames into a larger batch for processing. This strategy not only significantly improves the utilization of hardware resources, such as the massive parallel processing capabilities of GPUs, but also reduces the computational overhead per request, enabling the system to maintain efficient inference performance under lower loads.

Conversely, during peak periods or emergencies, when the system faces high load challenges, dynamic batch processing technology can quickly reduce batch sizes to ensure that each request can be responded to in a timely manner, thereby guaranteeing the immediacy and safety of the autonomous driving system. This ability to dynamically adjust batch sizes based on real-time load is crucial for autonomous driving systems to cope with unpredictable traffic fluctuations, enhancing system stability and ensuring efficient operation under various load conditions.

Heterogeneous Execution

Modern hardware platforms integrate diverse computing units, including CPUs, GPUs, and NPUs (Neural Network Processors), each with its unique advantages. Heterogeneous execution strategies intelligently allocate computational tasks to fully leverage the performance characteristics of these different processors. Specifically, this strategy assigns computational tasks to the most suitable processors based on the characteristics of different parts of the model and the current hardware status. For example, computationally intensive convolution operations are typically offloaded to GPUs or NPUs, as these processors excel at handling large matrix operations, while tasks involving complex control flows and data preprocessing are better suited for CPUs.

In autonomous driving, object detection models often need to perform various types of computational tasks. Among them, convolution layers, due to their computationally intensive nature, are very suitable for execution on GPUs or NPUs to fully utilize their powerful parallel processing capabilities. In contrast, logical judgments and data filtering operations that rely on complex control flows are better handled by CPUs. By adopting heterogeneous execution strategies, the inference engine of the autonomous driving system can automatically schedule convolution layer tasks to high-performance processors like GPUs while assigning data preprocessing and post-processing tasks to CPUs, achieving efficient and rapid overall computational flow. This strategy not only enhances system performance but also ensures that the autonomous driving system maintains excellent response speed and accuracy in various scenarios.

Memory Management and Allocation

During inference, efficient memory management and allocation strategies are crucial for ensuring operational efficiency. These strategies encompass multiple aspects, such as reusing memory buffers to reduce unnecessary data copying, intelligently preloading parts of the model’s data into caches to lower access latency, and implementing memory fragmentation management mechanisms to maximize available memory resources. These measures not only help reduce memory usage but also significantly enhance data read and write speeds, playing a vital role in the rapid execution of models.

In the process of autonomous driving vehicles, real-time processing of high-resolution images poses significant challenges to memory resources. To address this challenge, inference engines typically adopt intelligent memory management strategies. For example, through circular buffer reuse techniques, the inference engine can reuse the memory space occupied by previous frames to store intermediate results such as feature maps before processing new image frames. This approach not only avoids frequent memory allocation and deallocation operations, reducing memory fragmentation, but also significantly improves memory utilization efficiency and the overall response speed of the system. These refined memory management strategies ensure that the autonomous driving system can operate efficiently and stably, supporting real-time and accurate decision-making.

Big-Little Core Scheduling

On mobile devices, the performance differences between big cores (high-performance cores) and little cores (low-power cores) are significant. To maximize the efficiency of hardware resource utilization, the inference engine must be able to dynamically adjust the allocation of computational tasks between these two types of cores. This requires the inference engine to possess advanced scheduling capabilities to intelligently assign tasks to the most suitable processor cores based on the current task load and type.

For processors using big.LITTLE architecture, the inference engine can adopt refined task scheduling strategies. Specifically, it can assign computationally intensive and performance-demanding tasks, such as complex logical operations for vehicle path planning in autonomous driving, to high-performance big cores to ensure sufficient computational power and processing speed. In contrast, lightweight tasks such as data preprocessing and simple state monitoring can be handled by low-power little cores to save energy and extend battery life.

In the context of autonomous driving, this dynamic load balancing capability of the inference engine is particularly important. It can adjust the allocation of tasks between big and little cores in real-time based on the actual task demands during vehicle operation, ensuring that sufficient computing power is available for processing complex tasks while effectively reducing energy consumption during lightweight tasks, providing a more efficient, stable, and energy-saving operating environment for the autonomous driving system.

Multi-Replica Parallelism and Batching Technology

In distributed systems or multi-core processor architectures, multi-replica parallelism technology demonstrates its unique advantages. This technology creates multiple replicas of the model and assigns them to different computing units for parallel execution, achieving linear or near-linear performance acceleration. Meanwhile, batching technology, as an effective parallel processing strategy, significantly enhances the system’s throughput and resource utilization when facing a large number of small requests by merging multiple independent requests into a single batch for centralized processing.

In real-time applications of autonomous driving, the requirements for latency response are extremely strict. Multi-replica parallelism technology is perfectly applied here, especially in multi-GPU systems. Each GPU runs a replica of the model, capable of simultaneously processing multiple sensor data streams, achieving parallel inference and significantly shortening decision-making time, ensuring that vehicles can quickly respond to various road conditions. On the other hand, batching technology also performs excellently when processing a single high-resolution image frame. By segmenting the image into multiple small regions, each region is processed as a small batch, ensuring real-time performance while significantly improving the overall throughput of the system, allowing the autonomous driving system to operate efficiently in complex driving environments.

3.7 High-Performance Operators

Operator Optimization

Operator optimization is key to improving model running efficiency. By optimizing the operators in the model, it is possible to effectively reduce computational load, lower memory usage, and increase computation speed. Common optimization methods include:

• Fusion Optimization: In neural network models, many operators may have redundant computational steps or memory copy operations. Fusion optimization aims to merge adjacent operators into a single operator, thereby reducing the overall number of computations and memory usage. This optimization strategy can significantly improve the computational efficiency of the model, especially when running on hardware accelerators (such as GPUs or TPUs).
• Quantization Optimization: Quantization is a technique that converts floating-point operations into integer operations to reduce computational complexity and improve model running speed. By lowering the precision of data representation, quantization can reduce the resources required for computation and may allow the model to run on resource-constrained devices. Although quantization may lead to slight accuracy loss, this loss is acceptable in many applications.
• Sparsity Optimization: Sparsity optimization is a special optimization for sparse matrices or sparse tensors. In neural network models, weight matrices or feature tensors may contain a large number of zero-value elements that do not contribute to computations. Through sparsity optimization, we can skip these invalid computations and only process non-zero elements, thereby improving computational efficiency. For example, using sparse matrix multiplication algorithms instead of conventional matrix multiplication algorithms can significantly reduce computational load. Additionally, sparsity optimization can be combined with quantization optimization to further lower computational complexity and memory usage.

Operator Scheduling

Operator scheduling is a crucial link in high-performance computing, involving the reasonable planning and determination of the execution order and location of operators based on the actual availability of hardware resources and the characteristics of the operators. The choice of scheduling strategy directly affects the computational efficiency and performance of the entire system when constructing a high-performance operator layer.

1. Heterogeneous Scheduling

Heterogeneous scheduling is an intelligent operator scheduling strategy that allocates operators to the most suitable hardware based on their type, complexity, and computational requirements, combined with the characteristics of different hardware (such as CPU, GPU, FPGA, etc.). For example, for computationally intensive operators, they may be prioritized for scheduling on GPUs, as GPUs have more computing cores and higher parallel computing capabilities; whereas operators that require frequent memory access may be chosen to execute on CPUs, as CPUs have advantages in memory access. By employing heterogeneous scheduling, we can fully leverage the advantages of different hardware, maximizing the utilization of computational resources and thereby improving overall computational efficiency.

2. Pipelined Scheduling

Pipelined scheduling is an efficient operator scheduling strategy that arranges multiple operators in a logical order to form a computation pipeline. In the pipeline, each operator has its own processing unit and buffer, allowing for independent computation without waiting for the previous operator to complete all calculations. When the first operator completes part of its computation and passes the result to the next operator, it can continue processing new data, achieving continuous and concurrent computation. Pipelined scheduling can greatly enhance computation speed, reduce waiting time, and significantly improve the throughput of the entire system.

3.8 Inference Process Summary

Based on the introduction of the inference engine structure above, we can derive the following inference process, which involves multiple steps and components, including preparation work in offline modules and the online execution process, all working together to complete inference tasks.

First, the inference engine needs to process models from different AI frameworks, such as MindSpore, TensorFlow, PyTorch, or PaddlePaddle. The models trained from these frameworks will be sent to model conversion tools for format conversion to adapt to the specific format of the inference engine.

The converted inference model needs to undergo compression processing. Compressing models is a common step in the inference engine, as uncompressed models are rarely seen in practical applications. After compression, the model needs to undergo environmental preparation, which involves a lot of configuration work, including big-little core scheduling, obtaining model documentation, etc., to ensure that the model can run in the correct environment.

After completing environmental preparation, the inference engine will proceed with development and compilation, generating the process used for executing inference. This inference process is the core component for executing inference tasks, relying on the APIs provided by the inference engine to offer the necessary interfaces for module or task development.

Development engineers will work according to this process. After development is completed, the inference engine will execute inference tasks, running them in runtime. At this point, the execution of the inference engine depends on the results of inputs and outputs, which involves the online execution part.

Developing inference programs is a complex process involving multiple steps such as model loading, configuration, data preprocessing, inference execution, and result post-processing. Below is a simple example illustrating how to develop an inference program.

1. Model Conversion

First, the trained model needs to be converted into a format that the inference engine can understand. This step is usually completed by model conversion tools, forming a unified expression format.

2. Configure Inference Options

This typically involves setting the model path, selecting the device to run on (such as CPU, GPU, etc.), and whether to enable computation graph optimization. An AI framework will provide relevant APIs to set these options and automatically apply them during model loading.

Config config;
config.setModelPath("path_to_model");
config.setDeviceType("GPU");
config.setOptimizeGraph(true); 
config.setFusion(true);       // Enable operator fusion
config.setMemoryOptimization(true); // Enable memory optimization

3. Create Inference Engine Object

Once the configuration options are set, the next step is to create the inference engine object. This object will manage the entire inference process, including loading the model and executing inference. Creating the inference engine object typically requires passing the configuration object as a parameter.

Predictor predictor(config);

4. Prepare Input Data

Before executing inference, the input data must be prepared. This includes preprocessing the raw data, such as subtracting the mean, scaling, etc., to meet the model’s input requirements. Then, the names of all input Tensors of the model need to be obtained, and the pointer for each input Tensor is retrieved through the inference engine, followed by copying the preprocessed data into these Tensors.

// Preprocess data
auto preprocessed_data = preprocess(raw_data);

// Get input Tensor names and pointers
auto input_names = predictor->GetInputNames();
for (const auto& name : input_names) {
   auto tensor = predictor->GetInputTensor(name);
   tensor->copy(preprocessed_data);
}

5. Execute Inference

Once the input data is ready, inference can be executed. This typically involves calling the Run method of the inference engine, which will initiate the inference process of the model.

predictor->Run();

6. Obtain Inference Results and Post-Processing

After the inference execution is complete, the output Tensors need to be obtained, and the inference results copied out. Then, post-processing is performed based on the model’s output, such as cropping images based on the positions of detection boxes in object detection tasks.

// Get output Tensor names and pointers
auto out_names = predictor->GetOutputNames();
std::vector<processedoutput> results;
for (const auto& name : out_names) {
   auto tensor = predictor->GetOutputTensor(name);
   ProcessedOutput data;
   tensor->copy(data);
   results.push_back(processOutput(data));
}

// Post-processing, e.g., cropping images, etc.
for (auto& result : results)
    postprocess(result);
</processedoutput>

References:

1. zhuanlan.zhihu.com/p/707141458…
2. zhuanlan.zhihu.com/p/687336210…

“Understanding a little about everything, but mastering nothing,

able to do anything, but not excelling at anything,

professional entry-level discouragement, a true jack of all trades programmer.”

Welcome to leave a message if you have your own public account: Apply for Reprint!

Pure dry goods will be continuously updated, welcome to share with friends, like, collect, watch, underline, and comment for communication!

Surprise:

1. This public account provides a WeChat technical exchange group to discuss automotive software technology together (add WeChat: thatway1989, note the technical direction of interest).

2. If you need to advertise or collaborate, you can also contact the author.

3. Appreciation of 1 yuan to make friends

   Recommended by the author: functional safety for those interested to buy a book