Currently well-known computing processor architectures (data computation-centric) include: CPU, GPU, and NPU. What role does the NPU play here?In the context of today’s generative AI, heterogeneous processor combinations are used to achieve heterogeneous computing for different tasks by integrating NPUs, aiming for optimal application performance, energy efficiency, and power consumption.

1 Concept

1.1 Heterogeneous Computing

Why is heterogeneous computing necessary?The diverse requirements and computational demands of generative AI necessitate different processors to meet them. A heterogeneous computing architecture that supports diversity can leverage the strengths of each processor, such as NPUs designed specifically for AI, as well as CPUs and GPUs. Each processor excels at different tasks: CPUs are good at sequential control and immediacy, GPUs are suitable for parallel data stream processing, and NPUs excel at scalar, vector, and tensor mathematical operations, which are essential for core AI workloads.Heterogeneous computing can achieve optimal application performance, energy efficiency, and battery life, maximizing the user experience for generative AI.Different processors excel in various scenarios;

Currently, the core technology underlying LLM/LVM implementations is the Transformer architecture; thus, NPUs primarily focus on computing for transformer model architectures to achieve high performance and low power consumption.The irreplaceability of transformers:

• Revolution in sequence modeling: Completely replaces RNN/LSTM, solving long-range dependency issues (supports >100K context)
• Advantages of parallel computing: Self-attention mechanism can be parallelized → Training speed is over 10 times faster than RNN
• Generalization ability: The same architecture adapts to text, images, and audio (e.g., Vision Transformer)

1.2 NPU

NPUs are specifically designed to accelerate AI inference workloads with low power consumption and high performance; AI workloads mainly consist of neural network layer computations involving scalar, vector, and tensor mathematics, as well as nonlinear activation functions. An excellent NPU design can make the right design choices for handling these AI workloads.

2 NPU Architecture

When studying NPU architecture, it is found that the most in-depth information available online is about the hardware architecture of the ASCEND NPU.

2.1 NPU Chip Architecture

Here, we take the ASCEND 910 NPU chip architecture as an example:

From the image above, we can see the main components of the system:

• AI Core: The computing core responsible for executing matrix, vector, and scalar computation-intensive operator tasks;
• Taishan CPU: Handles non-matrix complex computations, part of which is AI CPU, responsible for some AI computations, and part is Ctrl CPU, responsible for SoC control functions. The number of cores occupied by the two types of CPUs is allocated by software; ARM architecture;
• TaskScheduler Core: Responsible for task scheduling, splitting operator tasks, and distributing them to the AI Core or AI CPU for computation through the hardware scheduler (HWTS);
• ARM CPU: Controls the overall operation of the chip;
• Chip NOC: On-chip interconnect network;
• Cache/Buffer/MEM: Storage
• DVPP: Digital Visual Preprocessing Subsystem, responsible for image and video encoding/decoding
• High-speed interfaces, low-speed interfaces;

2.1 NPU Da Vinci Architecture

1 Computing Units

Cube Core: Can perform fp16 matrix operations in each execution, i.e., C = A(1616) * B(1616), larger matrix operation blocks are processed in chunks and cached in L1 Buffer;VectorUnit: For mathematical operations such as reciprocal, square root, etc., all source and target data for Vector calculations are stored in Unified Buffer (Unified Buffer interacts with L1 Buffer) and aligned to 32 Bytes;ScalarUnit: Responsible for various types of scalar data operations and program flow control, with the lowest computing power, functionally similar to a small core CPU, completing the entire program loop control, branch judgment, Cube/Vector instruction address and parameter calculations, and basic arithmetic operations;Accumulator: Adds the current matrix multiplication result to the previous computation result, which can be used to complete operations such as adding bias in convolution;

2 Storage Units

Memory Control Unit (MTE): Acts as the internal data path transmission controller for the AI Core, responsible for managing read and write operations of data within different buffers in the AI Core, as well as completing a series of format conversion operations;Input Buffer: L1 Buffer: The AI Core adopts a large-capacity on-chip buffer design, increasing the amount of data in on-chip cache to reduce the frequency of data transfers from off-chip to the AI Core, thereby reducing power consumption and latency during data transfer, effectively controlling overall computational energy consumption and improving performance;Output Buffer (Unified Buffer): Used to store intermediate results of each layer’s computation in the neural network, facilitating data retrieval when entering the next layer;Registers (SPR/GPR): Register resources are mainly used by scalar computing units.

The data flow path on the AI core has the following characteristics:

• Multiple inputs and single output, allowing for improved efficiency of data inflow through parallel inputs;
• After processing various input data, only the output feature matrix is generated, with relatively single data types, and a single output data path, which can save chip hardware resources;

3 Control Units

The system control module: Controls the execution process of “task blocks” (the smallest execution task computation granularity in the AI Core). After the task block execution is completed, the system control module will handle interrupts and state reporting.Instruction cache;Scalar instruction processing queue;Instruction dispatch module: Sends instructions to the corresponding execution queues based on instruction types;Event synchronization module.

3 NPU Working Process

3.1 NPU Parallel Computing Architecture

• Asynchronous instruction flow: Scalar computing units read instruction sequences and send vector computation, matrix computation, and data transfer instructions to the corresponding instruction queues. Vector computing units, Cube computing units, and DMA transfer units execute the received instructions asynchronously in parallel (“asynchronous parallel”: decomposing serial instruction streams);
• Synchronization signal flow: There may be dependencies between instructions, and to ensure that instructions in different queues are executed according to the correct logical relationships, the Scalar computing unit will also issue synchronization instructions to the corresponding units;
• Computational data flow: The DMA transfer unit moves data to Local Memory, Vector/Cube computing units complete data calculations, and write the results back to Local Memory, while the DMA transfer unit moves the processed data back to Global Memory.The following diagram illustrates this process very clearly:

3.2 Computing Modes

Cube units execute matrix operations, compared to scalar and vector units:The required computation cycles are shown in the following diagram;

Matrix preprocessing:

• Tiling: Due to limited on-chip cache capacity, the entire matrix B is divided into multiple sub-matrices, which are then sequentially moved to the cache, ultimately resulting in the output matrix C;
• Padding: A and B are both divided into equally sized blocks (16*16), and any unfilled areas can be filled with zeros.

4 CANN Abstract Computing Platform

The CANN platform supports various AI frameworks, including MindSpore, PyTorch, TensorFlow, etc., and serves AI processors and programming, playing a key role in enhancing the computational efficiency of Ascend AI processors. It also provides multi-level programming interfaces for diverse application scenarios, enabling users to quickly build AI applications and services based on the Ascend platform.The position is shown below:It is still necessary to have the opportunity to directly download and use it for learning and experimentation;