【Editor’s Note】 Following Huang’s GTC, Huawei also held the Ascend AI Developer Day in Tianjin on March 23, 2024. We can review the GPU performance (data sheet) for quick reference in 2023 and the latest notes on GPU: Huawei NPU/GPU Evolution (2024) regarding the differences and performance comparisons between the N and H cards. Huawei launched the Ascend series AI processors (NPU, Neural-Network Processing Units) in 2018. This article organizes the introduction of the Da Vinci architecture from Huawei’s official website for reference and study.
【Summary】The Ascend AI processor is essentially a System on Chip (SoC), which can be roughly classified into: chip system control CPU (Control CPU), AI computing engine (including AI Core and AI CPU), multi-level on-chip cache (Cache) or buffer (Buffer), digital vision pre-processing module (Digital Vision Pre-Processing, DVPP), etc.The Ascend AI processor has powerful computing capabilities and is specially optimized for deep neural networks in its hardware architecture, enabling it to efficiently perform forward computations for mainstream deep neural networks, thus having broad application prospects in fields such as smart terminals.
The main architectural components of the Da Vinci architecture include scalar, tensor, and Cuba operations, with flexibility from front to back, and efficiency conversely.The Huawei Da Vinci core is divided into three types: Max, Lite, and Tiny, with Max being the most complete. Max can perform 8192 MAC operations in one cycle, while Tiny can only perform 512.
The Da Vinci Core is just a part of the NPU, which is further subdivided into many units, including the core 3D Cube, Vector computation unit, Scalar computation unit, etc., each responsible for different computational tasks to achieve parallel computation models, ensuring efficient processing of AI computations.Two buffers L0A and L0B serve as inputs, one temporarily storing activations, and the other temporarily storing weights. The red Cube module, similar to Google’s TPU pulsating array structure, is a multiply-accumulate array, with the computation results stored in another buffer L0C. The Vector Unit, similar to a DSP, retrieves the multiply-accumulate results from buffer L0C and then performs pooling/padding, BN, activation, addition, etc. If the final result is not yet obtained, it temporarily returns to the buffer L0C; if the final result is obtained, it is passed to the Unified Buffer. The Scalar Unit is used to control scheduling, similar to an MCU, which can be ARM or RISC-V.
Source: Hisilicon Official Website Davinci
The processor is essentially a System on Chip, primarily applicable in scenarios related to image, video, voice, and text processing. Its main architectural components include specialized computation units, high-capacity storage units, and corresponding control units.
- Multi-Core (AI CPU + Control CPU): The Ascend AI processor integrates multiple CPU cores, each with independent L1 and L2 Cache, while all cores share a single on-chip L3 Cache.
- The integrated CPU cores can be classified based on function into Control CPU, dedicated to controlling the overall operation of the processor, and AI CPU, responsible for handling complex non-matrix calculations. The number of CPU cores allocated for these two tasks can be dynamically adjusted by software based on the actual operating conditions of the system.
- AI Core: Besides the CPU, the true computational powerhouse of this processor is the AI Core that employs the Da Vinci architecture. These AI Cores are designed with specialized architectures and circuits to achieve high throughput, substantial computing power, and low power consumption, making them particularly suitable for handling common computations required in deep learning neural networks, such as matrix multiplication.
- 8 MB On-Chip L2 Cache: To accommodate the large number of parameters and numerous intermediate values in deep neural networks, this processor is equipped with an on-chip L2 buffer, designed to cache a large amount of reusable intermediate data on-chip, thereby providing high bandwidth, low latency, and efficient data exchange and access, while also helping to reduce the overall power consumption of the system.
Task Scheduler (TS): Dedicated CPU for efficiently allocating and scheduling computational tasks on the AI Core. This CPU specifically serves the AI Core and AI CPU, without undertaking any other tasks.Digital Vision Pre-Processing Module (HW) (Digital Vision Pre-Processing, DVPP): Before video and image data enter the computing engine of the Ascend AI processor for processing, they need to be formatted to meet processing requirements, such as input format and resolution. Therefore, the digital vision pre-processing module is required to perform preprocessing to achieve format and precision conversions (a corresponding hardware circuit module is designed for each different function to complete the computation).
-
Video Codec: Video decoding (Video Decoder, VDEC), video encoding (Video Encoder, VENC) (supports 4K resolution).
-
Image Codec: JPEG decoding (JPEG Decoder/Encoder, JPEGD/E), PNG decoding (PNG Decoder, PNGD).
-
Image Preprocessing can perform various functions such as up/down sampling, cropping, and color tone conversion on the input images.
Source: Ascend (HUAWEI Ascend) 310 Key Features
Domain-Specific Architecture to AI Core
Unlike traditional CPUs and GPUs that support general-purpose computing, and different from ASIC chips dedicated to specific algorithms, the Da Vinci architecture is essentially designed to accommodate common applications and algorithms in a specific domain, commonly referred to as Domain-Specific Architecture (DSA) chips.The computational core of the Ascend AI chip primarily consists of the AI Core, responsible for executing computation-intensive operators related to scalar, vector, and tensor operations.The AI Core employs the Da Vinci architecture, with its basic structure as shown in the above figure, which can be considered a relatively simplified modern microprocessor architecture.It includes three basic computational resources:Matrix Computation Unit (Cube Unit), Vector Computation Unit (Vector Unit), and Scalar Computation Unit (Scalar Unit). These three computation units correspond to tensor, vector, and scalar, three common computation modes, each performing its respective role during actual computations, forming three independent execution pipelines, working together under the unified scheduling of system software to achieve optimized computational efficiency. Additionally, the matrix computation unit and vector computation unit provide various precision and type computation modes. The matrix computation unit in the AI Core currently supports INT8, INT4, and FP16 computations; the vector computation unit currently supports FP16 and FP32 computations.To facilitate data transmission and handling within the AI Core, a series of on-chip buffers are distributed around the three computational resources, such as input buffer (Input Buffer, IB) and output buffer (Output Buffer, OB) for placing overall image feature data, network parameters, and intermediate results, as well as high-speed register units for providing temporary variables, which are located within each computation unit. The design architecture and organization of these storage resources vary, but their purpose is to better adapt to the format, precision, and data layout requirements under different computation modes. These storage resources are connected to associated computational resources or to the bus interface unit (Bus Interface Unit, BIU) to access data on the external bus.
In the AI Core, after the input buffer, there is a Memory Transfer Unit (MTE). This is one of the features of the Da Vinci architecture, primarily aimed at achieving data format conversion with extremely high efficiency. For example, as mentioned earlier, the GPU needs to implement convolution through matrix computation, and first, it must rearrange the input network and feature data into a specific format using the Im2Col method. This step is implemented in software on the GPU, which is relatively inefficient. The Da Vinci architecture employs a dedicated Memory Transfer Unit to complete this process, solidifying it entirely in hardware circuitry, allowing the entire transposition process to be completed in a very short time. Since computations similar to transposition occur very frequently in deep neural networks, the design of such customized circuit modules can enhance the execution efficiency of the AI Core, enabling uninterrupted convolution computations.The control unit in the AI Core mainly includes the system control module, scalar instruction processing queue, instruction dispatch module, matrix operation queue, vector operation queue, storage transfer queue, and event synchronization module. The system control module is responsible for directing and coordinating the overall operation mode of the AI Core, configuring parameters, and implementing power consumption control. The scalar instruction processing queue primarily decodes control instructions. When instructions are decoded and dispatched sequentially through the instruction dispatch module, they will be sent to the matrix operation queue, vector operation queue, and storage transfer queue based on the different types of instructions.Instructions in the three queues are output to the matrix computation unit, vector computation unit, and storage transfer unit for corresponding computations in a first-come, first-served manner. Different instruction arrays and computational resources form independent pipelines that can be executed in parallel to improve instruction execution efficiency. If there are dependencies or strict sequential requirements during instruction execution, the event synchronization module can be used to adjust and maintain the execution order of instructions. The event synchronization module is entirely controlled by software, and during software development, the execution timing of each pipeline can be adjusted by inserting synchronization symbols to achieve the desired instruction execution order.In the AI Core, the storage unit provides transposed data that meets the requirements for each computation unit, the computation units return the results of computations to the storage unit, and the control unit provides instruction control for the computation and storage units, with the three collaborating to complete computation tasks.Thus, the Ascend series AI processors (NPU, Neural-Network Processing Units) can provide powerful and efficient multiply-accumulate computing power for integer (INT8, INT4) or floating-point (FP16) numbers.【Next post video introduction】
https://www.hiascend.com/forum/thread-0270130417281785016-1-1.htmlGTC 2024 REPORT: Witnessing the AI Transformation
GTC2024: Grace Blackwell
NVIDIA Blackwell Official Introduction
GTC2024—AI Models
GTC2024-CEO Keynote PART-AI Robot
Latest Notes on GPU: Huawei NPU/GPU Evolution
Latest High-Performance GPU Server Hardware Topology and Cluster Networking-1
GPU Performance (Data Sheet) Quick Reference (2023)