With the rapid development of artificial intelligence technology, Deep Neural Networks (DNN) have become an indispensable computational model in various fields, especially in image recognition, speech processing, natural language processing (NLP), and intelligent driving. To effectively handle large-scale DNN computational tasks, traditional computing architectures (such as CPUs and GPUs) can no longer meet the growing computational demands. At this point, dedicated AI accelerators—especially Neural Processing Units (NPUs)—have become an important component of AI computing. Huawei’s Ascend architecture, as Huawei’s self-developed AI acceleration platform, has become one of the most advanced AI accelerators due to its efficient computing power, flexible hardware design, and strong scalability. The Ascend architecture is not only suitable for deep learning tasks in data centers and the cloud but can also be widely applied in edge computing, the Internet of Things (IoT), and smart hardware scenarios. Huawei’s Ascend series chips, including Ascend 310, Ascend 910, and its upgraded version (Ascend 910B), demonstrate significant advantages in the field of AI computing by adopting innovative computing units and memory architectures.
Design Principles and Technological Innovations of the Ascend Architecture
The design of the Ascend architecture innovates from multiple aspects, including heterogeneous design of computing units, optimization of memory architecture, task scheduling, and deep integration with the software stack.
Figure 1. Ascend Architecture Diagram.
Heterogeneous Computing Units and Da Vinci Architecture
The design of the Ascend NPU adopts the Da Vinci Architecture. The Da Vinci Architecture integrates various computing units, including scalar units, vector units, and cube units, which are designed to maximize the computational throughput of DNN tasks and overcome the bottlenecks of traditional computing architectures when handling deep learning tasks.
Scalar Computing Unit
The scalar computing unit is similar to the integer ALU unit in classic RISC processors, mainly responsible for control flow operations and some simple computational operations, such as addition, subtraction, and multiplication. Although these computing units have limited computational capabilities, they primarily undertake control tasks and manage simple computational operations in the Ascend architecture, ensuring efficient execution of task scheduling and control flow.
Figure 2. Scalar Computing Unit.
Vector Computing Unit
The vector unit plays an important role in the Ascend architecture, similar to the SIMD (Single Instruction Multiple Data) units in traditional CPUs and GPUs, widely used in high-performance computing (HPC) and computer vision (CV) fields. It can perform most computational operations in inference and training tasks (e.g., normalization, activation functions). However, using vector units does not fully utilize data reuse in DNN models. Therefore, it leads to bandwidth-limited data paths between high-density ALUs and local memory, constrained by physical layout and routing (P&R). Thus, one-dimensional vector units are insufficient to meet the increasing computational demands of DNN applications.
Figure 3. Vector Computing.
Cube Computing Unit
To overcome the limitations of vector units, two-dimensional computing units have been used for General Matrix-Matrix Multiplication (GEMM). Mathematically, there are two methods to accelerate GEMM in 2D units. One is a dot-product-based method that executes GEMM as multiple General Matrix-Vector Multiplications (GEMVs). The other is an outer-product-based method that converts GEMM into the accumulation of multiple vector outer products. To further achieve better computational efficiency, 3D (cube) computing units have been introduced to provide substantial computational power. Cube units are specifically designed for DNN applications to achieve higher data reuse. They help alleviate the mismatch between computational throughput and limited memory bandwidth in DNN accelerators. The typical size of matrices in 3D computations is 16*16*16. Therefore, the entire cube is equipped with 4096 multipliers and 4096 accumulators. Additionally, each operand in matrix computations is reused 16 times. Thus, compared to vector units, the energy consumption of loading/feeding operands into the cube computing unit is reduced to 1/16, enabling higher data reuse in DNN applications and improving computational throughput. This design is particularly suitable for large-scale matrix operations, such as convolution and matrix multiplication (GEMM), which are fundamental AI computing operations. Compared to traditional two-dimensional vector units, cube units achieve higher computational density and greater data reuse by increasing dimensions, significantly enhancing computational performance.
Figure 4. Cube Computing Unit.
Memory Architecture and Data Path Optimization
As shown in Figure 5, the memory system of the Ascend architecture adopts a multi-level cache structure, including L0 cache, L1 cache, and High Bandwidth Memory (HBM). This design aims to reduce memory bandwidth bottlenecks and improve computational efficiency. By optimizing memory access paths, the Ascend architecture can effectively enhance data transfer efficiency, especially when handling large-scale AI tasks, maintaining high-speed data transfer and processing capabilities.
Figure 5. Ascend Memory Architecture.
Memory Transfer Engine (MTE)
The Memory Transfer Engine (MTE) is a key component in the Ascend architecture, responsible for transferring data between different memory levels. MTE supports functions such as compression and decompression, matrix transformation, etc., to improve memory access efficiency. Multiple functional modules in MTE, including decompression modules, image transformation modules (img2col), and matrix transposition modules, effectively enhance the efficiency of data transfer between different computing units, reducing data transfer latency.
High Bandwidth Memory (HBM)
The Ascend architecture also employs High Bandwidth Memory (HBM) technology, providing higher bandwidth than traditional memory. This is particularly important for processing large-scale datasets, especially in deep learning training and inference tasks, where data access bottlenecks can significantly impact system performance. The Ascend chip effectively addresses this issue by combining high bandwidth memory with on-chip caches, ensuring that computing units can quickly access the required data.
Task Scheduling and Communication Queues
In the Ascend architecture, communication queues are used to control data transfer and synchronization, achieved by coordinating the flow of data and the execution order of tasks between computing units. Specifically, queues are used to store data packets to be processed and ensure that data is transmitted correctly and in order between different computing units through a series of synchronization mechanisms. Whenever a computing unit completes its operation, it passes the result to the next computing unit, ensuring that tasks are executed according to predetermined dependencies. The queue not only facilitates data transfer but also controls the execution rhythm of computing units and task scheduling, optimizing overall computational performance and avoiding resource conflicts. Additionally, the Queue design in the Ascend architecture supports dynamic adjustment and priority control to adapt to the needs of different application scenarios.
Figure 6. Communication Queue Controls Data Synchronization.
Programming Model
The programming language used in the Ascend architecture, Ascend C, follows the SPMD (Single-Program Multiple-Data) paradigm. SPMD is a commonly used method for parallel computing and an effective means to improve computational speed. Assuming that processing input data to output data requires three stages of task processing (T1, T2, T3). As shown in Figure 7, under the SPMD model, the system starts a group of processes to process the data in parallel: first, the data to be processed is divided into multiple data shards, and the divided data shards are then distributed to different processes for processing, with each process receiving one or more data shards and independently processing these shards through the three tasks.
Figure 7. SPMD Paradigm of Ascend C.
SPMD programming model splits the data to be processed and runs it simultaneously on multiple computing cores (analogous to multiple processes mentioned above), thereby achieving higher performance. Multiple AI Cores share the same instruction code, with the only difference in the running instance on each core being the block_idx. Each core identifies its identity through different block_idx. The concept of a block is similar to the concept of a process mentioned above, and block_idx is the process ID that uniquely identifies the process.
Figure 8. AI Core Processes Different Data in Parallel.
Support for Software Stack and Compilation Optimization
The collaborative design of hardware and software in the Ascend architecture enables it to support mainstream AI development frameworks such as TensorFlow, PyTorch, and MindSpore. In particular, MindSpore, as Huawei’s self-developed deep learning framework, has been deeply integrated with the Ascend architecture to maximize the computational performance of Ascend chips. Additionally, the Ascend architecture is equipped with an efficient compiler that can automatically optimize computation graphs for different hardware configurations, generating efficient execution code. The compiler of the Ascend architecture supports various optimization strategies, including computation graph optimization, operator fusion, and data layout transformation. Through these optimizations, Ascend can significantly enhance the execution efficiency of AI models, reduce the mismatch between memory bandwidth and computational throughput, thereby improving the overall computational capability of the system.
Ascend Chip Series
The Ascend architecture includes several different models of AI acceleration chips, each designed for different application scenarios, providing varying computational performance and power characteristics.
Ascend 310
The Ascend 310 chip is the entry-level product in the Ascend series, primarily aimed at edge computing and Internet of Things (IoT) devices. This chip features 8 Da Vinci AI cores and 4 ARM A73 cores, supporting deep learning inference tasks. The Ascend 310 has low power consumption characteristics, making it very suitable for use in low-power devices, such as smart homes, smart monitoring, and in-vehicle computing scenarios. The Ascend 310 provides relatively low computational power, approximately 16 TFLOPS of FP16 performance, but its low power consumption and efficient computing capabilities have led to widespread applications in edge devices. This chip can handle basic AI tasks such as image recognition and speech recognition while achieving a good balance between high performance and low power consumption.
Ascend 910
The Ascend 910 is the flagship product in the Ascend architecture, designed for data centers and cloud computing environments, supporting high concurrency and multi-task training and inference workloads. The Ascend 910 offers 256 TFLOPS of FP16 performance and supports large-scale parallel computing. With 32 Da Vinci AI cores, the Ascend 910 supports ultra-large-scale AI model training and inference tasks. In AI inference and deep learning training, the Ascend 910 demonstrates powerful computational capabilities, particularly suitable for high-load, high-concurrency AI tasks. Its high bandwidth memory and advanced memory architecture further enhance system performance, making it a significant player in cloud computing and big data processing.
Future Development and Challenges
Despite the significant achievements of the Ascend architecture, it still faces some challenges in future development, particularly in expanding into global markets and improving the software ecosystem. The Ascend architecture has achieved good results in the Chinese market, but in the international market, it still faces strong competition from rivals like NVIDIA. The Ascend architecture needs to further optimize its software ecosystem and enhance compatibility with mainstream AI frameworks to secure a place in the global market. As AI technology continues to advance, the Ascend architecture must continuously innovate in both hardware and software to tackle increasingly complex AI tasks. For example, there is still considerable room for improvement in supporting multimodal AI and reinforcement learning.
Conclusion
Huawei’s Ascend architecture, with its innovative hardware design, flexible computing units, and efficient memory architecture, has become an important force in the field of AI acceleration. Through an in-depth analysis of the various technologies and performance of the Ascend architecture, we can see that Ascend not only provides efficient computing support but also promotes the popularization and application of AI technology. With continuous technological advancements, the Ascend architecture is expected to exert a greater influence globally and drive the development of AI technology.
Source: Ascend NPU Processor Hardware Architecture and Key Technologies