In recent days, there has been a lot of discussion about a major company shifting from NPU to GPGPU, but the internet is filled with many mindless interpretations. First, I recommend reading a professional analysis: the author is Zha Zong. https://mp.weixin.qq.com/s/iqRXwdHjyMI7egMrXEIfFw

The content is quite lengthy, so let’s summarize it briefly:

NVIDIA’s GPU development shows a clear cycle, from the early fixed pipeline DSA architecture for graphics rendering, to the development of a unified Shader architecture and SIMT abstraction to address load imbalance issues, and then back to DSA with the introduction of Tensor Core in the AI era. Throughout this period, NVIDIA maintained its industry position through continuous optimization of the CUDA ecosystem. In contrast, domestic giant Ascend has approached the problem of ecological fragmentation caused by heterogeneous architectures from a system-level unified perspective, proposing a homogeneous ISA and hybrid execution model, and designing a unified bus (UB) to break resource silos, aiming to achieve programming unification and efficient interconnection among CPU, GPU, NPU, etc.

Both SIMT and SIMD architectures have their advantages and disadvantages. SIMT has an advantage in programming flexibility, making it suitable for handling sparse data and complex control flows, while SIMD performs better in hardware efficiency, making it suitable for dense vector operations. The key trade-off between the two lies in the design of the software-hardware delivery interface. The ecosystem and user habits significantly impact technology promotion, as evidenced by NVIDIA’s two key compromise cases, which illustrate that “revolutionary designs” that deviate from industry standards often struggle to succeed.

In summary: whether it is NPU or GPGPU, the core difference between the two architectures lies in the balance between programming flexibility and hardware efficiency. Therefore, it was correct for domestic giants to initially choose NPU, and it is also correct to start developing GPGPU now; there is no claim that the initial strategic direction was wrong.

Next, let’s provide some detailed explanations about NPU and GPU.

NPU is a processor specifically designed for artificial intelligence computing, primarily used for efficiently executing operations related to neural networks. Compared to traditional CPUs and GPUs, NPUs have advantages in energy efficiency and computing speed, making them particularly suitable for mobile devices, edge computing, and embedded AI scenarios. In recent years, NPUs have also been applied to the training and inference of large AI models, and there are already many products on the market that use NPU architecture as hardware to serve AI network inference.

These companies favor NPUs mainly because NPU chip design is controllable and reliable. Its hardware instruction architecture is relatively simple, primarily involving data movement and computation. Compared to some complex instructions on GPUs, all ISA instructions related to NPU can fit into an extended instruction set of ARM or RISC-V. Moreover, with dense computing arrays, NPUs have low power consumption, and the single and repeated use of large memory reduces sensitivity to high bandwidth. These factors combined make it the best choice for domestic companies in recent years to address the “bottleneck” problem in chips—being able to define the instruction set framework independently, achieving performance comparable to GPUs, coping with relevant export control restrictions, and having a lower hardware threshold than traditional CPUs and GPUs.

As the chairman of Loongson Technology mentioned in a recent interview, from the perspective of design complexity, general-purpose CPUs are the most difficult, followed by graphics GPUs, while AI-used NPUs are relatively easier. If the complexity of CPUs is 1, then the complexity of graphics GPUs is about 0.3-0.4, while the complexity of AI chips is less than 0.1; they primarily involve matrix multiplication and convolution operations, which require stacking materials.

Thus, the more materials stacked, the greater the computing power. Therefore, it is meaningless to compare the computing power of AI chips alone. NVIDIA’s chip computing power is also referred to as Jensen math, a unique calculation method by Jensen Huang, primarily for product promotion.

The first rule of Jensen math is that the floating-point operations per second (FLOPs) announced by NVIDIA are calculated based on a sparsity of 2:4 (which no one uses) rather than based on dense floating-point operations (which is the performance metric in the real world). In other words, the 989.4 trillion FP16 (half-precision floating-point) operations of the H100 are claimed to be 1979.8 trillion floating-point operations.

Returning to NPU, the biggest problem with NPUs is that the software complexity far exceeds the hardware. This is why we believe that simply comparing computing power parameters is meaningless; the usability of the software is crucial. Currently, domestic restrictions on data center construction are due to low-end computing processes, while high-end computing is in short supply. The so-called low-end computing involves a large number of domestic GPU cards deployed, but due to poor software usability or hardware instability, these data centers have been unable to operate normally or have a relatively low utilization rate.

In the multi-level memory architecture of NPU, the L1 cache blocks are usually limited in size and have storage conflict issues. To fully utilize performance on NPU, engineers must accurately partition data to avoid address conflicts while maximizing the limited cache space. More troublesome is that NPUs lack the latency hiding mechanisms found in the SIMT (Single Instruction Multiple Threads) architecture of GPUs, exposing memory access latency directly, which can lead to the computing core “idling” if not handled carefully.

Another pain point is the fragmented ecological barrier. As an emerging DSA, NPUs have not yet undergone the “brutal convergence” that GPUs experienced—different manufacturers’ architectures have undergone generational changes, and even products of the same generation have vastly different designs. This means that software optimized for a specific NPU is difficult to port directly to another product, forcing developers to invest repeatedly for each type of hardware, significantly increasing the cost of application deployment.

Early graphics cards were quite similar to today’s NPUs: both were “dedicated calculators” remotely controlled by CPUs, with simple structures and single functions. However, as graphics tasks became increasingly complex, graphics cards gradually integrated independent control units, ultimately evolving into today’s GPUs. This evolution was not accidental: the physical distance of the bus determined that CPUs could not control remote devices in real-time, necessitating the delegation of some control logic to meet the real-time demands of complex tasks.

NVIDIA’s breakthrough was not only equipping GPUs with control units but also abstracting the SIMT programming model—allowing developers to handle parallel tasks as if writing single-threaded programs, significantly lowering the barrier to parallel computing. In contrast, NPUs still lack powerful control units, resembling a “calculator that passively executes instructions,” with all complex scheduling relying on external CPUs. This is also the root of their struggles in training large AI models—during training, the rapidly changing task flow requires real-time adjustments to computing strategies, and NPUs without autonomous control capabilities cannot cope.

From an architectural perspective, the hierarchical differences between GPUs and NPUs are even more pronounced: GPUs have a three-level control chain of “CPU→GPU→Tensor Core,” while NPUs resemble a two-level structure of “CPU→NPU.”

Today, AI tasks are transitioning from single inference to a full process of “training + inference,” moving from fixed scenarios to complex and variable general intelligence. This compels chip architectures to evolve towards “efficient computing + flexible control.”

For domestic AI chip manufacturers, this path has gradually become clear: introducing SIMT front-end in NPUs, strengthening control units, and moving closer to GPU architecture. For GPUs, many internal modules also need to be DSA-ified.

The rise of NPU is an inevitable result of the specialization of AI computing power; its approach towards GPUs reflects the principle of “separation leads to convergence” in technological evolution. In this computing power revolution, there are no eternal “specialists,” only versatile “all-rounders” that continuously adapt to changes—this may be the most profound survival logic in the AI chip race.