The NPU (Neural Network Processing Unit, 神经网络处理器) is a dedicated chip designed for artificial intelligence computations, with its core advantage being the efficient execution of neural network-related operations (such as matrix multiplication, convolution operations, activation function calculations, etc.). Compared to traditional CPUs and GPUs, NPUs perform better in terms of energy efficiency and computational speed, especially suited for mobile devices, edge computing, and embedded AI scenarios.
In recent years, NPUs have also been attempted in the training and inference of large AI models, with many products on the market adopting NPU architecture to provide hardware support for AI network inference. The core reason for the preference for NPUs by enterprises lies in their controllability and reliability in chip design. However, as AI technology evolves, the limitations of NPUs are gradually becoming apparent, while GPGPU (General-Purpose computing on Graphics Processing Units, 通用图形处理器) is considered a more suitable solution for future demands. Recently, there have even been reports that a domestic manufacturer is restructuring its AI chip from NPU to GPGPU.
01Differences Between NPU and GPGPU
GPUs were originally created to accelerate graphics rendering, focusing on processing parallel tasks in computer graphics — such as vertex transformations, lighting calculations, texture mapping, etc. As technology has evolved, their powerful parallel computing capabilities have extended into general computing, forming GPGPU, which is widely used in scientific computing, deep learning training, video encoding and decoding, and other scenarios.
NPUs, on the other hand, are specialized chips designed specifically for artificial intelligence (AI) and machine learning (ML) tasks, particularly for neural network inference and training in deep learning. Their architecture is deeply optimized for core tasks such as matrix operations, convolution operations, and activation functions (like ReLU, Softmax, Pooling), efficiently executing matrix multiplication and convolution operations through large-scale multiply-accumulate (MAC) arrays; at the same time, they optimize data flow transmission, reducing the data interaction overhead between memory and computing units, significantly improving energy efficiency. However, the instruction set and hardware design of NPUs are highly specialized, only suitable for specific neural network models, with limited support for other tasks; and their ecosystem is relatively new, relying on specialized SDKs and toolchains provided by chip manufacturers (such as Huawei’s Da Vinci architecture, Google’s Edge TPU toolchain), requiring developers to optimize for specific architectures, resulting in lower flexibility but higher deployment efficiency.
From the perspective of architectural core differences, GPGPU adopts a SIMT (Single Instruction Multiple Threads) front end (essentially encapsulating a SIMD front end), achieving high performance with a general scheduling mechanism and user-friendly programming interface, allowing for “out-of-the-box” usability; while NPU/TPU still follows the traditional SIMD (Single Instruction Multiple Data) architecture, requiring manual pipeline orchestration, with latency hiding efficiency far inferior to SIMT, making it difficult to write high-performance kernels, resulting in low efficiency, and lacking usability, with its ecosystem lagging far behind GPGPU. (SIMT is the core parallel computing model in GPU architecture, driving multi-threaded parallel execution through a single instruction stream, with each thread processing an independent data set. Its core mechanism is to group threads into thread warps of 32 threads, achieving branch logic processing through lock-step execution and dynamic masking control)
Therefore, in recent years, some NPU chips have begun to introduce SIMT front ends (still leaning towards heterogeneous pipelines). Additionally, while GPGPU’s control units are quite weak, these control units still possess optimization designs for parallel computing — such as hardware thread switching, memory access control, and latency hiding; whereas NPUs have almost no control units, making it somewhat forced to classify them as “processors (PU)” rather than “computers/co-processors.”
According to the traditional “processor-co-processor” architectural framework analysis, the positioning of the two can be summarized as follows: the typical architecture of NPU is “CPU+NPU”; while GPGPU is “CPU+GPU+DSA (Domain-Specific Architecture)”, taking NVIDIA’s implementation as an example, which is “CPU+CUDA Core+Tensor Core”. It should be noted that here, the GPU and DSA are not parallel co-processors, but form a hierarchical relationship where the CPU controls the GPU, and the GPU then controls the DSA.
02Why Do We Need GPGPU?
So, why do we need GPUs to control domain-specific architectures (DSA)? Or more fundamentally, why do GPUs need control units?
In fact, early graphics cards, and today’s NPUs, are structurally quite similar aside from their different tasks. However, unlike floating-point co-processors that gradually integrated into CPUs and eventually became part of the CPU, graphics cards have existed independently since their inception. The physical distance limitation of the bus makes it difficult for CPUs to control graphics cards in real-time; as the complexity of tasks increases, integrating some real-time control logic into the graphics card itself has become an inevitable choice — this is a classic “control transfer” strategy in hardware design. NVIDIA not only built such control units but also innovatively abstracted the SIMT programming model, a breakthrough that is a milestone in the field of parallel computing.
From the practical experiences in the AI chip field in recent years, as tasks become increasingly complex and variable, the “expansion” of control units has become an objective trend — only with powerful control units can we adapt to the ever-changing application scenarios. Although SIMT is no longer the optimal programming model for AI chips, AI computation essentially still belongs to the realm of parallel computing, inevitably facing common issues of parallel computing. At this point, people realize that NVIDIA’s GPU design, which seems to merely “patch” the tensor computing core onto the existing architecture, lacks disruptive innovation, but its actual application effects far exceed expectations, hiding many valuable insights.
The importance of GPGPU stems from researchers’ inability to predict the evolution path of AI technology in the next 5-10 years. If choosing NPU, the adaptation pressure must be entirely transferred to the software layer — although such hardware design may be the optimal solution for a specific stage, the side effects cannot be ignored. It is precisely for this reason that, starting from the demands of training scenarios, the next generation of NPUs introducing SIMT front ends is a logical choice.
Objectively speaking, NPUs perform reasonably well in inference scenarios and are a good option; however, in training scenarios, their path is exceptionally challenging — the difficulty is at least an order of magnitude higher than inference. Even if NPUs have advantages in performance-to-power ratio, they have not achieved a crushing breakthrough over GPUs in actual performance; moreover, the sensitivity of cloud scenarios to power consumption is inherently limited.
In the actual implementation of NPUs, there are also the following issues:
Firstly, usability lags behind GPUs. Just as PyTorch defeated TensorFlow, which is more conducive to performance optimization due to its static graph characteristics, due to its usability and development experience, developers are particularly sensitive to the friendliness of toolchains, caring more about usability and experience. Specifically, NPUs are limited by simple address control modules, only able to handle access blocking of different instructions to the same address, lacking the latency hiding mechanisms for threads found in SIMT architectures; to fully unleash performance on NPUs, developers must deeply understand their multi-level memory mechanisms and design targeted partitioning schemes — but the L1 memory block is limited by capacity and storage conflicts, making it extremely difficult to control partitioning granularity, further increasing the usage threshold.
Furthermore, DSA like NPU has not undergone a rigorous convergence phase. Currently, each company’s NPU chip architecture undergoes significant generational changes, let alone the greater differences in NPU architectures provided by different companies.
Secondly, the ecological barrier is high. Domestic GPUs, if directly compatible with the CUDA ecosystem, require no rewriting of operators and inference engine frameworks, only needing recompilation for reuse, significantly reducing migration costs; however, investing in NPUs incurs enormous costs on the software side — developers often need to work overtime continuously to adapt operators, optimize performance, and build deep learning engine frameworks. Except for a few companies with strong capabilities (even requiring on-site support from chip manufacturers), most companies find it difficult to bear such costs. For customers, unless there is pressure of “completely unable to buy GPUs,” they will obviously not choose to invest far more time and effort in NPUs than in GPUs.
Thirdly, the development difficulty is high and the closed nature restricts the growth of downstream developers. In the GPU ecosystem, developers can independently develop operators and perform performance optimizations, with relatively low thresholds; but in the NPU system, except for a few large manufacturers, most developers can only rely on SDKs and solutions provided by manufacturers, making it difficult to conduct deep optimizations independently.
03Domestic NPU Companies May Introduce SIMT Front Ends, Shifting Towards GPGPU Routes
The GPGPU’s SIMT model was originally designed to efficiently handle large-scale parallel and data-intensive tasks, particularly suitable for graphics rendering and scientific computing. This model allows a group of threads (i.e., a warp) to synchronously execute the same instruction, while each thread can independently operate on different data — this feature significantly enhances efficiency when processing data sets containing a large number of homogeneous operations. For complex tasks, especially those that can be highly parallelized, have low data correlation, and exhibit converging branching behavior, the advantages of the SIMT model are even more pronounced.
From a hardware perspective, NPUs indeed possess many advantages. This also explains why the number of tensor computing cores on GPGPU has continuously increased in recent years, even becoming a core metric for measuring performance. It can be said that NPUs are architectures born for AI, but compared to the environment when GPUs were born, the growth path of domain-specific architectures (DSA) represented by NPUs is more arduous: on one hand, they face the growing pains of their immature architecture, and on the other hand, they must endure the absorption of their technical characteristics and market encirclement by GPUs, making development difficulties self-evident.
For scenarios like large language models (LLM) — which involve a large number of matrix multiplication (GEMM) calculations and require continuous responses to new demands — to lower the software development threshold, either the SIMT model must be adopted, or reliance on out-of-order execution and data prefetching mechanisms is necessary. The architectural design of NPUs is limited by the technological vision of the time, making it difficult to predict future trends, so it is hard to simply determine that the early choice of the NPU route was a strategic mistake. However, the technology route of SIMT + tensor computing cores, due to its compatibility with CUDA’s API and source code, has become the mainstream choice for current NVIDIA competitors, and its advantages are self-evident.
Many domestic companies’ NPU development began in edge scenarios, initially focusing more on power consumption optimization, and at that time, the types of AI algorithms were limited; meanwhile, the NPU architecture was relatively insensitive to bandwidth, theoretically alleviating key issues related to HBM (High Bandwidth Memory). However, the key point is that these AI chip companies did not fully foresee the explosive growth of large AI models in the early stages — a combination of factors has led to a mismatch between the early architectural design direction and the later surging demand for AI computing power. A core issue in the domestic AI chip field is the decision-makers’ insufficient emphasis on programmability and generality. Excessive resources have been invested in optimizing specific scenarios, which may present impressive performance in promotional data, but once expanded to broader application scenarios, their average performance appears mediocre.
Returning to the essence, the core logic of GPGPU and the SIMT programming model lies in meeting the demands of large-scale parallel computing while ensuring the Turing completeness of the chip. The problem with DSA lies precisely in the neglect of usability, which now must be remedied. It is worth noting that as GPUs introduce tensor cores and DMA (Direct Memory Access) mechanisms, their native SIMT model has made breakthroughs; meanwhile, NPUs are also gradually strengthening their control capabilities — both are showing a trend of “mutual advancement” in technological evolution.
However, there are reports that domestic companies will also introduce SIMT front ends into NPUs, gradually addressing the shortcomings in general scheduling and programming usability.
