The secret is that those who follow us are industry leaders and elites.

Author: Peng Cheng

Source: Semiconductor Industry Overview

The NPU (Neural Network Processing Unit, a dedicated chip designed for artificial intelligence computing, has a core advantage in efficiently executing neural network-related operations (such as matrix multiplication, convolution operations, activation function calculations, etc.). Compared to traditional CPUs and GPUs, NPUs perform better in terms of energy efficiency and computational speed, especially suited for mobile devices, edge computing, and embedded AI scenarios.

In recent years, NPUs have also been attempted in the training and inference of large AI models, with many products on the market adopting NPU architecture to provide hardware support for AI network inference. The core reason for the preference for NPUs by enterprises lies in their controllability and reliability in chip design. However, as AI technology evolves, the limitations of NPUs have gradually become apparent, while GPGPU (General-Purpose computing on Graphics Processing Units) is considered a more suitable solution for future needs. Recently, there have even been reports that a domestic manufacturer is restructuring its AI chip from NPU to GPGPU.

01The Differences Between NPU and GPGPU

GPUs were originally created to accelerate graphics rendering, focusing on processing parallel tasks in computer graphics — such as vertex transformation, lighting calculations, texture mapping, etc. As technology has evolved, their powerful parallel computing capabilities have extended into general computing, forming GPGPU, widely used in scientific computing, deep learning training, video encoding and decoding, and other scenarios.

NPUs, on the other hand, are specialized chips designed specifically for artificial intelligence (AI) and machine learning (ML) tasks, particularly for neural network inference and training in deep learning. Their architecture is deeply optimized for core tasks such as matrix operations, convolution operations, and activation functions (like ReLU, Softmax, Pooling), efficiently executing matrix multiplication and convolution operations through large-scale multiply-accumulate (MAC) arrays; at the same time, they optimize data flow transmission, reducing the data interaction overhead between memory and computing units, significantly improving energy efficiency. However, the instruction set and hardware design of NPUs are highly specialized, only suitable for specific neural network models, with limited support for other tasks; and their ecosystem is relatively new, relying on specialized SDKs and toolchains provided by chip manufacturers (such as Huawei’s Da Vinci architecture, Google’s Edge TPU toolchain), requiring developers to optimize for specific architectures, resulting in lower flexibility but higher deployment efficiency.

From the perspective of architectural core differences, GPGPU adopts a SIMT (Single Instruction Multiple Threads) front end (essentially encapsulating a SIMD front end), achieving high performance with a “plug-and-play” approach through a general scheduling mechanism and friendly programming interface; while NPU/TPU still follows the traditional SIMD (Single Instruction Multiple Data) architecture, requiring manual pipeline orchestration, with latency hiding efficiency far inferior to SIMT, making it difficult to write high-performance kernels, resulting in low efficiency, and both usability and ecosystem maturity lagging far behind GPGPU. (SIMT is the core parallel computing model in GPU architecture, driving multi-threaded parallel execution through a single instruction stream, with each thread processing an independent data set. Its core mechanism is to group threads into thread warps of 32 threads, achieving branch logic processing through lock-step execution and dynamic masking control)

Therefore, in recent years, some NPU chips have begun to introduce SIMT front ends (still leaning towards heterogeneous pipelines). Additionally, GPGPU, although its control units are weak, still possesses optimization designs for parallel computing — such as hardware thread switching, memory access control, and latency hiding; while NPUs have almost no control units, making it somewhat forced to classify them as “processors (PU)” rather than “calculators/co-processors”.

According to the traditional “processor-co-processor” architecture framework analysis, the positioning of the two can be summarized as: the typical architecture of NPU is “CPU+NPU”; while GPGPU is “CPU+GPU+DSA (Domain-Specific Architecture)”, taking NVIDIA’s implementation as an example, which is “CPU+CUDA Core+Tensor Core”. It should be noted that here, the GPU and DSA are not parallel co-processors, but form a hierarchical relationship where “the CPU controls the GPU, and the GPU controls the DSA”.

02Why Do We Need GPGPU?

So, why do we need GPUs to control domain-specific architectures (DSA)? Or more fundamentally, why do GPUs need control units?

In fact, early graphics cards, apart from their different tasks, were structurally quite similar to today’s NPUs. However, unlike floating-point co-processors that gradually integrated into CPUs and eventually became part of the CPU, graphics cards have existed independently since their inception. The physical distance limitation of the bus made it difficult for CPUs to control graphics cards in real-time; as the complexity of tasks increased, integrating some real-time control logic into the graphics card became an inevitable choice — this is a classic “control transfer” strategy in hardware design. NVIDIA not only built such control units but also innovatively abstracted the SIMT programming model, a breakthrough that is a milestone in the field of parallel computing.

From the practical experience in the AI chip field in recent years, as tasks become increasingly complex and variable, the “expansion” of control units has become an objective trend — only with powerful control units can they adapt to the ever-changing application scenarios. Although SIMT is no longer the optimal programming model for AI chips, AI computing essentially still belongs to the category of parallel computing, inevitably facing common issues of parallel computing. At this point, people realize that NVIDIA’s GPU design seems to merely “patch” the tensor computing core onto the existing architecture, lacking disruptive innovation, but its actual application effects far exceed expectations, hiding many valuable insights.

GPGPU’s importance stems from researchers’ inability to predict the evolution path of AI technology in the next 5-10 years. If choosing NPU, the adaptation pressure must be completely transferred to the software layer — although this hardware design may be the optimal solution for a specific stage, the side effects cannot be ignored. It is precisely for this reason that, starting from the needs of training scenarios, the next generation of NPUs introducing SIMT front ends is a logical choice.

Objectively speaking, NPUs perform reasonably well in inference scenarios and are a good option; however, in training scenarios, their path is exceptionally difficult — at least an order of magnitude more challenging than inference. Even if NPUs have advantages in performance-to-power ratio, they have not achieved a crushing breakthrough over GPUs in actual performance; moreover, the sensitivity of cloud scenarios to power consumption is inherently limited.

In the actual implementation of NPUs, there are also the following issues:

First, usability lags behind GPUs. Just as PyTorch defeated TensorFlow, which is more conducive to performance optimization due to its static graph characteristics, due to its usability and development experience, developers are particularly sensitive to the friendliness of toolchains, caring more about usability and experience. Specifically, NPUs are limited by simple address control modules, only able to handle access blocking of different instructions to the same address, lacking the latency hiding mechanisms for threads in SIMT architecture; to fully unleash performance on NPUs, developers must deeply understand their multi-level memory mechanisms and design targeted partitioning schemes — but L1 memory blocks are limited by capacity and storage conflicts, making it extremely difficult to control partitioning granularity, further increasing the usage threshold.Furthermore, DSA like NPU has not undergone a brutal convergence phase. Currently, each company’s NPU chip architecture has significant generational changes, let alone the greater differences in NPU architectures provided by different companies.

Second, high ecological barriers. Domestic GPUs can directly be compatible with the CUDA ecosystem, where operators and inference engine frameworks do not need to be rewritten, only needing to be recompiled for reuse, significantly reducing migration costs; however, investing in NPUs requires a huge cost on the software side — developers often need to work overtime continuously to adapt operators, optimize performance, and build deep learning engine frameworks. Except for a few companies with strong capabilities (even requiring on-site support from chip manufacturers), most companies cannot bear such costs. For customers, unless there is pressure of “completely unable to buy GPUs,” they will obviously not choose to invest far more time and energy in NPUs than in GPUs.

Third, high development difficulty and strong closure restrict the growth of downstream developers. In the GPU ecosystem, developers can independently develop operators and perform performance optimizations, with relatively low thresholds; but in the NPU system, except for a few large manufacturers, most developers can only rely on SDKs and solutions provided by manufacturers, making it difficult to carry out deep optimizations independently.

03Domestic NPU Companies May Introduce SIMT Front Ends and Shift to GPGPU Routes

The SIMT model of GPGPU is designed to efficiently handle large-scale parallel and data-intensive tasks, particularly suitable for graphics rendering and scientific computing. This model allows a group of threads (i.e., a warp) to execute the same instruction synchronously, while each thread can independently operate on different data — this feature significantly enhances efficiency when processing data sets containing a large number of homogeneous operations. For complex tasks, especially those that can be highly parallelized, have low data correlation, and exhibit converging branching behavior, the advantages of the SIMT model are even more pronounced.

From a hardware perspective,NPU indeed has many merits. This also explains why the number of tensor computing cores on GPGPU has been continuously increasing in recent years, even becoming a core indicator of performance. It can be said that NPU is an architecture born for AI, but compared to the environment when GPUs were born, the growth path of domain-specific architectures (DSA) represented by NPUs is much more difficult: on one hand, they face the growing pains of their immature architecture, and on the other hand, they must endure the absorption of their technical characteristics by GPUs and market encirclement, making development challenges evident.

For scenarios like large language models (LLM) — which involve a large number of matrix multiplication (GEMM) calculations and require continuous responses to new demands — to lower the software development threshold, either the SIMT model must be adopted, or reliance on out-of-order execution and data prefetching mechanisms is necessary. The architectural design of NPUs is limited by the technological vision of the time, making it difficult to predict future trends, so it is hard to simply determine that the early choice of the NPU route was a strategic mistake. However, the technology route of SIMT + tensor computing cores, due to its compatibility with CUDA’s API and source code, has become the mainstream choice for current NVIDIA competitors, and its advantages are self-evident.

Many domestic companies’ NPU development began in edge scenarios, initially focusing more on power consumption optimization, and at that time, the types of AI algorithms were limited; meanwhile, the NPU architecture was relatively insensitive to bandwidth, theoretically alleviating key issues related to HBM (High Bandwidth Memory). However, the key point is that these AI chip companies did not fully foresee the explosive growth of large AI models in the early stages — a combination of factors led to a mismatch between the early architectural design direction and the later surging demand for AI computing power. A core issue in the domestic AI chip field is the decision-makers’ insufficient emphasis on programmability and generality. Excessive resources invested in optimizing specific scenarios may yield impressive performance in promotional data, but once expanded to broader application scenarios, their average performance appears mediocre.

Returning to the essence, the core logic of GPGPU and the SIMT programming model lies in meeting the demands of large-scale parallel computing while ensuring the Turing completeness of the chip. The problem with DSA lies precisely in the neglect of usability, which now necessitates a return to remedial measures. It is worth noting that as GPUs introduce tensor cores and DMA (Direct Memory Access) mechanisms, their native SIMT model has made breakthroughs; meanwhile, NPUs are also gradually strengthening their control capabilities — both are showing a trend of “mutual advancement” in technological evolution.

However, it has been reported that domestic companies will also introduce SIMT front ends into NPUs in the future, gradually addressing the shortcomings in general scheduling and programming usability.

*Disclaimer: This article is original by the author. The content of the article reflects the author’s personal views, and Xing Wuxian Capital reprints it only to convey a different perspective, not representing agreement or support for that view. If there are any objections, please contact us.

Recommended Reading

How Investors Are Made

Liu Guanghao of Binfu Capital: Investment Thoughts on China’s Smart Electric Vehicle Industry Chain

Liu Chuanwen of Xinjun Capital: Investment Logic of Energy Equipment in Surgery

Zheng Junyan of Hongxin Capital: Current Situation and Outlook of China’s Semiconductor Industry

Liu Huarui of Hechuang Capital: Focus on Semiconductor Investment in Downstream Markets and Industrial Chain Investment Opportunities

Shi Chenxing of Lenovo Venture Capital: Analyzing the Robotics Track, What is the Core Proposition for Entrepreneurs?

Zhong Donglin of Eagle Fund: The Tuition I Paid for Investing in Hard Technology Over the Years

Zhang Yang of Qingyuan Investment: Investment Logic Behind Early New Material Projects

Qin Yi of Xing Tuo Capital: Co-entrepreneurship with Enterprises, Walking with Excellence

Summer Jiazh of Sanze Venture Capital: New Perspectives on Investment in Detection and Medical Devices

Ni Zhigang of Tenggu Capital: Upgrading Industrial “Intelligence” to Make Industries Stronger

Fan Xuesong of Delian Capital: Delian Capital’s Investment Layout in Autonomous Driving

Wang Haining of Longmen Capital: Professional Medicine, Focusing on Innovation

Deng Lijun of Junsheng Investment: The Third Generation Semiconductor Welcomes Favorable Timing, Location, and Human Harmony

Yang Zhilong of Zhejiang Merchants Venture Capital: Knowing and Acting as One, Being a True Value Investor

Feng Bo of Changlei Capital: Software Service Enterprises at the Application Layer Will Welcome a Golden Period

Li Hao of Junsheng Investment: AR May Become the Next Generation Collaborative Computing Platform

Shen Yangguang of Ruisheng Capital: Where Will the Real Economy Go Under the Pandemic?

Yin Jie of Yuansheng Venture Capital: The Latest Trends and Logic of Early Investment in Medical Devices in China and the US

Gao Yun of Zhongqi Capital: Sharing Investment Logic in New Energy Batteries

Zhang Yunpeng of Qingcheng Capital: Investment Insights in the Laser Manufacturing Field

Zhang Wenjun of Tongchuang Weiye: The Industrial Internet Welcomes Huge Opportunities

Zhong Donglin of Langmafeng Venture Capital: Venture Capital Funds Help Chinese Chip Companies Break Through

Zhang Jie of Hongtai Capital: Post-investment Management is More Important Than You Think!

Zhang Yang of Qingyuan Investment: OLED Equipment Opens New Investment Opportunities in the Industry

Zhen Canming: AI Industry Applications, Finance and Retail Have More Investment Value

Wang Jin: Structural Investment Opportunities in the Automotive Industry

Peng Cheng: Investment Trends in Artificial Intelligence

Wang Haiquan: A Detailed Explanation of the Industrial Internet of Things from an Investment Perspective

Zhang Dan: Opportunities and Risks in the Biopharmaceutical Industry

Liu Lizhe: The Trend is Inevitable, Domestic IC Design Companies Have Great Potential in the Future

Liao Yucheng: What Industrial Revolutions Will the 5G Trillion Market Bring?

Wu Yuanji: Investment Logic and Opportunities in the Semiconductor Field

Wang Mu: The Cold Winter Has Arrived, Practice Internal Skills. The AR Industry Will Welcome an Explosion

Liu Zhengnan: AI, A Sword Empowering the Rise of Traditional Financial Industries

Chen Hongwu: AI Should Focus on the Essence of Business and Deeply Integrate with Scenarios

Zhang Yifei: Investment Logic and Opportunities in the Autonomous Driving Field in China

Guo Jia: The Landscape and Boundaries of New Drug Research and Development in China

Xing Wuxian Capital

ID:xingwuxian8 There is no difficult finance in the world, everything is in the Xing Wuxian Financial Community.Long press the QR code to follow