The NPU in the PC industry actually started earlier than most people realize.
On September 21, 2023, Microsoft released a convertible laptop called the Surface Laptop Studio 2. From a product positioning perspective, it is clearly not aimed at the “mass market,” but this product will hold a special place in the history of PCs as it is the first mass-produced “AI PC” equipped with an NPU.
Of course, everyone knows that it wasn’t until three months later that Intel would launch the first generation of Meteor Lake family processors with built-in NPUs. Therefore, the NPU in the Surface Laptop Studio 2 is not integrated into the processor but uses an external Movidius 3700VC solution based on the PCIe channel, providing approximately 7 TOPs of AI computing power.
We also have a product that uses the Movidius solution, but it is an earlier model.
Moreover, the “Movidius 3700VC” can actually be seen as the 12nm prototype of Intel’s own NPU. This is because the integrated “NPU3” in Meteor Lake and Arrow Lake has the internal model “Movidius 3720,” which is essentially an enhanced and overclocked version of the former.
Why is Intel doing this? Clearly, they hope to achieve continuity in their NPU offerings to avoid scaring developers away with frequent updates and incompatible architectures for this “new gadget.” However, the pursuit of architectural continuity objectively limits the performance ceiling of the “NPU3.”
As a result, Intel launched the “NPU4” with 48 TOPs on Lunar Lake, and it will be updated to the “NPU5” with 50 TOPs on Panther Lake. According to current leaks, we may see the NovaLake processor equipped with a 74 TOPs “NPU6” as early as next year.
Why can’t NPU be popularized? There are currently two mainstream viewpoints.
It is clear that the computing power of NPUs is rapidly expanding, but has it been popularized and accepted by ordinary consumers?
The answer is actually no. On one hand, in today’s mainstream desktop PC market, processors without NPUs still occupy the majority market share, such as the Ryzen 9 9950X3D, Ryzen Threadripper, Core i7 251E, and Xeon W9, Xeon 6, which are high-end processors for either home or professional use, and they do not even have built-in NPUs.
People dislikeWindows Recall which is essentially a “human nature issue.”
On the other hand, even in devices with high-performance built-in NPUs, the “benefits” that users can currently perceive seem very limited. For example, on one of our commonly used “Snapdragon laptops,” while the NPU can indeed be used for camera blurring, microphone noise reduction, and assisting the system in real-time recording of operation history (Windows Recall), beyond that, it does not provide any “acceleration” for the third-party software that is heavily used daily.
Why has it been over two years since the NPU appeared, yet it has not truly popularized and played an important role in the daily use of most users?
In fact, the two examples mentioned earlier reflect two different perspectives on the current industry regarding the difficulty of NPU popularization. Some believe that the advantage of NPU lies purely in “high energy efficiency,” meaning that when processing AI tasks, NPUs are inherently slower than high-performance graphics cards, and their advantage lies only in low power consumption. Therefore, NPUs are only suitable for devices like laptops with limited power supply and cooling conditions, and should not appear in desktop computers (i.e., believing that desktops have powerful CPUs and discrete graphics cards, and their AI ecosystem should be built on these two).
Another viewpoint is that NPUs are currently not favored simply because their computing power is not high enough. As long as they iterate a few more times and their computing power increases, “people will be willing to use them.”
A Rare Device Reveals the Truth About NPU
In fact, we at San Yi Life previously agreed with the above two ideas, but until recently we saw a product that could overturn these notions.
This is a laptop model called Dell Pro Max 16 Plus. So what is special about it? Simply put, this laptop can be equipped with a standalone NPU card from Qualcomm, model “AI-100 Ultra.”
According to product information, this NPU card will occupy the graphics card installation slot in the laptop, has a power consumption of up to 150W, comes with 128GB of LPDDR5X “video memory,” and can provide an AI computing power level of 870 TOPs in INT8 format.
Clearly, this is not a configuration intended for “mass consumers.” Reports indicate that it is more aimed at AI developers and those who need sufficient high-end edge computing power in non-networked scenarios like factories.
Even so, this “standalone NPU” has refreshed our understanding. On one hand, it seems to indicate that the current common NPU solutions, such as those integrated into processors with 15 to 50 TOPs of computing power, are indeed too low in performance to be supported in truly professional software.
On the other hand, a closer look at the parameters of this “standalone NPU” reveals that a power consumption of 150W can only provide 870 TOPs of computing power, which does not seem to have a particularly large advantage.
For example, the RTX 5090 on a laptop also has a power consumption of 150W but can achieve AI computing power of 912 TOPs; while the RTX PRO 6000 graphics card on a desktop can achieve up to 4000 TOPs of AI computing power with a power consumption of 600W. This means that if we really want to calculate the “energy efficiency ratio,” the AI energy efficiency ratio of high-performance discrete graphics cards is not necessarily lower than that of the various integrated NPUs seen today.
Of course, we are not saying that NPUs in modern PCs are necessarily more power-hungry than discrete graphics cards, as most integrated NPUs are small and low in computing power, so comparing “absolute power consumption” will naturally lead to the conclusion that integrated NPUs are more power-efficient than discrete graphics cards.
However, compared to ordinary users, developers are clearly not so easily fooled. Therefore, when a high-power, not particularly efficient “large NPU” appears on computers intended for developers, it actually constitutes a mockery of the current “consumer NPU.” It makes us realize that the vast majority of current NPUs have insufficient absolute performance, making them unable to be truly adopted by third-party software. At the same time, its existence also suggests that the so-called “high efficiency of NPU, and that NPUs are more power-efficient than graphics cards” seems worth re-evaluating.
[Some images in this article are sourced from the internet]