NPU and heterogeneous computing take the lead?
Author| Zhou Ya
Image| Midjourney
Today’s technological era resembles the .com world over 20 years ago. When the internet emerged, there were voices proclaiming that “every computer would connect to the internet,” and now, the same voices are emerging around personal computers (PC), with the keyword being AI.
Since last year, almost all major PC manufacturers, including Lenovo, Dell, HP, and Microsoft, have suddenly launched or are about to launch AI PC products; discussions in the industry have also intensified. For example, NVIDIA CEO Jensen Huang once stated “The PC industry is undergoing a rebirth, and AI PCs will replace traditional PCs in the next decade”, while Dell’s VP of commercial computers and software Meghana Patwardhan bluntly stated — “Every PC will evolve into an AI PC in the future”.
However, what exactly is an AI PC?
01
Talking About AI PCs
AI PC, as the name suggests, is a personal computer integrated with AI. Notably, the IDC “AI PC Industry (China) White Paper” pointed out that an AI PC is a “hybrid” that integrates hybrid AI computing units in hardware and can locally run “personal large models,” create personalized local knowledge bases, and achieve natural language interaction. Unlike existing general-purpose PCs, AI PCs will bring users four major values:1. Personalized services in general scenarios. AI PCs can provide personalized services, including creative services, personal assistant services, and device management services tailored to work, study, and life scenarios.2. Immediate and reliable service responses. In a 2023 IDC survey on user experience with AIGC platforms, “slow response” and “long feedback time” were the main negative feedback from users. AI PCs primarily rely on local inference, supplemented by edge and cloud inference, intelligently and reasonably allocating tasks between hybrid computing and models, effectively reducing response times.3. Lower costs for using large models. Because local inference is supplemented by cloud inference, it reduces the cost for individual users using AI large model services to a certain extent.
4. Trustworthy and secure personal data and privacy protection, etc.
At first glance, it may not seem apparent, but to put it simply, AI PCs can better support large language models and generative AI applications, providing a more intelligent and personalized experience, perhaps allowing PCs to return to their namesake, becoming true “personal computers” that serve everyone.
New concepts ultimately exist to facilitate the landing of new applications. When it comes to generative AI applications, Qualcomm’s Senior Vice President of Product Management Ziad Asghar recently summarized during a media communication meeting, citing Qualcomm’s latest white paper “Unlocking Generative AI on the Terminal Side through NPU and Heterogeneous Computing” that generative AI applications can be divided into three categories:
1. On-demand: Triggered by users and require immediate responses. This includes photo/video shooting, image generation/editing, code generation, recording transcription/summary, and text (emails, documents, etc.) creation/summary. For example, a user generates a meeting summary on a PC or queries the nearest gas station by voice while driving.2. Continuous: Run for a longer duration. This includes voice recognition, super-resolution for games and videos, audio/video processing for video calls, and real-time translation. For example, using a mobile phone for real-time conversation translation or running super-resolution frame by frame while playing games on a PC.
3. Ubiquitous: Runs continuously in the background. This includes always-on predictive AI assistants, context-aware AI personalization, and advanced text auto-completion. For instance, a mobile phone automatically suggests a meeting based on conversation content, or a PC adjusts learning materials in real-time based on user test responses.
However, just as large models require concurrent computing power in data centers, these generative AI applications also face two key challenges at the hardware facility level.
First, due to the power consumption and heat dissipation limitations of terminals, general-purpose CPUs and GPUs struggle to meet the demanding and diverse computational needs of these generative AI applications. Second, as these generative AI applications continue to evolve and diversify, deploying them on a single hardware platform is counterintuitive.
In response, “NPU and heterogeneous computing” have become key for hardware manufacturers to specifically address the challenges of generative AI on the terminal side.
02CPU, GPU, NPU, Confused?
It is well known that traditional PCs typically have two processing units:
1. CPU (Central Processing Unit): a large-scale integrated circuit, whose main logical architecture includes the control unit, arithmetic logic unit (ALU), cache, and the bus that connects data, control, and status among them. In simple terms, it consists of computation units, control units, and storage units.
CPUs follow the von Neumann architecture, whose core principle is storing programs/data and executing them sequentially. Therefore, the CPU architecture requires a large amount of space for storage and control units, while the computation unit (ALU) occupies a relatively small portion, limiting the CPU’s capability for large-scale parallel computation, making it more adept at logical control processing.
While CPUs cannot perform large-scale parallel computations, GPUs can.
2. GPU (Graphics Processing Unit): a large-scale parallel computing architecture consisting of numerous computation units, initially separated from CPUs to specifically handle image parallel computation data, designed to simultaneously process multiple parallel computation tasks.
Compared to CPUs, less than 20% of the CPU chip space is occupied by ALUs, while over 80% of the GPU chip space is occupied by ALUs. This means GPUs have more ALUs for data parallel processing.
Consequently, GPUs have many advantages, including multithreading, providing a foundation for multi-core parallel computation, with a very high core count, supporting large-scale data parallel computations and processing neural network data far more efficiently than CPUs; they also have higher memory access speeds and greater floating-point computation capabilities. Thus, GPUs are better suited for deep learning involving large training data, extensive matrices, and convolution operations.
It should be noted that while GPUs excel in parallel computation capabilities, they cannot work independently and require collaboration with CPUs for neural network model construction and data flow transmission.
However, GPUs also have inherent drawbacks, namely high power consumption, large size, and high cost. The higher the performance of a GPU, the larger its size, the higher its power consumption, and the more expensive it becomes, making it unsuitable for smaller devices and mobile devices.
As the saying goes: “To do a good job, one must first sharpen one’s tools.” A stone is a general tool that can be used for many purposes, but if it is carefully polished to make it a sharp tool, can it not be used to cut things?
Thus, a small-sized, low-power, high-performance, and highly efficient ASIC chip called NPU is that polished stone.
NPU (Neural Processing Unit), known as “Neural Processing Unit”, is specifically optimized for matrix operations, addressing the inefficiencies of traditional chips in neural network computations.
Literally speaking, a neural network is a network composed of neurons in your brain, a complex web filled with human intelligence. The NPU mimics the neural networks of the human brain to achieve intelligence.
How does it mimic? The NPU operates by simulating human neurons and synapses at the circuit level, directly processing large-scale neurons and synapses using a deep learning instruction set, where one instruction completes the processing of a group of neurons. Compared to CPUs and GPUs, NPUs integrate storage and computation by emphasizing weights, thus improving operational efficiency.
In summary, NPUs are born to achieve “accelerating AI inference with low power consumption” because they are constructed to mimic biological neural networks. While CPUs and GPUs require thousands of instructions to complete neural processing, NPUs can do so with just one or a few instructions, thus showing significant advantages in the processing efficiency of deep learning.
Especially in light of the highly variable demands of generative AI today and the trend towards edge AI, NPUs excel.
Ziad Asghar pointed out that in the aforementioned “continuous” applications, there is a need to achieve sustained stable peak performance with low power consumption, where NPUs can play their greatest advantage. In use cases based on large language models (LLM) and large vision models (LVM), such as Stable Diffusion or other diffusion models, the performance per watt of NPUs is outstanding.
03The Past and Present of NPU: This N is Not That N
However, the name NPU has not just emerged now. Rather, with the evolution of AI application trends, the connotation of NPU has undergone a series of changes.What we refer to now is the redefined NPU.
About a decade ago, early NPUs were primarily designed for audio and voice AI, used for simple CNNs, mainly requiring scalar and vector mathematical operations.
From 2016 to 2022, as AI became popular in photography and video applications, new models based on Transformers, RNNs, Long Short-Term Memory (LSTM), and higher-dimensional CNNs emerged. These workloads require a large amount of tensor mathematical operations, leading NPUs to add tensor accelerators and convolution accelerators, significantly improving performance while reducing memory bandwidth usage and energy consumption.
As a result, smartphone SoCs have been utilizing NPUs for years to enhance user experience in areas such as imaging, audio, connectivity performance, security, and more.
By the time of the large model era in 2023, beyond computational demands, trade-offs must be made between performance, power consumption, efficiency, programmability, and area, where a dedicated custom-designed NPU can make the right choices.
The Evolution of NPU
For instance, the Hexagon NPU integrated into Qualcomm Snapdragon X Elite achieves 45 TOPS of computing power, processing a 7-billion-parameter Llama 2 model on a PC at a speed of 30 Tokens/s; additionally, Snapdragon 8 Gen 3 benefits from the Hexagon NPU, supporting various generative AI large models including Meta and Llama 2, capable of running large models with over 10 billion parameters on the terminal side, eliminating the need for complete reliance on the cloud.
However, as the AI era continues to evolve, models are becoming increasingly complex and parameter scales are climbing from billions to hundreds of billions, with a growing trend towards multimodality. Generative AI applications also require simultaneous invocation of more models, indicating that while we have praised NPUs many times, it is unrealistic for processors to solely rely on NPUs.
Therefore, everyone is focusing on the same direction — heterogeneous computing.
04Heterogeneous Computing Takes the Lead
“Computing power is the premise for realizing various functions of AI PCs, and terminal heterogeneous hybrid (CPU + NPU + GPU) computing power is an inevitable requirement for the large-scale implementation of AI.” The “AI PC Industry (China) White Paper” points out that heterogeneous hybrid computing utilizes different types of instruction sets and architecture groups of computing units for local computing systems, combining applications of computing devices like CPUs, NPUs, and GPUs to fully leverage the performance of each hardware and provide flexible solutions for different AI workloads.
As previously mentioned, the three types of generative AI — on-demand, continuous, and ubiquitous. The key performance indicator for on-demand applications is latency, as users do not want to wait; when these applications use small models, CPUs are typically the right choice; when models grow larger (for example, with billions of parameters), GPUs and NPUs are often more suitable; while battery life and energy efficiency are critical for continuous and ubiquitous applications, making NPUs the best choice.
“Choosing the right processor for related tasks is crucial, but it is also important to pay attention to the overall workload situation of the SoC.” Ziad Asghar illustrated that if a user is playing a resource-intensive game, the GPU will be fully occupied; if the user is browsing multiple web pages, the CPU may be overly occupied, at which point the NPU, as a true AI dedicated engine, will demonstrate significant advantages, ensuring an excellent experience in AI use cases.
Choosing the right tool from a toolbox depends on many factors
So how can heterogeneous computing take on the challenges of edge AI and generative AI? Let’s take Qualcomm’s approach as an explanation.
Integrating countless capabilities into a single processor, allowing different computing units to perform their respective roles, has always been Qualcomm’s commitment and expertise.
Specifically, the Qualcomm AI Engine includes multiple hardware and software components to accelerate AI on Snapdragon and Qualcomm platforms. In terms of integrated hardware, the Qualcomm AI Engine features a leading heterogeneous computing architecture, including Hexagon NPU, Adreno GPU, Qualcomm Kryo or Qualcomm Oryon CPUs, Qualcomm sensor hub, and memory subsystems, all meticulously designed to work together to run AI applications quickly and efficiently on the terminal side.
So how does it work?
For example, if you want the “virtual assistant” on your PC to create a travel plan for you.
— What you need to do: Tell the virtual assistant to help me plan a one-week trip to Mauritius.
— What you see the AI assistant doing: Providing flight itinerary suggestions and adjusting the itinerary through voice dialogue with the user, finally creating a complete flight schedule through a certain plugin.
— What the AI actually does:
- First, what you say to the virtual assistant is converted to text by an automatic speech recognition (ASR) model called Whisper, which is a model with approximately 240 million parameters released by OpenAI. This model mainly runs on the Qualcomm sensor hub.
- Then, the AI assistant utilizes the Llama 2 or Baichuan large model to generate text responses based on the text content. This model runs on the Hexagon NPU.
- Next, the text is converted into speech using an open-source TTS (Text to Speech) model running on the CPU.
- Simultaneously, the virtual assistant’s rendering must synchronize with the speech output to achieve a sufficiently realistic user interaction interface. Using blendshape technology to match the speech with the virtual avatar’s lip movements achieves voice synchronization. This traditional AI workload runs on the NPU.
- At the same time, the rendering of the virtual assistant is completed using Unreal Engine’s MetaHuman, which is done on the Adreno GPU.
- Finally, network connectivity is established using Snapdragon modem technology, and booking operations are completed through the Skyscanner plugin.
Indeed, the internal structure of a single processor comprises multiple processing units working together to fully unleash the performance of the AI engine. However, Qualcomm believes that this is not enough.
Because any AI terminal, whether it’s an AI PC, AI phone, or IoT device, can only create the best experience when hardware and software are combined.When hardware performance is ready, how do we ensure software keeps pace?
Thus, the Qualcomm AI Hub was born. This is a comprehensive model optimization library launched by Qualcomm during MWC 2024, providing developers with over 75 mainstream models, such as Stable Diffusion, ControlNet, Baichuan-7B, etc. All these models are optimized to fully utilize the core hardware acceleration capabilities of the Qualcomm AI Engine, achieving a fourfold increase in inference speed.
For developers, this allows for quick and seamless integration of large models into applications, shortening time to market. Additionally, these optimized models are also made available on GitHub and Hugging Face, allowing developers to run models on cloud-hosted terminals equipped with Qualcomm and Snapdragon platforms.
More importantly, Qualcomm has built the Qualcomm AI Software Stack on top of all hardware AI capabilities, supporting all mainstream AI frameworks (such as TensorFlow, PyTorch, ONNX, and Keras), and also supporting all mainstream AI runtimes (such as DirectML, TFLite, ONNX Runtime, ExecuTorch), as well as different compilers, mathematical libraries, and other AI tools. Developers can directly couple through the Qualcomm AI Engine Direct SDK to accelerate the development process. Furthermore, the Qualcomm AI Software Stack integrates the Qualcomm neural network processing SDK for inference, with different versions available for Android, Linux, and Windows.
Image of the Qualcomm AI Software Stack
In conclusion, facing the wave of large AI models and generative AI, the terminal side is undergoing a new evolution. Whether it’s phones, PCs, cars, or more terminals, perhaps in the future, we won’t need to specifically talk about AI, as AI will be everywhere.
And this so-called future is already here. According to IDC, it is estimated that by 2024, the shipment proportion of new AI PC machines in China will exceed 50%, expected to usher in a year of development. Boston Consulting Group predicts that by 2028, AI PCs will account for 80% of the PC market.
“In the past, when the internet first appeared, only a few people could use PCs to go online, while the emergence of smartphones allowed billions to connect. The development of terminal-side generative AI will be similar; it will enable everyone to fully utilize generative AI, changing work and life experiences, transforming various industries,” Ziad Asghar stated in an interview.
· FIN ·
Produced by the Tech Walker Team
From “Pharmaceuticals” to “Smart Medicines,” how AI is creating ripples of innovation in the life sciences?
Dialogue with the founder of Lanma Technology, Zhou Jian: The next frontier for large models, AI Agents!
