Abstract
The Neural Processing Unit (NPU) adopts a “data-driven parallel computing” architecture, particularly adept at processing large-scale multimedia data such as video and images. This article analyzes the working principles of NPUs in modern smart devices, architectural features, differences from CPUs/GPUs, and application scenarios, providing a detailed explanation of NPUs and their distinctions from CPUs and GPUs.
Table of Contents
- 1. What is an NPU?
- 2. NPU Processor Modules
- 3. NPU: The Core Carrier of Mobile AI
- 4. NPU vs. GPU
- 5. Characteristics of Different Processing Units
- 6. Practical Applications of NPUs
- 7. Definitions of Various Processing Units
- 8. Future Development Trends of NPUs
- 9. References
What is an NPU?
The Neural Processing Unit (NPU) is a dedicated processor optimized for neural networks and artificial intelligence workloads. With the explosive growth of deep learning and AI applications, traditional CPUs and GPUs, while capable of executing similar tasks, exhibit significant disadvantages in efficiency and power consumption. NPUs are designed specifically for neural network computations through hardware-level optimizations, providing higher AI computing performance at lower power consumption.
We are currently at the early stage of an explosion in the demand for neural network and machine learning processing. Traditional CPUs/GPUs can perform similar tasks, but NPUs optimized for neural networks far outperform CPUs/GPUs. Gradually, similar neural network tasks will be handled by dedicated NPU units.
NPUs (Neural Processing Units) are specialized processors for processing network application data packets, employing a “data-driven parallel computing” architecture, particularly skilled at handling massive multimedia data such as video and images.
NPUs are also integrated circuits, but unlike Application-Specific Integrated Circuits (ASICs) with single functions, network processing is more complex and flexible. Typically, we can use software or hardware for special programming to achieve specific network purposes based on the characteristics of network computing.
Core Advantages of NPUs
The main advantages of NPUs lie in their dedicated parallel computing architecture, specifically reflected in the following aspects:
| Advantage | Specific Description |
|---|---|
| High Parallelism | Capable of running multiple parallel threads simultaneously, processing large-scale matrix operations. |
| Dedicated Optimization | Hardware-level optimizations, including efficient cache systems and simplified processing cores. |
| Low Precision Computation | Focus on low-precision algorithms to improve throughput rather than merely pursuing low latency. |
| High Energy Efficiency Ratio | For AI tasks, the energy efficiency ratio is 10-100 times higher than that of general-purpose processors. |
| Storage-Compute Integration | By simulating neurons and synapses at the circuit level, it achieves the fusion of storage and computation. |
The highlight of NPUs is their ability to run multiple parallel threads—NPUs elevate parallel computing to a new level through some special hardware-level optimizations (for example, providing easily accessible cache systems for some unique processing cores). These high-capacity cores are simpler than typical “conventional” processors because they do not need to perform various types of tasks. This set of “optimization” combinations makes NPUs more efficient, which is why a significant amount of R&D investment is focused on ASICs.
One of the advantages of NPUs is that most of their time is focused on low-precision algorithms, new data flow architectures, or memory computing capabilities. Unlike GPUs, NPUs care more about throughput than latency.
NPU Processor Modules
NPUs are designed for IoT AI, aimed at accelerating neural network computations and addressing the inefficiencies of traditional chips in neural network operations. NPU processors include multiply-accumulate modules, activation function modules, two-dimensional data operation modules, decompression modules, and more.
The multiply-accumulate module is used for calculating matrix multiplication and addition, convolution, dot products, and other functions. The NPU has 64 MACs internally, while the SNPU has 32.
The activation function module is used to implement activation functions in neural networks by using up to 12th-order parameter fitting, with 6 MACs in the NPU and 3 in the SNPU.
The two-dimensional data operation module is used for operations on planes, such as downsampling and plane data copying. The NPU has 1 MAC and 1 SNPU internally.
The decompression module is used to decompress weight data. To address the limited memory bandwidth of IoT devices, the NPU compiler compresses weights in neural networks, achieving a compression effect of 6-10 times with almost no impact on accuracy.
Modern NPUs typically include the following core processing modules, which work together to achieve efficient neural network computation:
| Core Processing Module | Description |
|---|---|
| Multiply-Accumulate Unit (MAC) | The core computing unit of the NPU, used for executing matrix multiplication, addition, convolution, dot products, and other basic operations. High-end NPUs typically integrate hundreds or even thousands of MAC units for large-scale parallel computing. |
| Activation Function Unit | Responsible for implementing nonlinear transformations in neural networks. Modern NPUs typically use high-order parameter fitting methods to implement various activation functions (such as ReLU, Sigmoid, Tanh, etc.), improving computational efficiency while ensuring accuracy. |
| Two-Dimensional Data Operation Unit | Specialized in handling plane data operations, such as downsampling and plane data copying. These operations are particularly important in image processing and computer vision tasks. |
| Data Compression/Decompression Unit | To address the limited memory bandwidth of mobile devices, modern NPUs typically integrate dedicated weight data compression/decompression modules. Advanced compression algorithms can achieve a compression ratio of 6-10 times with minimal impact on computational accuracy. |
| Tensor Acceleration Unit | Units designed specifically to accelerate tensor operations, capable of efficiently processing multi-dimensional data structures, which are key components in deep learning model inference. |
Architectural Diagram of Neural Processing Unit (NPU):
NPU: The Core Carrier of Mobile AI
It is well known that the normal operation of mobile phones relies on SoC chips, which are tiny chips that integrate all the “organs” and work together to support various functions of the phone. The CPU is responsible for smooth switching of mobile applications, the GPU supports fast loading of game graphics, while the NPU is specifically responsible for AI computation and the implementation of AI applications.
This began with Huawei, which was the first company to use NPUs (Neural Processing Units) in mobile phones and the first to integrate NPUs into mobile CPU designs.
Evolution of NPUs in Mobile Devices
The application of NPUs in smartphones began in 2017 when Huawei first integrated an NPU into a commercial smartphone. Since then, major chip manufacturers have launched their own NPU solutions:
- • 2017: Huawei first integrated an NPU in the Kirin 970 processor
- • 2018: Apple introduced a neural engine in the A12 Bionic chip
- • 2019-2020: Qualcomm, Samsung, MediaTek, and others launched mobile SoCs with integrated NPUs
- • 2021-2023: NPU performance became a key competitive point for flagship mobile chips
- • 2024-2025: Mobile NPU performance will significantly improve, supporting large AI models to run on-device
Current Mainstream Mobile NPU Comparisons
NPUs have become standard configurations for the latest mobile processors, with significant performance differences:
- • Apple A17 Pro: 26-core neural engine, AI performance improved by about 40% compared to A16
- • Qualcomm Snapdragon 8 Gen 3: New Hexagon processor, AI performance improved by about 45% compared to the previous generation
- • MediaTek Dimensity 9300: 6th generation APU, AI performance improved by over 100%
- • Samsung Exynos 2400: Next-generation NPU, AI performance improved by about 14.7 times
NPU vs. GPU
Although GPUs have advantages in parallel computing capabilities, they do not operate independently and require CPU collaboration. The construction of neural network models and data flow still occurs on the CPU. Additionally, GPUs have issues with high power consumption and large size. The higher the performance, the larger the GPU, the higher the power consumption, and the more expensive it becomes, which is impractical for smaller devices and mobile devices. Therefore, the emergence of NPUs, which are small, low-power, high-performance, and highly efficient dedicated chips, is timely.
The working principle of NPUs is to simulate human neurons and synapses at the circuit level, directly processing large-scale neurons and synapses through a deep learning instruction set, where a single instruction can complete the processing of a group of neurons. Compared to CPUs and GPUs, NPUs improve operational efficiency by integrating storage and computation through synaptic weights.
CPUs and GPUs require thousands of instructions to complete neuron processing, while NPUs only need one or a few instructions to do so, thus having a significant advantage in deep learning processing efficiency. Experimental results show that under the same power consumption, the performance of NPUs is 118 times that of GPUs.
| Comparison Item | GPU | NPU |
|---|---|---|
| Architectural Differences | General-purpose parallel computing architecture, requiring CPU collaboration to process tasks. The construction of neural network models and data flow still occurs on the CPU. | Simulates human neurons and synapses at the circuit level, capable of directly processing large numbers of neurons and synapses using a deep learning instruction set. |
| Computational Efficiency Comparison – Instruction Efficiency | CPUs and GPUs require thousands of instructions to complete neuron processing. | NPUs only need one or a few instructions to complete neuron processing. |
| Computational Efficiency Comparison – Energy Efficiency Ratio | Under the same power consumption, general-purpose GPUs have AI computing performance far below that of dedicated NPUs. | Under the same power consumption, dedicated NPUs can achieve AI computing performance 10-50 times that of equivalent GPUs. |
| Computational Efficiency Comparison – Memory Access | Due to the lack of a special storage-compute integration mechanism, memory access overhead is high. | NPUs achieve storage-compute integration through synaptic weights, significantly reducing memory access overhead. |
| Application Scenario Differences | Advantageous Scenarios: General computing, graphics rendering, large-scale training. | Advantageous Scenarios: Edge device inference, low-power scenarios, real-time AI applications. |
Characteristics of Different Processing Units
CPU—70% of transistors are used to build cache and control units. Fewer computing units, suitable for logical control operations.
GPU—Transistors are mostly used to build computing units, with low computational complexity, suitable for large-scale parallel computing. Mainly used for big data, backend servers, and image processing.
NPU—Simulates neurons at the circuit level, achieving storage-compute integration through synaptic weights. A single instruction completes the processing of a group of neurons, improving operational efficiency. Mainly used in communication, big data, image processing.
FPGA—Programmable logic, high computational efficiency, closer to low-level IO. Achieves logical editability through redundant transistors and connections. Essentially instructionless, does not require shared memory, and has higher computational efficiency than CPUs and GPUs. Mainly used in smartphones, portable mobile devices, and automobiles.
| Type | Characteristics | Applications |
|---|---|---|
| CPU | About 70% of transistors are used to build cache and control units.Fewer computing units, suitable for logical control operations.Strong versatility, but low AI computing efficiency. | General computing, device control |
| GPU | Transistors are mainly used to build computing units.Low computational complexity, suitable for large-scale parallel computing. | Big data, backend servers, image processing |
| NPU | Simulates neurons at the circuit level, achieving storage-compute integration through synaptic weights.A single instruction can complete the processing of a group of neurons, improving operational efficiency. | Communication, big data, image processing, edge AI |
| FPGA | Programmable logic, high computational efficiency, closer to low-level IO.Achieves logical editability through redundant transistors and connections.Essentially instructionless, does not require shared memory, and has higher computational efficiency than CPUs and GPUs. | Smartphones, portable mobile devices, automobiles |
Practical Applications of NPUs
- • AI scene recognition during photography and image editing calculations via NPU
- • NPU determines light sources and dark detail synthesis for super night scenes
- • Voice assistant operations realized through NPU
- • NPU collaborates with GPU Turbo to predict the next frame for pre-rendering, enhancing game smoothness
- • NPU predicts touch operations to improve responsiveness and sensitivity
- • NPU assesses differences in front-end and back-end network speed requirements, optimizing network connections through Link Turbo technology
- • NPU intelligently adjusts resolution based on game rendering load
- • Reducing AI computation load through NPU to save power in gaming
- • NPU enables dynamic scheduling of CPU and GPU
- • NPU assists in big data advertising push
- • AI intelligent association function of input methods realized through NPU
Definitions of Various Processing Units
- • APU: Accelerated Processing Unit, AMD product used for accelerating image processing
- • BPU: Brain Processing Unit, Horizon’s embedded processor architecture
- • CPU: Central Processing Unit, mainstream product at the core of PCs
- • DPU: Data Stream Processing Unit, AI architecture proposed by Wave Computing
- • FPU: Floating Point Processing Unit, floating point module in general-purpose processors
- • GPU: Graphics Processing Unit, multi-threaded SIMD architecture designed for graphics processing
- • HPU: Holographic Processing Unit, Microsoft’s holographic computing chip and device
- • IPU: Intelligent Processing Unit, AI processor product from Graphcore (DeepMind investment)
- • MPU/MCU: Microprocessor/Microcontroller Unit, typically used in low-computation applications in RISC computer architecture products
- • NPU: Neural Network Processing Unit, a new type of processor based on neural network algorithms and acceleration
- • TPU: Tensor Processing Unit, Google’s processor specifically designed to accelerate AI algorithms
- • VPU: Vision Processing Unit, a chip launched by Movidius, acquired by Intel, focused on accelerating image processing and AI
Future Development Trends of NPUs
With the rapid development of AI technology and the continuous expansion of application scenarios, NPUs, as the core hardware for AI computing, are undergoing unprecedented innovation and transformation. The following is an in-depth analysis of the future development trends of NPUs:
Architectural Innovation
Further Integration of Heterogeneous Computing Architectures
Future SoC designs will achieve closer integration of NPUs with other computing units (such as CPUs, GPUs, DSPs), enabling seamless switching and collaboration of different computing tasks through unified memory architecture and intelligent scheduling systems. Reconfigurable computing architectures will become a trend, with the next generation of NPUs adopting dynamically reconfigurable architectures that adapt hardware resource allocation based on the characteristics of different AI tasks, achieving a better balance between generality and specificity. Mixed-precision computing will be supported, allowing for mixed precision computing modes from INT4 to FP16, with NPUs automatically selecting the most suitable computing precision based on the sensitivity of different parts of the model, balancing performance and accuracy requirements.
Maturation of In-Memory Computing Technology
Non-volatile in-memory computing will utilize new non-volatile memory technologies such as ReRAM and MRAM to achieve in-memory computing, significantly reducing data transport overhead and improving energy efficiency ratios by 10-100 times. Analog in-memory computing can further enhance energy efficiency and computational density by directly executing analog computations within storage arrays, particularly suitable for computation-intensive tasks like convolutional neural networks. 3D integrated in-memory computing will adopt 3D stacking technology to vertically integrate computing units with storage units, significantly increasing bandwidth and reducing latency, providing more efficient hardware support for large-scale neural networks.
Commercialization of Brain-like Computing Architectures
Specialized hardware for Spiking Neural Networks (SNN) will be designed for biologically inspired spiking neural networks, capable of achieving computation efficiency close to biological neural systems at extremely low power consumption. Neuromorphic chips that mimic the working mechanisms of neurons and synapses in the human brain will transition from research to commercialization, demonstrating unique advantages in scenarios such as time series prediction and anomaly detection. Adaptive learning hardware will support online learning and adaptive NPU architectures, capable of continuously optimizing models based on new data, achieving true edge intelligence without relying on the cloud.
Performance Enhancement
Further Optimization of Dedicated Operators
<spandomain-specific (such="" 10-100="" able="" accelerators="" access,="" achieved="" ai="" and="" application="" areas="" as="" be="" compilers="" computational="" computations,="" computer="" computing="" designed="" developed="" different="" dynamic="" efficiency.="" enabling="" etc.),="" for="" fusion="" improvements="" improving="" in="" intelligent="" intermediate="" language="" multiple="" natural="" networks="" neural="" npus,="" of="" operator="" operators="" operators,="" performance="" processing,="" providing="" recommendation="" reducing="" result="" significantly="" skip="" span="" sparse="" sparsity="" specifically="" storage="" systems,="" the="" through="" throughput.
Widespread Application of Quantization Technology
Extremely low-bit-width quantization techniques, such as 2-bit or even 1-bit quantization, will be widely applied in NPUs, achieving performance improvements of several times to dozens of times with acceptable accuracy loss. Adaptive quantization will automatically adjust quantization strategies based on the sensitivity of different layers of the model, maximizing performance improvements while ensuring accuracy. Hardware support for quantization-aware training will enable NPUs to directly support quantization-aware training, allowing models to better adapt to low-precision hardware characteristics and reduce accuracy loss caused by quantization.
Continuous Progress in Chip Technology
With the application of advanced process nodes such as 2nm or even more advanced processes, the transistor density of NPUs will continue to increase, significantly enhancing computational power per unit area. Advanced packaging technologies such as Chiplet design and 3D stacking will allow NPUs to break through single-chip area limitations, achieving larger-scale integration of parallel computing units. The application of new semiconductor materials such as carbon nanotubes and graphene will bring higher energy efficiency and lower operating temperatures to NPUs.
Application Expansion
Significant Enhancement of Edge AI Capabilities
Future NPUs will support lightweight edge training, enabling devices to perform personalized model fine-tuning based on user data while protecting privacy. Multi-device collaborative computing will allow multiple smart devices in home or office environments to achieve distributed collaborative computing to complete complex AI tasks. Next-generation NPUs will support ultra-low power modes, achieving all-day AI perception and analysis while controlling power consumption at the milliwatt level.
Standardization of On-device Large Model Inference
Optimized large language models (LLMs) with parameter scales of 10-100 billion will be able to run smoothly on mobile devices, providing a local interactive experience similar to ChatGPT. Multi-turn dialogue understanding will be specifically optimized for NPUs, supporting more natural and coherent human-computer interactions. On-device NPUs will support efficient integration with local knowledge bases, enabling AI assistants to access and understand users’ personal data, providing more relevant services.
Widespread Adoption of Multimodal AI Applications
Real-time fusion analysis of video, audio, text, and other multimodal data will be supported by NPUs, achieving more comprehensive scene understanding. Generative AI applications such as text-to-image, video editing, and music creation will achieve real-time responses with the acceleration of NPUs. In combination with AR/VR devices, NPUs will support real-time environmental understanding and virtual content generation, creating a more natural and immersive mixed-reality experience.
AI Security and Privacy Protection
Federated learning hardware acceleration will provide dedicated federated learning acceleration units in NPUs, enabling devices to participate in model training without uploading raw data. Built-in differential privacy mechanisms in NPUs will ensure data privacy protection during AI computation processes, balancing utility and privacy protection needs. Model tampering prevention mechanisms will integrate secure computing units in NPUs to prevent models from being maliciously tampered with or extracted, protecting AI intellectual property.
Deepening Vertical Industry Applications
Dedicated NPUs will enable wearable devices to monitor health indicators in real-time and perform complex analyses, providing early warnings for potential health risks. Optimized industrial-grade NPUs will support real-time production line optimization and predictive maintenance, improving manufacturing efficiency and product quality. Automotive-grade NPUs will process multi-sensor fusion data, achieving millisecond-level environmental perception and decision-making, advancing autonomous driving technology.
Ecological Evolution
Maturation of Development Toolchains
A unified programming model for NPUs across platforms and devices will significantly reduce the difficulty of AI application development and accelerate application innovation. AI-assisted compilers will automatically optimize models for different NPU architectures, realizing the vision of “train once, deploy anywhere.” Specialized performance analysis and debugging tools for NPUs will help developers accurately identify performance bottlenecks and optimize resource utilization.
Standardization and Open Ecosystem
Standardization of NPU hardware interfaces will promote the prosperity of hardware and software ecosystems, reducing development and adaptation costs. Open-source NPU designs will lower the barriers to AI hardware innovation, driving rapid development of NPUs in specific fields. NPU as a Service (NPUaaS) will provide customizable NPU services in the cloud, allowing developers to tailor virtual NPU resources according to application needs.
References
- 1. Sze, V., Chen, Y. H., Yang, T. J., & Emer, J. S. (2020). Efficient processing of deep neural networks. Synthetic Lectures on Computer Architecture, 15(2), 1-341.
- 2. Dally, W. J., Turakhia, Y., & Han, S. (2022). Domain-specific hardware accelerators. Communications of the ACM, 65(9), 48-57.
- 3. Technical white papers and product specifications from major chip manufacturers (2023-2025)
- 4. AI chip market research report (2024)
- 5. Mobile computing platform performance evaluation report (2025)
This article is translated from Neural Processing Unit (NPU) Explained[1]
Citation Link
<span>[1]</span> Neural Processing Unit (NPU) Explained: https://www.utmel.com/blog/categories/integrated%20circuit/neural-processing-unit-npu-explained