Click on the blue word to follow us
In today’s digital age, artificial intelligence (AI) is everywhere, profoundly changing our way of life. From the intelligent voice assistant accurately identifying instructions on mobile phones, to the safe driving of self-driving cars in complex road conditions, to the keen capture of abnormal behavior by security monitoring systems, there is a key role behind these AI applications – the Neural Network Processing Unit (NPU). The NPU, as a hardware accelerator designed for neural network computing tasks, occupies a pivotal position in the field of artificial intelligence. The traditional central processing unit (CPU) is like a versatile worker in our daily life, capable of handling various types of tasks, but it struggles with the large number of complex matrix operations and parallel computing required in deep learning. Graphics processors (GPUs) excel in image processing and deep learning training, but they are not optimized for AI inference. The emergence of NPUs is akin to a “super brain” tailored for the field of artificial intelligence, specifically designed to tackle challenging problems in deep learning tasks. It adopts a unique architecture, possesses powerful parallel computing capabilities, and can efficiently perform large-scale matrix operations and data transmission required for deep learning. This enhances the speed of AI inference while also excelling in energy efficiency, operating with lower power consumption, making it especially suitable for devices and application scenarios sensitive to power consumption.
The NPU employs a highly parallel computing architecture, integrating a large number of small computing units inside, much like a vast factory with countless efficient workers, each responsible for specific computing tasks. Taking the Convolutional Neural Network (CNN) as an example, the convolutional layers and fully connected layers can be subtly decomposed into parallel matrix operations. The NPUs can perform these matrix operations in parallel by scheduling thousands of calculation units simultaneously, greatly improving computing efficiency. In contrast to the CPU and GPU, the CPU mainly relies on sequential execution, akin to a steward managing various transactions in an orderly manner. Although it can handle various complex tasks, it appears less competent and efficient when faced with deep learning’s numerous parallel calculations. While GPUs are adept at parallel computing, they have many computing cores and can handle large-scale matrix operations, their architecture is relatively general and not specifically designed for neural network computing. When dealing with neural network tasks, redundant control logic consumes a certain amount of resources and time. The hardware structure of the NPU, such as the pulse array, directly and closely maps to the neural network computing model, effectively reducing redundant control logic, allowing computing resources to be utilized more fully, thus demonstrating clear advantages when handling neural network tasks.
The NPU has carefully designed a dedicated instruction set for neural network operations. These instructions are like special tools tailored for neural networks, capable of accurately completing various complex operations. For instance, for common neural network operations such as convolution, pooling, and activation functions, there are corresponding special instructions. These instructions are highly functional, with a single instruction capable of completing complex calculation steps, significantly reducing the overhead of instruction decoding and enhancing calculation speed and efficiency. Simultaneously, the NPU has built-in hardware acceleration modules for common operations, such as convolution accelerators and tensor cores. These hardware acceleration modules function like high-speed production lines in a factory, performing calculations directly with hardware circuits, far exceeding software simulation speeds. When executing convolution operations, the convolution accelerator can rapidly perform convolution on the input data, greatly improving operational speed, making the NPU more efficient and rapid in processing neural network tasks.
In neural network computing, the flow and storage of data are critical. The NPU employs a series of optimization techniques to enhance data flow efficiency and memory utilization. It is equipped with on-chip high-bandwidth memory, such as tightly coupled on-chip cache (SRAM) or high-bandwidth memory (HBM), which acts like a high-speed data channel, reducing data handling latency and meeting the high throughput demands of neural networks. The NPU also utilizes data reuse technology to optimize data flow through local principles. In convolutional neural networks, methods such as weight reuse and input feature map chunking are employed to minimize external memory access, thereby reducing power consumption. When processing an image, the NPU will handle the feature map of the image, using the same weight data to compute multiple times in a small block, avoiding frequent data reads from external memory, which not only saves time but also reduces power consumption.
In the initial stage of neural network computing, input data (such as images, audio, etc.) must be pre-processed to convert it into a format suitable for neural network models. Taking image data as an example, common preprocessing operations include normalization, image cropping, and scaling. The NPU supports acceleration of these common data processing tasks with hardware. When performing image recognition, the input image may vary in size, and the NPU will utilize the hardware acceleration module to quickly scale the image to a uniform size, while normalizing values and mapping them to a specific value range, allowing the neural network to process the data more effectively and improving the accuracy and efficiency of subsequent calculations. The convolutional layer is a crucial component in the Convolutional Neural Network (CNN), primarily responsible for extracting features from the input image.
In the NPU, the convolution operation is intricately mapped to a large number of parallel hardware units, significantly accelerating the computational process. The NPU optimizes the calculation process based on the size and stride of the convolution kernel, thereby reducing redundant operations and markedly improving calculation efficiency. When processing a high-resolution image, the NPU can schedule many computing units simultaneously, performing convolution calculations on different areas of the image in parallel, and quickly extracting various features of the image, such as edges and textures, to provide key information for subsequent analysis and judgment.
Following the convolution layer, it is typically succeeded by an activation function (such as ReLU, sigmoid, etc.), which performs a non-linear transformation of the convolution results and enhances the neural network’s expressive capability. The NPU is specifically equipped with units for accelerating the computation of common activation functions. Taking the ReLU function (f(x)=max(0,x)) as an example, when the input data passes through this activation function, the NPU can efficiently compute directly through hardware. It can quickly determine whether the input value exceeds 0. If it does, the value is directly output; if it does not, 0 is output. This hardware-level acceleration significantly enhances calculation speed, reduces computation time, and ensures smoother and more efficient neural network operations. The pooling layer’s main function is to reduce the dimensionality of the image, decreasing the spatial dimension of the data, thereby reducing computational load while retaining the main features of the image. When performing maximum pooling or average pooling, the NPU’s powerful parallel processing capabilities can greatly increase speed. During maximum pooling, the NPU divides the image into multiple small blocks, then finds the maximum value in each block in parallel, serving as the output for that block; during average pooling, the average value of all elements in each block is calculated as output in parallel. Through this parallel processing method, the NPU can swiftly complete the pooling operation, effectively reducing data volume and enhancing neural network operational efficiency. The fully connected layer connects the output of the previous layer to the next layer through matrix multiplication. In tasks such as image recognition, the computational load of the fully connected layer is often substantial. The NPU employs a dedicated matrix multiplication unit to expedite this process, specifically optimized for the matrix multiplication operations of the fully connected layer, enabling efficient execution of large-scale matrix multiplications.
When addressing image classification tasks, the fully connected layer must integrate features extracted from the previous convolutional and pooling layers and map them to different categories. The dedicated matrix multiplication unit of the NPU can rapidly complete this complex calculation process, providing accurate data support for the final classification result, significantly enhancing the computing efficiency of image recognition tasks. After multiple levels of complex calculations, the NPU will pass the output results (such as classification labels, position coordinates, etc.) to subsequent modules. The entire inference process is efficiently executed by the NPU, ensuring real-time performance and accuracy. In autonomous driving systems, the NPU processes images input by cameras, quickly outputting the categories of objects surrounding the vehicle (such as pedestrians, vehicles, traffic signs, etc.) and their position coordinates. These results are immediately relayed to the vehicle’s decision-making module to assist in making correct driving decisions, such as acceleration, deceleration, and turning, ensuring safety and smooth driving.
The emergence of NPUs has undoubtedly injected a strong impetus into the development of artificial intelligence and opened up new pathways. From their initial emergence to widespread application across various fields, the development of NPUs has been marked by innovation and breakthroughs, and their future development prospects are promising. Currently, the development of NPUs has achieved remarkable milestones, playing an indispensable role in smartphones, autonomous driving, edge computing, and cloud computing, bringing significant convenience and changes to people’s lives and work. However, like any emerging technology, NPUs also face challenges in their development process.
The NPU will increasingly integrate with other technologies such as cloud computing, edge computing, and the Internet of Things, forming a more intelligent computing ecosystem. In the realm of cloud computing, the NPU will further accelerate the training and inference processes of machine learning models, enhancing the performance and efficiency of cloud computing services; in edge computing and the Internet of Things, the NPU will provide more powerful localized intelligent processing capabilities for edge devices, enabling real-time data analysis and decision-making, reducing data transmission delays, and improving system response speed and reliability.
The NPU plays a crucial role and holds broad prospects in advancing artificial intelligence development. It will continue to lead innovation and development in artificial intelligence technology, providing key support for intelligent transformation across various fields, making our lives more intelligent, convenient, and beautiful. Let us wait and witness the NPU create more miracles in the future.
For more details, please contact:
Professional electronic components authorized agent and IDH
DST INTERNATIONAL GROUP (H. K) LIMITED
（Shenzhen Guangzhou Beijing Shanghai Xian Suzhou HK）
E-mail：[email protected]
Web：www.szdst.com.cn
Scan code to follow
Wechat｜DST-EIEC
｜DST electronics
1
Great review of the past
STMicroelectronics: Challenge to the wave of semiconductors
Falling into the sea of stars in breaking and restructuring
Full Analysis from Principles to Advantages and Disadvantages
Milestones and new journey of China’s chip industry
Italian Semiconductor: New Technology Engine of Ancient Country
Semiconductor Equipment: Current Situation, Challenge and Future
Explore the secrets behind its investment in chip companies
UCIE Bus: Breaking Chiplet Interconnection Dilemma
Guangzhou’s ‘chip’ power, rising in progress
Cadence: The Legendary Evolution History of the EDA Field