Why Do Autonomous Vehicles Need NPU? Is GPU Not Enough?

Image CPU，GPU，NPU Comparison, from the internet

Can Pure GPU Achieve Autonomous Driving?

Technically, a pure GPU can achieve a certain level of autonomous driving, but it has significant shortcomings and is difficult to meet the demands of high-level autonomous driving.

GPU can handle the parallel computing tasks required for autonomous driving (such as sensor data fusion, image recognition, etc.), but its original design was for graphics rendering, leading to the following limitations:

• Low energy efficiency: The general computing units of the GPU consume high power when processing AI tasks, making it unsuitable for battery-powered scenarios in vehicles.

• Real-time challenges: Autonomous driving requires millisecond-level responses, and the general architecture of the GPU may lead to uncertain latency fluctuations.

• High cost: High-end GPU is expensive and requires additional cooling designs.

Some early autonomous driving test vehicles attempted to use pure GPU solutions. For example, a test vehicle based on the NVIDIA GTX1080 GPU had a target detection delay of about 80 milliseconds when processing single-channel camera data, and when the vehicle was traveling at 60 kilometers/ hour, it would move forward 1.33 meters in 80 milliseconds, which poses safety risks in emergency situations.

Early on, Tesla also used GPU (NVIDIA PX2), but later switched to self-developed NPU (FSD chip) to optimize energy efficiency.

In terms of data processing capability, L4 level autonomous vehicles generate about 5-10GB of data per second, and pure GPU processing requires multiple GPUs to work together. A test showed that using 4 NVIDIA TITAN X GPUs to process 8 channels of camera and 1 channel of LiDAR data resulted in a power consumption of 320W, which would reduce the range of electric vehicles by about 30%.

Additionally, when running complex deep learning models, pure GPU is less efficient. For example, using the ResNet-152 model, processing a single frame of 4K image on a GPU takes 28 milliseconds, while the same task on a dedicated NPU only takes 8 milliseconds, showing a significant difference.

Therefore, pure GPU can achieve basic functions for low-level autonomous driving, but in terms of latency, power consumption, and efficiency, it is difficult to meet the requirements for L3 and above levels of autonomous driving, at least not at a good cost-performance ratio.

Comparison of the Fundamental Principles of GPU, NPU, and TPU

GPU was originally designed for graphics rendering, with its core consisting of a large number of stream processors that form parallel computing units. Taking the NVIDIA GTX 1080 as an example, it has 2560 stream processors that work in thread blocks and support various computation types such as floating-point and integer.

When processing graphical data, GPU can compute millions of pixels simultaneously, completing operations such as texture mapping and lighting calculations. In deep learning, it can parallel process matrix operations, but due to its general architecture, 30%-40% of hardware resources remain idle when executing neural network computations.

Image GPU Architecture Diagram, from the internet

Noticing the control unit of the GPU, if control is not needed, can we reduce a lot of energy consumption and idle computing power? The answer is yes.

NPU is a chip specifically designed for neural network computations, integrating a large number of MAC (Multiply-Accumulate) units. For example, Huawei’s Ascend 310B contains 2048 MAC units arranged in an array, optimized directly for matrix multiplication and accumulation operations.

NPU adopts a data flow architecture, where the transmission path of data between storage units and computing units is fixed and short. When processing convolution operations, data enters the MAC array from the cache, completes the computation directly, and outputs the result, with more than 60% fewer intermediate steps than GPU.

TPU is a chip customized by Google for machine learning, adopting a pulsating array architecture. Taking TPU v2 as an example, its pulsating array size is 512×512, and once data enters the array, it flows between units like a pulse, with each unit completing a multiply-accumulate operation and passing the result to the next unit.

In this architecture, once data enters the array, it continuously circulates internally and completes calculations, reducing the number of external storage accesses. When processing large matrix multiplications, TPU has a data reuse rate more than 3 times higher than GPU.

Image NPU and TPU share common characteristics, computing unit arrays, from the internet

However, TPU (the pulsating array is proposed specifically for TPU) and some NPU are pulsating arrays, while most NPU are MAC arrays.

Pulsating arrays are a special array structure inspired by the human circulatory system. The core concept is to allow data to flow within the array of computing units, reducing memory access frequency. The entire array operates in a “pulse” manner, with each processing unit ( PE, Processing Element) processing a portion of data in each computation cycle and passing it to the next interconnected PE. For example, in a 4×4 pulsating network, the matrix elements participating in the computation flow between array units in a specific order, with each unit completing a multiply-accumulate operation and passing the result to the next unit. Once data enters the array, it continuously circulates internally and completes calculations. In this structure, data flows within the array like a pulse, greatly enhancing data reuse rates and reducing external storage access frequency, achieving a data reuse rate more than 3 times higher than traditional architectures. For example, Google’s TPU adopts a pulsating array architecture, with the pulsating array size of 512×512, efficiently utilizing data and reducing the overhead of data movement within and outside the chip when performing large matrix multiplications.

MAC (Multiply-Accumulate unit) arrays are primarily composed of a large number of multiply-accumulate units arranged in an array format. A multiply-accumulate unit is the basic hardware unit that completes one multiplication and one addition operation. For example, Huawei’s Ascend 310B NPU contains 2048 MAC units, which are specifically optimized for matrix multiplication and accumulation operations in neural network computations. When processing convolution operations, data enters the MAC array, where the multiply-accumulate units perform multiply-accumulate operations on input data and weight data, completing calculations directly and outputting results. MAC arrays typically adopt a data flow-driven architecture, deeply optimizing the flow path of data between storage units and computing units, directly accelerating key operations in neural networks through hardware-implemented activation function units, pooling units, etc., reducing data movement frequency and improving computational efficiency, with intermediate steps reduced by more than 60% compared to traditional general architectures.

The main difference between pulsating arrays and MAC arrays is the control timing. The control timing of pulsating arrays is strictly periodic, with data flowing between units at a fixed pulse, synchronizing the operations of each processing unit with data transmission precisely, orderly like a pulse; while the control timing of MAC arrays is more flexible, with each unit responding independently to instructions without strictly following a unified data flow pulse, focusing more on efficiently executing multiply-accumulate operations.

Image Pulsating Array in Matrix Multiplication Animation, from the internet

Image MAC Unit Array, from the internet

In summary, the array form of NPU and TPU is more economical in neural network inference calculations than GPU (in terms of response time, cost, and energy consumption) because the simply arranged computing units in the array act as the neurons of the neural network, while the data pathways connecting them act as the weights of the neural network. As many articles state, the internal physical structure of NPU and TPU simulates the structure of neural networks.

Image Neural Network Inference Diagram, from the internet

Many readers may ask two questions.

First, how does the NPU handle neural networks larger than its native array?

From a hardware architecture perspective, NPU does face many challenges when processing neural networks larger than its array size, but it is not entirely unfeasible. NPU integrates a MAC (Multiply-Accumulate) array as its core component for neural network computations. For instance, Huawei’s Ascend 310B NPU contains 2048 MAC units arranged in an array, optimized for matrix multiplication and accumulation operations in neural network computations. When processing common operations in neural networks, such as convolution operations, data enters the MAC array from the cache, where it can directly complete calculations and output results, with more than 60% fewer intermediate steps than traditional architectures like GPU.

However, if the scale of the neural network exceeds the native processing capability of the MAC array, for example, when facing ultra-large-scale neural networks with parameter scales reaching tens of billions or even hundreds of billions, a single NPU MAC array may struggle to complete all data’s parallel processing within one computation cycle. The scale of the MAC array limits the amount of data it can process simultaneously, similar to a limited lane on a highway, which can become congested when traffic volume is too high. For example, a certain NPU designed for smart security has a MAC array designed for medium-scale image recognition neural networks, and when attempting to run a large-scale neural network designed for ultra-high-definition video analysis, the frame rate that could originally be processed in real-time dropped from 30 frames/ second to below 5 frames/ second, significantly increasing latency and failing to meet the real-time requirements of practical applications.

The solution is to frame the large network into smaller chunks that the NPU can process at once, but this may lead to performance issues.

Second, how to handle neural networks that are smaller than its native array but have inconsistent shapes? After all, the number of neurons in each layer of a neural network can vary.

The answer is padding, filling the empty neurons and weights with 0 (or special signals indicating to ignore). The goal is to ensure that during the multiply-accumulate operations, they do not have any effect, because multiplying 0 is still 0, and adding 0 is still 0, and no other calculations are performed.

TPU is similar.

The comparison table of the three is as follows:

Characteristics	GPU	NPU	TPU
Design Goal	Graphics Rendering/General Parallel Computing	Accelerating Neural Network Inference and Training	Tensor Operations (Google Specific)
Core Architecture	Thousands of SIMD Cores (General Computing Units)	Dedicated Matrix Computing Units (such as MAC Arrays)	Pulsating Arrays (Data Flow Optimization)
Advantages	High Flexibility, Suitable for Diverse Tasks	High Energy Efficiency, Low Latency Inference	Excellent Performance for Large-Scale Cloud Training
Typical Applications	Gaming, Scientific Computing, AI Training	Autonomous Driving, Edge AI, Mobile Side Inference	Google Cloud AI Services
Representative Products	NVIDIA A100, AMD Radeon Instinct	Tesla FSD, Huawei Ascend	Google TPU v4

Principle Differences：

• lGPU: Processes floating-point operations through large-scale parallel threads, but requires software-level optimization for AI tasks.

• lNPU: Hardware-level support for matrix multiply-accumulate ( MAC) operations, directly mapping to neural network computation graphs.

• lTPU: Uses pulsating arrays to reduce data movement overhead, optimized for TensorFlow.

Compared to each other, GPU is highly versatile but lacks specificity, NPU focuses on neural network computation efficiency, and TPU has higher computational density for specific machine learning tasks (optimized for TensorFlow).

If further discussed, GPU is more suitable for training because model training requires backpropagation algorithms, and calculations need to be performed in two directions, which requires additional controllers to change the computation direction. However, NPU (and TPU) are more suitable for inference on trained models, as inference only needs to proceed in one direction.

Why Does Thor Retain GPU and Also Have NPU

NVIDIA Thor is a computing chip for autonomous driving, which includes various PUs and is a typical example of a super heterogeneous fusion chip.

Image Thor Chip Framework and Functions, where ISP is not an Internet Service Provider, but an Image Signal Processor, for image signal processing

It retains GPU while also equipping NPU for several reasons.

From the perspective of functional division, there are different types of computing tasks in autonomous driving systems. NPU is suitable for handling neural network-related tasks, such as detecting pedestrians using the YOLOv8 model, with Thor’s NPU taking about 5 milliseconds to process a single frame of image; while GPU is responsible for preprocessing sensor data, such as converting LiDAR point cloud data from polar coordinates to Cartesian coordinates, with Thor’s GPU taking about 3 milliseconds to process 1 million point cloud data.

In terms of data interaction, both need to work together. The raw images collected by the camera are first corrected for distortion by the GPU, and the corrected images are then sent to the NPU for target recognition. Tests show that this collaborative mode improves efficiency by 40% compared to single-processor processing, as it avoids frequent data transfers between different chips.

Additionally, considering software compatibility, there are currently many traditional algorithms based on GPU development, such as some modules in SLAM (Simultaneous Localization and Mapping). Retaining GPU allows these algorithms to run directly without needing redevelopment, saving at least 18 months of adaptation time.

From a cost perspective, Thor’s GPU and NPU are integrated on the same chip, reducing hardware costs by 25% compared to separate designs, while also reducing 50% of circuit board space usage.

Comparison of GPU and NPU in Energy Consumption and Cost

Under the same AI computing power, NPU has significantly lower energy consumption than GPU.

NVIDIA Jetson AGX Xavier ( GPU solution) has an AI computing power of 32TOPS with a power consumption of 30W, resulting in an energy efficiency ratio of 1.07TOPS/W. Huawei’s Ascend 310B ( NPU solution) has a computing power of 22TOPS with a power consumption of 8W, resulting in an energy efficiency ratio of 2.75TOPS/W, which is 2.5 times that of the former.

The NPU part of Tesla’s FSD chip has a computing power of 144TOPS with a power consumption of 25W, resulting in an energy efficiency ratio of 5.76TOPS/W. To achieve similar AI computing power, 4 Jetson AGX Xavier chips would be needed, with a total power consumption of 120W, which is 4.8 times that of Tesla’s NPU.

In practical vehicle scenarios, a certain L4 level autonomous driving test vehicle using a pure GPU solution (total power consumption 150W) compared to a hybrid solution using NPU+GPU (total power consumption 60W), consumes more than 8 kWh per 100 kilometers, and based on an average of 15 kWh per 100 kilometers for electric vehicles, the range is reduced by about 53 kilometers.

In terms of single-chip costs, the bulk purchase price of NVIDIA Jetson AGX Xavier is about 800 USD/ chip, while Huawei’s Ascend 310B is about 300 USD/ chip.

To achieve 144TOPS of AI computing power, a pure GPU solution would require 5 Jetson AGX Xavier chips, with a total cost of 4000 USD; while using Tesla’s FSD chip (including NPU), the cost per chip is about 500 USD, only 12.5% of the pure GPU solution.

Including peripheral circuits and cooling systems, the total hardware cost of a pure GPU solution is about 5500 USD, while the NPU+GPU hybrid solution is about 1200 USD, making the former 4.6 times that of the latter.

From a mass production perspective, when the production volume reaches 100,000 units, the unit R&D cost of NPU can be amortized to 30 USD per unit, while GPU, due to its complex architecture, still requires 80 USD per unit.

Metrics	GPU	NPU
Power Consumption	High (50-300W)	Very Low (1-10W)
TOPS per Watt	1-5W/TOPS	0.1-0.5W/TOPS
Cost	High (High-end chips exceed 10,000 yuan)	Medium to Low (Cost decreases rapidly after scaling)
Applicable Scenarios	Training/Cloud Inference	Edge Inference/In-Vehicle Real-Time Processing

Data Sources: The energy efficiency ratio of NPU can reach more than 10 times that of GPU, and the process requirements are lower (for example, a 14nm NPU is comparable to a 7nm GPU).

Conclusion

Pure GPU can achieve low-level autonomous driving, but it has significant shortcomings in processing speed and energy consumption, making it unable to meet the demands of high-level autonomous driving.

From a principle perspective, GPU is general but inefficient, NPU is designed specifically for neural networks, and TPU has high computational density in specific scenarios, with architectural differences leading to different applicable scenarios.

NVIDIA Thor retains both GPU and NPU because both can collaborate to improve overall efficiency and also maintain compatibility with existing software, reducing costs.

Energy consumption and cost data show that NPU has an energy efficiency ratio of 2.5-5 times that of GPU, and under the same computing power, the hardware cost of the NPU solution is only 12.5%-40% of that of the pure GPU solution.

In summary, autonomous driving requires NPU because it can efficiently process neural network tasks with low power consumption, while GPU has some utility in general computing but cannot meet the requirements for high-level autonomous driving when used alone. In the future, NPU+GPU hybrid solutions will become mainstream, ensuring processing efficiency while considering compatibility and cost.

— END —

*Disclaimer: This article is original by the author. The content reflects the author’s personal views, and the publication by LuKe Verification is solely to convey a different perspective, not representing LuKe Verification’s endorsement or support of this view. If there are any objections, please feel free to contact LuKe Verification.