Why Does the Thor Chip Retain a GPU While Also Having an NPU?

Can a Pure GPU Achieve Autonomous Driving?

Technically, a pure GPU can achieve a certain level of autonomous driving, but it has significant shortcomings and is difficult to meet the demands of high-level autonomous driving.

GPUs can handle the parallel computing tasks required for autonomous driving (such as sensor data fusion and image recognition), but their original design was for graphics rendering, leading to the following limitations:

• Low energy efficiency: The general computing units of a GPU consume a lot of power when processing AI tasks, making them unsuitable for battery-powered scenarios in vehicles.

• Real-time challenges: Autonomous driving requires millisecond-level responses, and the general architecture of a GPU may lead to unpredictable latency fluctuations.

• High cost: High-end GPUs are expensive and require additional cooling designs.

Some early autonomous driving test vehicles attempted to use pure GPUs solutions. For example, a test vehicle based on the NVIDIA GTX1080 GPU experienced a target detection delay of about 80 milliseconds when processing data from a single camera, while the vehicle was traveling at 60 kilometers/ hour, meaning it would move 1.33 meters in 80 milliseconds, which poses safety risks in sudden situations.

Early on, Tesla also used GPUs (NVIDIA PX2), but later switched to self-developed NPUs (FSD chips) to optimize energy efficiency.

In terms of data processing capability, L4 level autonomous vehicles generate about 5-10GB of data per second. When using pure GPUs for processing, multiple GPUs need to work together. One test showed that using 4 NVIDIA TITAN X GPUs to process data from 8 cameras and 1 lidar resulted in a power consumption of 320W, which would reduce the range of electric vehicles by about 30%.

Additionally, when running complex deep learning models, pure GPUs are less efficient. For example, using the ResNet-152 model, processing a single frame of 4K images on a GPU takes 28 milliseconds, while the same task on a dedicated NPU only takes 8 milliseconds, showing a significant difference.

Therefore, while pure GPUs can achieve basic functions for low-level autonomous driving, their performance in terms of latency, power consumption, and efficiency makes it difficult to meet the requirements for L3 and above levels of autonomous driving, at least in terms of cost-effectiveness.

Comparison of the Fundamental Principles of GPU, NPU, and TPU

GPUs were originally designed for graphics rendering, with their core consisting of a large number of stream processors that form parallel computing units. Taking the NVIDIA GTX 1080 as an example, it has 2560 stream processors that work in thread blocks and support various computation types such as floating-point and integer.

When processing graphical data, GPUs can compute millions of pixels simultaneously, completing texture mapping, lighting calculations, and other operations. In deep learning, they can perform matrix operations in parallel, but due to their general architecture, 30%-40% of hardware resources remain idle when executing neural network computations.

Figure: GPU Architecture Diagram, Source: Internet

Note that the control unit of the GPU is for control purposes; if control is not needed, can we significantly reduce energy consumption and idle computing power? The answer is yes.

NPUs are chips specifically designed for neural network computations, integrating a large number of MAC (Multiply-Accumulate) units. For example, the Huawei Ascend 310B contains 2048 MAC units arranged in an array, optimized directly for matrix multiplication and accumulation operations.

NPUs adopt a data flow architecture, where the transmission path of data between storage units and computing units is fixed and short. When processing convolution operations, data enters the MAC array from the cache, completes the computation, and outputs the result, with more than 60% fewer intermediate steps than GPUs.

TPUs are chips customized by Google for machine learning, using a pulsating array architecture. For example, the TPU v2 has a pulsating array size of 512×512, where data flows like a pulse between units, with each unit completing a multiply-accumulate operation and passing the result to the next unit.

In this architecture, once data enters the array, it circulates internally and completes the computation, reducing the number of external storage accesses. When processing large matrix multiplications, TPUs have a data reuse rate more than 3 times higher than GPUs.

Figure: Common Features of NPU and TPU, Computing Unit Array, Source: Internet

However, while TPUs (the pulsating array is a concept proposed by Google for TPUs) and some NPUs use pulsating arrays, most NPUs are based on MAC arrays.

Pulsating arrays are a special array structure inspired by the human circulatory system. The core concept is to allow data to flow within the array of computing units, reducing memory access frequency. The entire array operates in a “pulsating” manner, where each processing element (PE, Processing Element) processes a portion of data in each computation cycle and passes it to the next interconnected PE. For example, in a 4×4 pulsating network, the matrix elements involved in the computation flow between array units in a specific order, with each unit completing a multiply-accumulate operation and passing the result to the next unit. Once data enters the array, it circulates internally and completes the computation. In this structure, data flows through the array like a pulse, greatly enhancing data reuse rates and reducing external storage access frequency, achieving a data reuse rate more than 3 times higher than traditional architectures. For instance, Google’s TPU adopts a pulsating array architecture, with the pulsating array size of 512×512, efficiently utilizing data and reducing the overhead of data movement within and outside the chip during large matrix multiplications.

MAC (Multiply-Accumulate) arrays are primarily composed of a large number of multiply-accumulate units arranged in an array format. A multiply-accumulate unit is the basic hardware unit that completes one multiplication and one addition operation. For example, the Huawei Ascend 310B NPU contains 2048 MAC units, which are specifically optimized for matrix multiplication and accumulation operations in neural network computations. When processing convolution operations, data enters the MAC array from the cache, where the multiply-accumulate units perform the multiply-accumulate operations on input data and weight data, directly completing the computation and outputting the result. MAC arrays typically adopt a data flow-driven architecture, deeply optimizing the flow path of data between storage units and computing units, directly accelerating key operations in neural networks through hardware-implemented activation function units, pooling units, etc., reducing data movement frequency and improving computational efficiency, with intermediate steps reduced by more than 60% compared to traditional general architectures.

The main difference between pulsating arrays and MAC arrays is the control timing. Pulsating arrays have strict periodic control timing, with data flowing between units at a fixed pulse, synchronizing the operations of each processing unit with data transmission precisely, orderly like a pulse; while MAC arrays have more flexible control timing, with each unit responding to instructions relatively independently, without strictly following a unified data flow pulse, focusing more on efficiently executing multiply-accumulate operations.

Figure: Animation of Pulsating Array in Matrix Multiplication, Source: Internet

Figure: MAC Unit Array, Source: Internet

In summary, the array-based NPU and TPU are more economical in neural network inference computations than GPUs because the simply arranged computing units act as neurons in the neural network, while the data pathways connecting them act as the weights of the neural network. As many articles state, the internal physical structure of NPUs and TPUs simulates the structure of neural networks.

Figure: Neural Network Inference Diagram, Source: Internet

Many readers may ask two questions.

First, how does an NPU handle neural networks larger than its native array?

From a hardware architecture perspective, NPUs face many challenges when processing neural networks larger than their array size, but it is not entirely impossible. The internal MAC (Multiply-Accumulate) array of an NPU is its core component for neural network computations. For instance, the Huawei Ascend 310B NPU contains 2048 MAC units arranged in an array, optimized for matrix multiplication and accumulation operations in neural network computations. When processing common operations in neural networks, such as convolution, data enters the MAC array from the cache, where it can directly complete computations and output results, with intermediate steps reduced by more than 60% compared to traditional architectures like GPUs.

However, if the scale of the neural network exceeds the native processing capability of the MAC array, for example, when facing ultra-large neural networks with parameter scales reaching tens of billions or even hundreds of billions, a single NPU MAC array may struggle to complete all data’s parallel processing within one computation cycle. This is similar to a limited lane highway where excessive traffic can lead to congestion. For instance, a certain NPU designed for smart security has a MAC array designed for medium-scale image recognition neural networks. When attempting to run a large-scale neural network designed for ultra-high-definition video analysis, the originally real-time process of image frame rates dropped from 30 frames/ second to below 5 frames/ second, significantly increasing latency and failing to meet real-time application requirements.

The solution is to frame the large network into smaller chunks that the NPU can process at once, but this may lead to performance issues.

Second, how to handle neural networks that are smaller than its native array but have inconsistent shapes? After all, the number of neurons in each layer of a neural network can vary.

The answer is padding, filling the empty neurons and weights with 0 (or special signals indicating to ignore). The goal is to ensure that during the multiply-accumulate operations, they do not have any effect, as multiplying 0 is still 0, and adding 0 is still 0, with no other calculations involved.

TPUs are similar.

The comparison table of the three is as follows:

Characteristics	GPU	NPU	TPU
Design Goal	Graphics Rendering/General Parallel Computing	Accelerating Neural Network Inference and Training	Tensor Operations (Google Specific)
Core Architecture	Thousands of SIMD Cores (General Computing Units)	Dedicated Matrix Computing Units (such as MAC Arrays)	Pulsating Arrays (Data Flow Optimization)
Advantages	High Flexibility, Suitable for Diverse Tasks	High Energy Efficiency, Low Latency Inference	Excellent Performance for Large-Scale Cloud Training
Typical Applications	Gaming, Scientific Computing, AI Training	Autonomous Driving, Edge AI, Mobile Side Inference	Google Cloud AI Services
Representative Products	NVIDIA A100, AMD Radeon Instinct	Tesla FSD, Huawei Ascend	Google TPU v4

Principle Differences:

• lGPU: Processes floating-point operations through large-scale parallel threads, but requires software-level optimization for AI tasks.

• lNPU: Hardware-level support for matrix multiply-accumulate ( MAC) operations, directly mapping to neural network computation graphs.

• lTPU: Uses pulsating arrays to reduce data movement overhead, optimized for TensorFlow.

Compared to each other, GPUs are highly versatile but lack specificity, NPUs focus on neural network computation efficiency, and TPUs have higher computational density for specific machine learning tasks (optimized for TensorFlow).

If we go further, GPUs are more suitable for training because model training requires backpropagation algorithms, and calculations need to be performed in both directions, which requires additional controllers to change the computation direction. However, NPUs (and TPUs) are more suitable for inference on trained models, as inference only needs to proceed in one direction.

Why Does Thor Retain a GPU While Also Having an NPU?

NVIDIA Thor is a computing chip designed for autonomous driving, containing multiple types of processing units (PUs), making it a typical example of a super heterogeneous fusion chip.

Figure: Framework and Functions of the Thor Chip, where ISP is not an Internet Service Provider, but an Image Signal Processor.

It retains a GPU while also equipping an NPU, mainly for the following reasons.

From a functional division perspective, there are different types of computing tasks in autonomous driving systems. NPUs are suitable for handling neural network-related tasks, such as detecting pedestrians using the YOLOv8 model, where Thor’s NPU takes about 5 milliseconds to process a single frame of image; while GPUs are responsible for preprocessing sensor data, such as converting the point cloud data from lidar from polar coordinates to Cartesian coordinates, where Thor’s GPU takes about 3 milliseconds to process 1 million point cloud data.

In terms of data interaction, both need to work together. The raw images captured by the camera are first corrected for distortion by the GPU, and the corrected images are then sent to the NPU for object recognition. Tests show that this collaborative mode improves efficiency by 40% compared to processing with a single processor, as it avoids frequent data transfers between different chips.

Additionally, considering software compatibility, there are currently many traditional algorithms based on GPUs development, such as some modules in SLAM (Simultaneous Localization and Mapping). Retaining GPUs allows these algorithms to run directly without needing redevelopment, saving at least 18 months of adaptation time.

From a cost perspective, Thor’s GPU and NPU are integrated on the same chip, reducing hardware costs by 25% compared to separate designs, while also reducing 50% of the circuit board space occupied.

Comparison of Energy Consumption and Cost Between GPU and NPU

At the same AI computing power, NPUs have significantly lower energy consumption than GPUs..

The NVIDIA Jetson AGX Xavier ( GPU solution) has an AI computing power of 32TOPS with a power consumption of 30W, resulting in an energy efficiency ratio of 1.07TOPS/W. The Huawei Ascend 310B ( NPU solution) has a computing power of 22TOPS with a power consumption of 8W, resulting in an energy efficiency ratio of 2.75TOPS/W, which is 2.5 times that of the former.

The NPU part of the Tesla FSD chip has a computing power of 144TOPS with a power consumption of 25W, resulting in an energy efficiency ratio of 5.76TOPS/W. To achieve similar AI computing power, 4 Jetson AGX Xavier chips would be needed, with a total power consumption of 120W, which is 4.8 times that of Tesla’s NPU.

In actual vehicle scenarios, a certain L4 level autonomous driving test vehicle using a pure GPU solution (total power consumption 150W) compared to a hybrid solution using NPU+GPU (total power consumption 60W), consumes more than 8 kWh per 100 kilometers, and based on an average of 15 kWh per 100 kilometers for electric vehicles, the range is reduced by about 53 kilometers.

In terms of single-chip costs, the bulk purchase price of the NVIDIA Jetson AGX Xavier is about 800 USD/ chip, while the Huawei Ascend 310B is about 300 USD/ chip.

To achieve 144TOPS of AI computing power, a pure GPU solution would require 5 Jetson AGX Xavier chips, with a total cost of 4000 USD; while using the Tesla FSD chip (including NPU), the cost per chip is about 500 USD, only 12.5% of the pure GPU solution.

Including surrounding circuits and cooling systems, the total hardware cost of a pure GPU solution is about 5500 USD, while the NPU+GPU hybrid solution is about 1200 USD, making the former 4.6 times that of the latter.

From a mass production perspective, when the production volume reaches 100,000 units, the unit R&D cost of the NPU can be amortized to 30 USD per unit, while the GPU, due to its complex architecture, still requires 80 USD per unit.

Metrics	GPU	NPU
Power Consumption	High (50-300W)	Very Low (1-10W)
TOPS per Watt	1-5W/TOPS	0.1-0.5W/TOPS
Cost	High (High-end chips exceed 10,000 RMB)	Medium to Low (Cost decreases rapidly after scaling)
Applicable Scenarios	Training/Cloud Inference	Edge Inference/Real-time Processing in Vehicles

Data Sources: The energy efficiency ratio of NPUs can reach more than 10 times that of GPUs, and the process requirements are lower (for example, a 14nm NPU is comparable to a 7nm GPU).

Conclusion

Pure GPUs can achieve low-level autonomous driving, but they have significant shortcomings in processing speed and energy consumption, making them unable to meet the demands of high-level autonomous driving.

From a principle perspective, GPUs are general but inefficient, NPUs are designed specifically for neural networks, and TPUs have high computational density in specific scenarios, with architectural differences leading to different applicable scenarios.

NVIDIA Thor retains both GPUs and NPUs because both can collaborate to improve overall efficiency and also maintain compatibility with existing software, reducing costs.

Energy consumption and cost data show that the energy efficiency ratio of NPUs is more than 2.5-5 times that of GPUs, and at the same computing power, the hardware cost of NPU solutions is only 12.5%-40% of that of pure GPU solutions.

In summary, autonomous driving requires NPUs because they can efficiently process neural network tasks at low power consumption, while GPUs have some utility in general computing but cannot meet the requirements for high-level autonomous driving when used alone. In the future, NPU+GPU hybrid solutions will become mainstream, ensuring processing efficiency while considering compatibility and cost.

Add WeChat to join the group for in-depth discussions on ideal operational situations and long-term fundamentals. This is not a car enthusiast group.