1: Classification of AI Chips
There are many AI chips on the market, dazzling to behold.
Based on their application scope, they can generally be divided into several categories.
Terminal AI Chips: Terminal AI chips require low power consumption and relatively low computational power, mainly for AI inference applications. Terminal AI chips are represented by various MCUs with AI modules, focusing on specific applications. For example, the AI chip in a smart speaker can be used for voice recognition, while the AI chip in a smart lock can be used for facial recognition, etc.
Cloud AI Chips: Cloud AI chips are used in data centers for cloud AI acceleration, capable of both inference and training. For example, NVIDIA’s GPGPU cards and Google’s TPUs. Cloud AI chips have strong performance and large sizes; for instance, the A100 reportedly has an area of 826mm² at 7nm, with robust performance!
In addition, there are Edge AI Chips.
So what are Edge AI chips used for?
When it comes to edge computing, there is a very famous “Octopus Theory.”
The octopus is a strange creature; it has eight legs, but some of its decisions do not need to be processed in its brain; instead, they are computed in its legs.
The octopus’s legs, compared to its brain (the cloud), represent the edge!
This analogy is so interesting that it is often referenced in edge computing scenarios.
For instance, autonomous driving or ADAS (Advanced Driver Assistance Systems) requires that the entire decision-making and design be completed locally.
There are many scenarios that require large data computations but have high real-time demands, not needing to go all the way to the cloud center for processing.
Examples include smart driving, smart factories, and traffic management combined with security.
Compared to terminal AI chips used in many consumer-level scenarios, edge AI chips are more focused on industrial applications.
Edge AI basically limits applications to a certain range, which can be a car, a train, a factory, or a store.
Within this range, there are some real-time AI decision-making and processing needs that must be met.
Accordingly, we refer to AI empowerment as autonomous driving, smart manufacturing, smart retail, etc.
The core purpose mainly emphasizes solving problems on the data source side.
This is the demand for the existence of edge AI chips.
2: Characteristics of Edge AI Chips
So what are the characteristics of edge AI chips?
1: Strong Computational Power: Edge AI chips need to have stronger computational power than terminal ones, typically solving problems independently. However, their performance is usually 1-2 orders of magnitude stronger than application-specific terminal AI chips used for facial recognition or voice recognition in consumer devices.
2: Rich Peripherals: Edge AI emphasizes the availability of information, such as the demand for multiple camera inputs, which greatly increases the number of interfaces like MIPI, allowing support for multiple camera inputs and other video/audio inputs simultaneously.
3: Programmability: Edge AI chips are typically aimed at industrial users, requiring AI to empower users. In other words, AI must integrate with user application scenarios, and programming is often needed to adapt to different models and scenarios based on the needs of various industrial users. They are not limited to a specific application.
A well-designed programmable architecture is key to solving problems. Edge AI chips are not directly provided to industrial clients; instead, they are tailored to meet the AI empowerment needs of industrial clients, which is a core feature of edge AI chips.
3: Architecture of Edge AI Chips
So what does the architecture of edge AI chips look like?
For example, the edge AI computing platform, JETSON, should be considered one.
The latest generation released is JETSON AGX Orin.
As NVIDIA’s edge AI computing platform, JETSON is not as famous as NVIDIA’s GPGPU.
However, JETSON also inherits the GPGPU architecture of Ampere and ARM Cortex-A78, allowing it to perform both inference and training on edge AI chips.
As an edge AI product, it has a processing performance of 200 TOPS (INT8).
Let’s take JETSON AGX Orin as an example to explore its internal chip architecture.
The computing part of this chip mainly consists of three components: CPU, GPU, and DSA (NVDLA + PVA).
CPU:
JETSON has three groups of 4-core A78 processors, with a frequency of up to 2GHz. This means that this chip has a 12-core A78 processor. Unlike mobile processors, the three clusters of A78 are symmetrical, not the big-little core design of mobile processors; they are primarily aimed at computational services rather than low power consumption for different loads in mobile applications. In some scalar computations, the multi-core A78’s computational capability is also quite formidable.
GPU:
The GPU is NVIDIA’s latest Ampere architecture, featuring 2048 CUDA cores and 64 Tensor cores. All of these are programmable. The Ampere architecture is the latest generation of GPGPU architecture, with previous generations being Kepler, Maxwell, Pascal, Volta, etc. The latest generation of Ampere architecture has upgraded tensor cores. By using Ampere GPUs, it differs from other edge AI chips in that it can support both inference and training.
Most importantly, this AI chip can be programmed using CUDA, and programmability is a core requirement for edge AI chips.
DSA:
As an AI acceleration unit, JETSON also has 2 NVDLA hard cores and a VISION accelerator PVA.
NVDLA is primarily used for inference. The core remains a large matrix convolution operation.
NVDLA has already been open-sourced, and interested friends can download and run the source code on GITHUB. See: nvdla · GitHub
This can be seen as practical results being used in the industry and has a significant promoting effect on the industry.
PVA uses a VPU architecture and employs a VLIW structure, which has good parallelism. The VIEW architecture simplifies the hardware structure, and the wide execution of VLIW does not come at the cost of performance and frequency. However, it also delegates the problem to software to execute.
IO Resources:
In addition to computational resources, IO resources are also quite rich. After all, edge AI needs rich inputs, supporting 6 cameras and 16-channel MIPI interfaces.
If you had to pick an important interface for edge AI chips, it would definitely be MIPI. After all, edge AI chips, besides computational power, also require MIPI interfaces.
MIPI is the “eyes” of edge AI chips (used to connect cameras). Unlike humans, it requires many “eyes” because edge AI chips need to “see in all directions and hear from all sides.”
Without eyes and ears, edge AI chips cannot function.
There are also USB interfaces that can support some USB cameras.
It can also support PCIe. Both RC and EP are supported, meaning it can serve as an accelerator card plugged into other hosts or as the main device plugged into other accelerator cards.
Additionally, in terms of networking, it supports 4 10G ports, enabling high-speed interconnectivity, which can achieve high-speed network transmission or interconnect several JETSON AGX units.
The following image shows the detailed parameters of JETSON AGX Orin; feel free to take it!
Based on these parameters, the chip’s area is not small. I believe this chip might be built on a 7nm process to balance area and power consumption.
Its typical power consumption is around 15W, 30W, and 45W in different levels.
4: Role of Edge AI Chips
So what can such a powerful AI chip do?
For example, during the pandemic, many places have restrictions on the flow of people (this venue limits flow to 100 people!).
From a small store to a large block, timely monitoring of foot traffic is a typical task.
By using facial recognition, one can obtain the density of people in an area and make real-time decisions to manage the flow of people in that area.
If it were a terminal AI MCU, it would be challenging to have significant computational power or to accept multiple video inputs simultaneously.
That’s where edge AI chips come into play.
As a solution provider, one must not only have a robust AI engine but also many video input sources.
Finally, a powerful AI framework (SDK) is needed to operate all this hardware.
This means that edge AI needs to be redeveloped based on user AI needs.
As mentioned earlier, one important feature of edge AI is the redevelopment of AI + scenarios based on industrial contexts.
Many AI chips have impressive paper specifications, but how to convert that computational power into user-perceived improvements is where much internal work can be done.
Thus, industrial users need an open AI platform, not just a chip with computational power; more importantly, AI business development should be based on user needs.
There is an old saying: “Work is not determined by the east; even if you work hard, it will be in vain.”
Without software or with poor software, it is like having martial arts skills without internal mastery.
Even with strong computational power, AI chips cannot perform without software (SDK).
Balancing hardware and software is an everlasting choice.
How to convert AI computational power into user productivity?
In this regard, JETSON AGX Orin provides JetPack 5.0, supporting CUDA 11 and the latest versions of cuDNN and TensorRT.
Through these software tools, especially CUDA, which benefit user development, the powerful computational capabilities and rich IO of the JETSON platform are combined.
Ultimately fulfilling the “mission” assigned to AI chips by edge computing.
What users ultimately receive is: User-Defined AI Chips.
Or Demand-Defined AI Chips.
This is the essence of edge AI chips!
I am Brother Wai Rui, a chip architect. If you find this article good, feel free to like, follow, and share.
Previous Readings:
Brother Wai Rui’s 2021 Year-End Summary