Generally speaking, Edge AI is a subset of conventional AI and should adhere to the same principles. However, when deploying artificial intelligence on edge devices, there are some special considerations that need to be taken into account. This article summarizes the differences between Edge AI and conventional AI based on the book AI at the Edge.
Training is Rare
Many AI applications are based on machine learning algorithms. In most cases, machine learning involves training a model to make predictions based on labeled data. Once the model is trained, it can be used for inference, which means making new predictions on data it has never seen before.
When discussing Edge AI and machine learning, it typically refers to inference. Compared to inference, training a model requires more computation and memory, and usually needs a labeled dataset. All of these factors are challenging to implement at the edge due to the limited resources of edge devices, and the data on edge devices is often raw and unprocessed.
For this reason, models used in Edge AI typically need to be trained before deployment to the device, using relatively powerful computing resources and datasets that are often manually cleaned and labeled. Technically, it is possible to train machine learning models on the edge device itself, but this is very rare, mainly due to the lack of labeled data for training and evaluation.
There are two more common types of on-device training. One is for tasks like facial or fingerprint verification on mobile phones, mapping a set of biometric information to a specific user. The second is for predictive maintenance, where on-device algorithms learn the “normal” state of a machine so that actions can be taken when the state becomes abnormal.
The Key is Sensor Data
Edge devices are located at the point of data generation. Typically, edge devices are equipped with sensors that can connect immediately to the external environment. The goal of deploying Edge AI is to understand this data, identify patterns, and use them to make decisions.
The inherent nature of sensor data is that it is often abundant, noisy, and difficult to manage. They can be acquired at a high frequency, potentially thousands of times per second. Embedded devices running Edge AI applications need to collect data, process it, input it into AI algorithms, and take action based on the results within a limited time. This is a significant challenge, especially since most embedded devices have limited resources and insufficient RAM to store large amounts of data.
The need to tame raw sensor data makes digital signal processing a critical part of most Edge AI deployments. In any efficient and effective application, signal processing and AI components must be designed as a system together, balancing performance and accuracy.
Many traditional machine learning and data science tools focus on tabular data, such as company finances or consumer product evaluations. In contrast, Edge AI tools need to handle a continuous stream of sensor data. This means that building Edge AI applications requires a completely different set of skills and technologies.
Machine Learning Models are Very Small
One of the purposes of designing edge devices is to reduce cost and power consumption. Therefore, their processor speeds are much slower than personal computers or web servers, and they have much less memory.
The resource constraints of these devices mean that when machine learning is used to implement Edge AI, the machine learning models must be very small. On mid-range microcontrollers, there may only be about 100 KB of ROM available to store the model, while some devices have much less ROM. Since larger models require more time to execute, the slower speed of the device processors also encourages developers to deploy smaller models.
Making models smaller requires some trade-offs. First, larger models have greater learning capacity. As the model size decreases, it loses some ability to represent its training dataset and may be less accurate. Therefore, developers creating embedded machine learning applications must balance model size with the required accuracy.
Various techniques can be used to compress models, reducing their size so that they fit smaller hardware and require less computation time. These compression techniques are very useful, but they can also affect the accuracy of the model, sometimes in subtle but risky ways.
In summary, not all applications require large, complex models. Those related to image processing, for example, require larger models because interpreting visual information involves many nuances. However, for simpler data, models of just a few kilobytes or smaller are sufficient.
Feedback Learning is Limited
AI applications are built through a series of iterative feedback loops. We need to measure its performance and then figure out how to improve it.
For example, suppose we build a fitness monitor that estimates the time it takes for the wearer to run 10 kilometers based on data collected from onboard sensors. To test whether it is functioning well, we can wait until the wearer actually runs 10 kilometers and see if the prediction is correct. If the prediction is inaccurate, we can add the wearer’s data from this run to our training dataset and try to train a better model.
If we have a reliable internet connection, this shouldn’t be too difficult—we can upload the data to a server. But the characteristic of Edge AI is that it is deployed on devices with limited connectivity. In this case, there may not be enough bandwidth to upload new training data. In many cases, it may not be possible to upload anything at all.
This presents a significant challenge for application development. How can we ensure that the system performs well in the real world when access to the system is limited? How can we improve our system when collecting more data is so difficult? This is a core theme of Edge AI development.
Diverse and Heterogeneous Computing
Most server-side AI applications run on standard x86 processors, often augmented with some graphics processing units (GPUs) to assist with deep learning inference. Due to recent server CPU developments from Arm and unique deep learning accelerators like Google’s TPU (Tensor Processing Unit), conventional AI has some diversity, but most workloads run on fairly standard hardware.
In contrast, the embedded world includes a dazzling array of device types:
-
Microcontrollers (MCUs), including 8-bit and 32-bit processors
-
System-on-Chip (SoC) devices running embedded Linux
-
General-purpose accelerators based on GPU technology
-
Field Programmable Gate Arrays (FPGAs)
-
Fixed-architecture accelerators that can quickly run a single model architecture
Each category includes countless devices from many different manufacturers, each with a unique set of build tools, programming environments, and interface options, which can be overwhelming. The diversity of hardware means that any specific application scenario may have multiple suitable systems. The difficult part is how to choose the most appropriate device.
The Goal is Often Just “Good Enough”
For conventional AI, the goal is often to achieve the best performance regardless of cost. The size of deep learning models used in server-side applications can reach gigabytes, and they rely on powerful GPU computing capabilities to run in real-time. When computation is not a barrier, the most accurate models are often the best choice.
The benefits of Edge AI come with some serious limitations. Edge devices have weaker computing capabilities, and when making trade-offs between device performance and accuracy, it often involves tricky choices.
This is undoubtedly a challenge, but it is not insurmountable. In many application scenarios, running AI at the edge has significant advantages that outweigh the drawbacks of slightly reduced accuracy. Even a small amount of intelligence on the device is much better than none at all.
The goal is to build applications that can fully leverage this “good enough” performance. Alasdair Allan elegantly describes this approach as “Capable Computing.” The key to achieving this is to use tools that help us understand how the application performs in the real world, taking into account possible performance losses.
Tools and Practices are Still Evolving
As a new technology that has just begun to gain widespread adoption, Edge AI still relies on tools and methods developed for large-scale server-side AI. In fact, most AI research still focuses on building large models on huge datasets. This has several implications.
First, current Edge AI often uses existing development tools from the fields of data science and machine learning. On the positive side, this means benefiting from a proven ecosystem with rich libraries and frameworks. However, existing tools rarely prioritize things that are important for Edge AI, such as model size, computational efficiency, and the ability to train on small amounts of data, so some extra work is needed to bring these to the forefront.
Second, since research in Edge AI is relatively new, the rapid development in this field is attracting more and more researchers and engineers, and new methods for improving efficiency are emerging, along with effective applications that combine suitable practices and technologies. This rapidly changing environment makes Edge AI a very exciting field.
In summary, this article outlines the differences between Edge AI and conventional AI: Edge AI is more focused on inference rather than training, emphasizes sensor data, utilizes smaller models, has limited feedback, features diverse and heterogeneous computing hardware, aims for “good enough” performance, and its tools and applications are still evolving.