Author: Wu Nü Wang (Peng Zhao)

IoT Think Tank Original

Reprint must indicate the source and origin

Introduction

The most widely distributed IoT devices are often very small and have limited power. They serve as terminal hardware, collecting various data through embedded sensors; their computing power is limited and extremely sensitive to power consumption. Can these devices also implement machine learning? One trend is that artificial intelligence (AI) is rapidly moving from the “cloud” to the “edge,” entering increasingly smaller IoT devices. The machine learning process implemented on microprocessors at the terminal and edge is referred to as Micro Machine Learning, or TinyML.

Total Word Count: 5500 words, Reading Time: 10 minutes

Wu Nü Wang: TinyML Holds Big Opportunities

This is my 166th article written for the Wu Nü Heart Sutra column.

The most widely distributed IoT devices are often very small and have limited power. They serve as terminal hardware, collecting various data through embedded sensors; their computing power is limited and extremely sensitive to power consumption. Can these devices also implement machine learning?

One trend is that artificial intelligence (AI) is rapidly moving from the “cloud” to the “edge,” entering increasingly smaller IoT devices.

The machine learning process implemented on microprocessors at the terminal and edge is referred to as Micro Machine Learning, or TinyML. More accurately, TinyML refers to the methods, tools, and techniques that engineers use to implement machine learning on devices operating below the mW power range.

A number of companies are leveraging TinyML-related technologies and products to explore how to better equip machine learning on these ubiquitous small devices to enhance their analytical capabilities and operational efficiency.

TinyML is the intersection of different technological fields and driving factors; it lies at the intersection of IoT devices, machine learning, and edge computing, and is advancing rapidly due to the combined effects of various driving forces.

In mid-February, the TinyML 2020 Summit was held in Silicon Valley. Companies like NVIDIA, ARM, Qualcomm, Google, Microsoft, and Samsung showcased the latest achievements in micro machine learning.

This was the second TinyML summit. The TinyML 2019 summit attracted hundreds of engineers from over 90 companies, while this year’s summit was unprecedented and reached many important conclusions:

• For many application scenarios, TinyML technology and hardware have evolved to a practical stage;

• Significant breakthroughs have been achieved in algorithms, networks, and ML models under 100KB;

• There is a rapid increase in low-power demand in the fields of vision and audio.

The TinyML summit was organized by the TinyML Foundation, which was established in July 2019. Its members include not only major well-known companies but also excellent startups like GreeWaves, Eta Compute, Esperanto, Xnor, and Picovoice.

The progress of technology and the development of the ecosystem have given a huge impetus to the development of TinyML. Currently, the influence of TinyML is continuing to ferment, and 2020 is bound to see a large number of products and solutions emerge, making it a field worthy of focus and attention. Therefore, this article will present a panoramic view of TinyML:

• What is TinyML?

• What is TinyML as a Service?

• Why is TinyML Important?

• Market Size Forecast for TinyML

• What are the Latest Products of TinyML?

• What are the Application Practices of TinyML?

• What are the Related Organizations of TinyML?

01

What is TinyML?

Machine learning (ML) is becoming increasingly smaller.

As mentioned at the beginning, TinyML is the application of machine learning on microcontrollers, representing ultra-low power edge artificial intelligence.

Data needs to be instantly available anytime and anywhere, and this trend is becoming increasingly apparent. Industries worldwide are undergoing a “digital transformation” driven by this demand. According to IDC’s analysis, by 2025, more than a quarter of the data created globally will be real-time data, with IoT real-time data accounting for over 95% of this data.

The massive influx of data has given rise to a series of new technologies, including machine learning, natural language processing, and artificial intelligence, which transform data analysis from an infrequent, retrospective practice into a proactive driver of strategic decision-making and actions. These technologies can significantly enhance the frequency, flexibility, and immediacy of data analysis across various industries, environments, and applications. According to IDC’s predictions, by 2025, the total amount of global data related to data analysis will grow 50 times to reach 5.2ZB; while the volume of analytical data “touched” by machine learning will grow 100 times to reach 1.4ZB.

Currently, machine learning can be divided into three forms: cloud ML, edge ML, and TinyML. TinyML specifically targets the IoT real-time data processing scenarios that account for over 95%.

Therefore, machine learning is clearly categorized into three types based on the different types of computing platforms and the amount of data created and used at different times, each playing its differentiated role:

• Cloud ML: refers to the application of machine learning in specific computing data centers within enterprises or in the cloud. These cloud servers cover all types, including public cloud, private cloud, and hybrid cloud. It also includes operational control centers, such as those managing power grids or telephone networks.

• Edge ML: refers to the application of machine learning in enterprise-level computers/devices that are not located in core data centers. Edge devices include server rooms, onsite servers, and small data centers located in various regions to accelerate response times.

• TinyML: refers to the application of ultra-low power machine learning in various IoT terminal microcontrollers. TinyML typically operates at a power level of milliwatts (mW) or even lower, making it suitable for various battery-powered devices and applications that need to be always online. These devices include smart cameras, remote monitoring devices, wearable devices, audio capture hardware, and various sensors, etc.

Looking at the current development status of TinyML, from the dimensions of algorithms, software, and hardware, TinyML has evolved to a stage where it is “good enough” for practical applications.

TinyML is an emerging field, a rapidly growing machine learning technology and application, representing a vast and underdeveloped blue ocean.

Billions of microcontrollers and various sensors combined may spark some very creative and more practical TinyML applications in the future.

To summarize TinyML simply:

• What: TinyML refers to ultra-low power (milliwatt level) edge machine learning applications.

• Why: TinyML can enhance the data analysis and decision-making capabilities of numerous IoT devices.

• How: Implementing TinyML requires a holistic collaborative design of hardware, software, and algorithms.

• When: Now is the best time to start laying out TinyML.

TinyML contains enormous opportunities, many of which are just emerging. Significant progress is likely to occur in this field within the next couple of years.

02

What is TinyML as a Service?

Next, let’s take a look at what TinyML as a Service is.

This is a concept proposed by Ericsson.

First, we need to clearly distinguish between two ways of applying machine learning in IoT terminals:

• Providing machine learning services for IoT terminal devices

• Embedding TinyML as a Service in IoT terminal devices

These two expressions may seem similar, but they are actually different.

In the first case, the machine learning services provided for IoT terminal devices generally “outsource” all ML tasks to edge devices and cloud servers, with terminal devices acting as passive receivers, executing various tasks dispatched by edge and cloud.

In the second case, embedding TinyML as a Service in IoT terminal devices allows them to “actively” participate in intelligent decision-making and execution. Although the resources of terminal devices are very limited compared to the cloud and edge, TinyML as a Service can still enhance the analytical capabilities of terminal devices so they can better process real-time IoT data.

Therefore, TinyML as a Service truly brings machine learning to IoT terminals.

At this point, you may have a question: the machine learning ecosystem is very large and resource-intensive. What machine learning tasks can small IoT devices perform?

To better answer this question, let’s first discuss the differences between TinyML and cloud ML, as they exist in two entirely different worlds.

The image above qualitatively and quantitatively compares the differences between TinyML and cloud ML from hardware and software perspectives. Compared to the resources that TinyML can call upon, cloud ML is practically a “rich man.” To successfully advance, TinyML must adopt a different mindset from cloud ML.

As seen in the three-ring diagram at the beginning of this article, there is not much overlap between TinyML and Linux. Many IoT devices do not have the capability to run Linux, and TinyML cannot leverage many mature tools, applications, and container-based virtualization technologies, necessitating a different approach to solve the extreme lack of software, hardware, and algorithms for deploying ML services.

Many companies are making various efforts around TinyML, and in the following sections, we will introduce various TinyML products and the latest practices.

Of course, it is unrealistic to expect to achieve the same experience on terminal IoT devices as with cloud ML. TinyML primarily implements inference, which means applying the capabilities learned from training to actual operations.

TinyML requires a complete and mature architecture to be realized. Ericsson has broken down the TinyML as a Service ecosystem into three basic components:

• Compiler Plugin Interface

• Orchestration Protocol

• Inference Module Specifications

Supporting these three components indicates that IoT hardware can meet the basic requirements for implementing TinyML as a Service. The diagram below further illustrates the interaction between these three components: green represents the compiler plugin interface, yellow represents the orchestration protocol, and the inference module specifications.

Implementing TinyML as a Service also faces tremendous challenges:

• TinyML must not only face all the difficulties of edge ML, but also its problem-solving capabilities are even more limited;

• How to leverage the successful experiences of cloud ML to smoothly advance the deployment of TinyML in terminals;

• How to provide a good experience of TinyML for a vast number of IoT devices.

03

Why is TinyML Important?

By now, you should have formed your own assessment of the importance of TinyML.

Let’s summarize it again.

The emergence of TinyML aims to better alleviate various problems that cannot be overcome in edge ML and cloud ML, including data privacy, network bandwidth, time delay, reliability, and energy efficiency:

• Data Privacy: A large number of end-users are very concerned about data privacy and maintain a cautious attitude towards data openness and sharing. Many users are reluctant to entrust their data to third-party cloud platforms and edge service providers for storage and management. Many users prefer to define clear “local” physical boundaries to preserve their critical production and operational data. TinyML attempts to process and analyze sensitive data directly on IoT devices, protecting data privacy.

• Network Bandwidth: Many IoT devices communicate with the network through narrowband IoT (NB-IoT) or other low-power wide-area IoT communication protocols, resulting in extremely limited bandwidth and data transmission capacity. These devices have a strong need to process data locally to reduce data transmission, alleviate the pressure on network bandwidth and transmission power consumption, and avoid creating bandwidth bottlenecks between terminal and edge devices that could affect the performance of the entire IoT solution.

• Time Delay: With the development of technologies like 5G, a massive number of IoT devices will be deployed, and many application scenarios are very sensitive to time delays, hoping for real-time data transmission. TinyML reduces the likelihood of network delays by transferring certain machine learning tasks to the devices themselves.

• Reliability: In applications in remote areas, offshore platforms, space stations, and extreme environments, network communication may not always be guaranteed. Therefore, having machine learning capabilities in these IoT devices is a necessary function. TinyML can transplant certain machine learning capabilities from edge and cloud to local, enhancing reliability.

• Energy Efficiency: Many IoT devices are battery-powered and have high demands for power consumption. Through ultra-low power TinyML data analysis, the amount of data transmitted over the network can be reduced, which can save battery consumption in IoT terminals to some extent.

Because it has the potential to solve various issues, it is expected to break through cost, bandwidth, and power consumption limitations, and thus, once proposed, TinyML has garnered widespread attention and high expectations.

04

Market Size Forecast for TinyML

A vast number of IoT devices are located at the network edge, and these devices are trending towards being increasingly smaller, so the future development of TinyML has immense imaginative potential.

Currently, there are 250 billion microcontrollers running worldwide, with 28.1 billion units sold in 2018 alone. IC Insights predicts that by 2023, the annual shipment of microcontrollers will grow to 38.2 billion units. Moreover, these microcontrollers correspond to devices that increasingly demand intelligence.

In other words, the 250 billion microcontrollers distributed in smoke sensors, cardiac pacemakers, and vehicle terminals may soon execute tasks previously only handled by computers and smartphones.

According to Silent Intelligence’s forecast, in the next five years, TinyML will trigger over $70 billion in economic value and maintain a compound annual growth rate (CAGR) of over 27.3%.

The market size of TinyML is larger than that of edge ML and cloud ML.

However, such a huge market is relatively difficult to develop. Therefore, TinyML involves collaboration between different companies, and the business models between companies also differ fundamentally. Taking the AI SaaS provider SensiML in the TinyML field as an example, its business model is illustrated in the diagram below.

In January 2019, SensiML, which had been established for six years at the time, was acquired by QuickLogic. According to QuickLogic’s financial report, QuickLogic acquired all of its issued and outstanding common stock through an all-stock transaction, totaling $1,169,752.

05

Latest Products of TinyML

The image above is the edge intelligence industry map compiled by Topio Networks.

Edge ML is a relatively mature field, with many companies refining their work within it, many of which have already begun experimenting with TinyML.

Companies such as ARM, Qualcomm, Google, Bosch, Apple, and Microsoft are attempting to accelerate the implementation of TinyML, equipping it on various sensors.

For example, Qualcomm has launched an ultra-low power always-on computer vision solution.

This solution features ultra-low power, remains always on, using less than 1mA from standard lithium batteries, with a typical frame rate of 1-30 fps.

The corresponding product, Qualcomm® QCC112, is already commercially available.

Bosch has launched MEMS sensors for IoT that enable TinyML applications.

Apple acquired the TinyML startup Xnor.ai for $200 million. Xnor.ai is dedicated to developing efficient, low-power TinyML applications that do not require powerful processing capabilities or cloud connectivity, but rather process data locally on the device.

Recently, ARM announced two chip designs, the ARM Cortex-M55 and Ethos-U55, which enable machine learning capabilities on devices without cloud connectivity.

ARM believes that the launch of these two products will usher in a new era of terminal intelligence.

ARM envisions use cases such as a 360-degree camera in a smart cane that can automatically identify obstacles and report to the owner, or intelligent sensors in high-speed trains that can identify problems on-site and alert in a timely manner to avoid delays.

The introduction of new products will greatly enhance ML inference speed and energy efficiency.

However, the related chips are not expected to be available until early 2021.

In addition to the giants, startups are also very active in the TinyML field.

GreenWaves uses multiple RISC-V cores to achieve TinyML applications at ultra-low power. Its second-generation product, GAP9, features 10 RISC-V cores, one serving as a control unit, while the other nine form a computing cluster. These controllers and computing clusters operate in independent voltage and frequency domains and further reduce power consumption through support for advanced FD-SOL processing technology.

Eta Compute’s ECM3532 is suitable for low-power IoT, featuring two cores, Arm Cortex-M3 and DSP. It enables image processing and sensor aggregation under long standby conditions, with a power consumption of only 100 microwatts. The chip has 512KB of flash memory and 256KB of SRAM, and Eta Compute’s showcased cases include voice, image, and video recognition, as well as applications in industrial sensing scenarios.

06

Application Practices of TinyML

TinyML is still in its early application phase, with initial practices emerging in some fields:

In-Vehicle Applications: Swim.AI uses TinyML during real-time data transmission to effectively enhance the intelligent processing capabilities of sensors regarding real-time traffic data, reducing passenger wait times, decreasing traffic congestion, improving vehicle emissions, and enhancing travel safety.

Smart Factories: In manufacturing, TinyML can reduce unplanned downtime caused by equipment failures by enabling real-time decision-making. It can remind workers to perform preventive maintenance when necessary based on equipment conditions.

QuickLogic’s subsidiary SensiML has made good attempts in this field, as their products can rapidly construct intelligent sensing solutions.

SensiML has also applied TinyML in predictive maintenance scenarios.

For developers of predictive maintenance applications, the advantage of the SensiML intelligent software toolkit is that it enables developers to build intelligent IoT sensing devices in days or weeks without requiring expertise in data science or embedded firmware.

The software includes the SensiML Data Capture Lab, an integrated tool for collecting and organizing training datasets.

For predictive maintenance, the dataset consists of time-series data from various sensors. The SensiML AI software toolkit analyzes labeled data to generate classifier algorithms that can implement actionable anomaly detection. The same tool is then used to compile the algorithms to run on selected hardware targets, such as microcontrollers or QuickLogic’s own QuickAI system-on-chip platform.

Smart Retail: By monitoring store shelves and immediately sending alerts when product quantities fall below a certain level, TinyML can prevent stockouts at the retail end.

Smart Agriculture: Farmers can suffer significant losses due to animal diseases. By utilizing remote monitoring devices equipped with TinyML to monitor livestock heart rates, blood pressure, body temperature, and soil moisture conditions in real-time, farmers can predict and prevent outbreaks of diseases in crops and livestock.

07

Related Organizations of TinyML

No successful technology can thrive without the promotion of core teams and related organizations.

The most active organization in the TinyML field is the TinyML Foundation, a community spontaneously built by engineers, established in July 2019, which frequently conducts small exchanges and holds the TinyML summit annually.

If interested, you can learn more about the TinyML Foundation and the summit by visiting the following website:

https://tinymlsummit.org/

—-In Conclusion—-

The tiny TinyML will grow big; this emerging phenomenon is gradually taking shape and requires collaborative efforts from software, hardware, algorithms, and applications to effectively build a complete ecosystem.

I will continue to track the latest developments in this field.

Summary of This Article:

1.TinyML is the application of machine learning on microcontrollers, representing ultra-low power edge artificial intelligence. TinyML typically operates at a power level of milliwatts (mW) or even lower, making it suitable for various battery-powered devices and applications that need to be always online.

2.It is unrealistic to expect to achieve the same experience on terminal IoT devices as with cloud ML. TinyML primarily implements inference, which means applying the capabilities learned from training to actual operations.

3.TinyML is an emerging field, a rapidly growing machine learning technology and application, representing a vast and underdeveloped blue ocean.

References:

How Engineers Are Using TinyML to Build Smarter Edge Devices

Engineers are Pushing Machine Learning to the World’s Humblest Microprocessors

How can we democratize machine learning on IoT devices?

TinyML as-a-Service: What is it and what does it mean for the IoT Edge?

TinyML as a Service and the challenges of machine learning at the edge

Engineering Tiny Machine Learning for the Edge

AI at the Very, Very Edge

TinyML: The challenges and opportunities of low-power ML applications

TinyML Sees Big Hopes for Small AI

Adapting the Microcontroller for AI in the Endpoint

AI for Edge landscape

Learn more about the detailed agency policy by clicking on the blue text below

“Recruiting Agents for National IoT Traffic Card Management Platform”

[Wu Nü Heart Sutra] Previous Selections

Why Edge Computing Will Flourish in 2020?

Why Did These Once Certain IoT Projects Ultimately Fail?

After a Year of Preparation, Siemens Unveils the “Ace” for Accelerating the Implementation of the Industrial Internet

Huawei Establishes Smart Car BU, Alibaba Invests 3.6 Billion in Qianfang Technology, Giants Enter the V2X Field with Different Paths

Why Do Tech Giants Like Microsoft, Google, and Amazon Ultimately Converge on AIoT?

Huawei/Alibaba/China Unicom’s Most Detailed IoT Strategy Analysis Ever