When it comes to AI, most people probably first think of large language models (LLM) and generative AI (genAI, AIGC) applications, which can chat, query data, and generate articles, images, and music, and most of these applications require massive cloud computing power to complete.
For micro AI applications, such as voice wake-up (voice commands), anomaly detection (vibration, unusual sounds, environmental sensors), motion detection (gestures, falls), image classification, image object detection, and image pose detection (full body, fingers), it is often possible to utilize a single chip (MCU) or microprocessor (MPU) in conjunction with smaller AI models to perform inference work at the edge (offline).
As mentioned in previous articles, AI inference mainly involves massive matrix multiply-accumulate operations (MAC), which is “a×b+c“. The acceleration calculation methods can include “increasing operational frequency speed,” “parallel/vector instruction set acceleration,” “multi-core acceleration,” and “NPU neural network accelerators” and other hardware acceleration methods.
In general, for a single-chip with a single-core CPU, the easiest methods to achieve are the first two, and this article will explain how to successfully implement micro AI applications without a neural network accelerator NPU from the perspective of “parallel/vector instruction set acceleration.”
1. Evolution of Arm’s Reduced Instruction Set
Currently, about 80-90% of commercially available 32-bit single chips are completed using the Arm Cortex-M series silicon (IP), with different series corresponding to different instruction sets (ARMv6-M~v8.1-M), where “-M” refers to the version of the instruction set that has been condensed specifically for single-chip use. The corresponding instruction sets are as follows:
ARMv6-M: Cortex M0(2009) / M0+(2012) / M1(2007)
ARMv7-M: Cortex M3(2004)
ARMv7E-M: Cortex M4(2010) / M7(2014)
ARMv8-M Baseline: Cortex M23(2016)
ARMv8-M Marnline: Cortex M33(2016) / M35P(2018)
ARMv8.1-M: Cortex M55(2020) / M85(2022) / M52(2023)
Fig. 1 Arm Cortex-M series instruction set comparison chart.
2. How Cortex-M Instructions Accelerate Calculation
The “v7-M” instruction set began to support single-cycle multiply-accumulate instructions MLA, which combines the original two instructions MUL, ADD for multiplication and addition into one instruction, thus doubling the calculation speed. With the “v7E-M” DSP extension instruction set, it began to support single instruction multiple data streams (Single Instruction Multiple Data, SIMD), such as QADD, QADD16, QADD8, which can split 32bit into 2 16bit or 4 8bit data to be computed together, thus increasing the calculation speed by 2~4 times.
When paired with operational frequencies increasing from tens of MHz to hundreds of MHz, micro AI applications such as voice wake-up words (Key Word Spotting, KWS), sensor anomaly detection (Anomaly Detection, AD), and motion gesture recognition (Motion / Gesture Detection) can generally be completed, but for image-related applications, it may still be insufficient.
The Arm Cortex-A series is a 64bit CPU, so it has NEON (advanced SIMD) instruction set, which can handle 128bit or 64bit SIMD calculations, making calculations faster, but such instructions are not supported in the 32bit Cortex-M, so Arm introduced the Helium M vector extension instruction (M-Profile Vector Extension, MVE) in the v8.1M instruction to enhance the computational power of Cortex-M single chips.
Helium can handle 128bit vector operations (equivalent to SIMD 16x8bit, 8x16bit, 4x32bit), with a total of 8 vector registers, capable of processing integers, fixed-point numbers (Q7, Q15, Q31), and floating-point numbers (half precision FP16/single precision FP32) calculations, with over 150 new instructions.
To process 128bit vector calculations, changing the data bus (Data bus) and the internal static memory (SRAM) of the single chip to 128bit width is unreasonable. The simplest method is to use four clock cycles to treat 128bit data as 4 32bit data to process, but this would make it no different from the single instruction cycle 32bit SIMD instructions. Therefore, Helium allows overlapping loading (VLDR) and multiply-accumulate (VMLA) calculations, loading and computing simultaneously, thus accelerating by more than double, as shown in Fig. 2.
In Cortex-M55/M85, a dual-beat approach is adopted, which means that 1 clock cycle reads two beats, i.e., 2×32=64bit, so 128bit vectors require 2 clock cycles to process, while the contents processed in one clock cycle can be one of the following combinations:
2 32bit (Q31 fixed-point or 32-bit integer or 32-bit floating-point)
4 16bit (Q15 fixed-point or 16-bit integer or 16-bit floating-point)
8 8bit (Q7 fixed-point or 8-bit integer)
In contrast, Cortex-M52 adopts a single-beat approach, which reduces the processing length to 32bit (1x32bit, 2x16bit, 4x8bit), requiring 4 clock cycles to process 128bit vectors.
Fig. 2 Arm Cortex-M instruction and acceleration calculation diagram.
3. Performance Comparison of Helium Instruction Set
Currently, there are four companies that support Helium instructions in their MCUs, as follows:
Cortex-M55:
ALIF – Ensemble E1(@160MHz), E3(@160MHz), E5(dual core,@160MHz, 400MHz), E7(dual core,@160MHz, 400MHz)
Himax – WiseEye2 HX6538(dual Core, @150MHz, 400MHz)
Nuvoton – NuMicro M55M1(@200MHz)
Cortex-M85:
Renesas – RA8D1(@480MHz), RA8M1(@480MHz)
Cortex-M52:
None
Below is a comparison of Cortex-M7 as a baseline (100%) against Cortex M55/M85/M52. As shown in Fig. 3, the horizontal axis represents traditional performance (DMIPS/MHz), while the vertical axis represents machine learning inference performance (ML Performance). For example, although the traditional performance of M55 is lower than that of M7, its machine learning performance is three times higher, illustrating the performance improvement offered by the Helium instruction set.
Fig. 3 Arm Cortex-M instruction and acceleration calculation diagram.
The above computing power can generally meet the needs of non-image micro AI applications, but for image-related applications, relying solely on the CPU’s SIMD and MVE instruction sets may still be insufficient. Fortunately, the aforementioned ALIF, Himax, Nuvoton MCUs all come with built-in Arm Ethos U55 NPU, which can increase computational power by several dozen times, meeting the needs for low-resolution, low-frequency image classification, object detection, and even pose estimation applications.
Conclusion
With the advancement of semiconductor technology and the gradual availability of supporting software development tools, utilizing single chips to increase operational frequency and highly parallel multiply-accumulate operations (SIMD, MVE) has opened up opportunities to perform micro AI computations without relying on the network, allowing for more creative applications in affordable, low-power edge intelligent devices (Edge AI & TinyML Device), enabling rapid realization of applications in smart living, intelligent care, smart manufacturing, smart buildings, and more.
==========================
