A recent research conclusion from NVIDIA has garnered significant attention in the industry— Small Language Models (SLM) are the future of intelligent agents. Following this, NVIDIA introduced its new small language model:Nemotron-Nano-9B-V2, which achieved the highest performance among its peers in certain benchmark tests.
In fact, the trend of small language models (SLM) has also reached the fields ofMCUs and MPUs.
Small models are “compressed” large models
Small language models (SLM) may already be familiar to us.SLM parameters range from millions to billions, while LLM can have hundreds of billions or even trillions of parameters.
SLM is derived from compressing LLM, and the compression process aims to retain the model’s accuracy while reducing its size. Common methods include:
-
Knowledge Distillation: Training a smaller “student” model using knowledge transferred from a larger “teacher” model;
-
Pruning: Removing redundant or less important parameters from the neural network architecture;
-
Quantization: Reducing the numerical precision used in computations (e.g., converting floating-point numbers to integers).
Small language models are more compact and efficient than large models. Therefore,SLM requires less memory and computational power, making it ideal for resource-constrained edge or embedded devices.
Many small yet powerful language models have emerged, proving that size is not everything. Common SLMs ranging from 1 billion to 4 billion includeLlama3.2-1B (a 1 billion parameter variant developed by Meta), Qwen2.5-1.5B (Alibaba’s 1.5 billion parameter model), DeepSeek-R1-1.5B (DeepSeek’s 1.5 billion parameter model), SmolLM2-1.7B (HuggingFaceTB’s 1.7 billion parameter model), Phi-3.5-Mini-3.8B (Microsoft’s 3.8 billion parameter model), and Gemma3-4B (Google DeepMind’s 4 billion parameter model).
Running SLM relies not only on computational power
For MPUs, running SLM seems to be a non-issue. However, for developers, how can they determine whether an MCU supports running generative AI?
This question does not have a single, straightforward answer— however, there is a hard requirement that the MCU must have a Neural Processing Unit (NPU) capable of accelerating Transformer operations.
In addition, running generative AI requires the MCU to have a bandwidth system bus and a large-capacity, high-speed, tightly coupled memory configuration.
In fact, many people now only use GOPS (billion operations per second) or TOPS (trillion operations per second) to compare the raw throughput of microcontrollers. Currently, the best-performing MCU can provide up to 250 GOPS of computing power, while MCUs used for generative AI will need to provide at least double that performance. However, raw throughput is not an ideal metric for measuring actual system performance.
Because successful generative AI applications require support for Transformer computations, which will transfer large amounts of data between the system’s internal memory, neural processing unit, central processing unit, and peripheral functions such as image signal processors. Therefore, a system with high raw throughput may theoretically process large amounts of data quickly, but if the system cannot quickly transfer data to the neural processing unit, the actual performance will be very slow and disappointing.
Of course, for MPUs, high bandwidth, memory, and tight coupling between buses are also crucial.
Aizip has partnered with Renesas on an SLM project
As early as last August, Aizip collaborated with Renesas to showcase ultra-efficient SLM and compact AI Agents for edge system applications on MPUs. These small and efficient models have been integrated into the Renesas RZ/G2L and RZ/G3S boards based on Arm Cortex-A55.
Aizip has created a series of ultra-efficient small language models (SLM) and artificial intelligence agents (AI Agents), named Gizmo, ranging from 300 million to 2 billion parameters. These models support various platforms, including MPUs and application processors for a wide range of applications.
SLM enables AI agents on edge applications to provide functionalities similar to large language models (LLM), but in a compact form factor. The models on devices offer enhanced privacy protection, resilience, and cost savings. While some companies have successfully reduced the size of mobile language models, ensuring accurate tool invocation for automation applications on low-cost edge devices remains a significant challenge for these SLMs.
Reportedly, these SLMs can achieve response times of less than 3 seconds on a single A55 core of the RZ/G2L running at a frequency of 1.2 GHz.
MCUs are also increasing their investment in SLM.
Alif Semiconductor recently released its latest series of MCUs and fusion processors— Ensemble E4, E6, and E8—primarily targeting the operation of generative AI models, including SLM.
Meanwhile, Alif is the first chip supplier to use the Arm Ethos-U85 NPU (Neural Processing Unit), which supports Transformer-based machine learning networks.
Benchmark results show that this series can perform high-energy efficiency object detection in less than 2 milliseconds, image classification in under 8 milliseconds, and the SLM executed on E4 devices consumes only 36mW of power when generating text to build stories based on user prompts.
Ensemble E4 (MCU) features dual Arm Cortex-M55 cores, while Ensemble E6 and E8 fusion processors are based on Arm Cortex-A32 cores and dualCortex-M55 cores, respectively. Notably, E4/E6/E8 are equipped with dual Ethos-U55+Ethos-U85, providing powerful computing capabilities.
Alif believes that they have an earlier layout compared to other manufacturers, as the first generation of Ensemble MCU series was released as early as 2021, and since then, they have been shipping E1, E3, E5 and E7 devices in volume. While other MCU manufacturers are still stuck on the first generation AI MCU, Alif has released second-generation products, and is also the industry’s first MCU to support Transformer-based networks, which are the foundation for LLM and other generative AI models.
SLM will be the future of embedded systems
SLM significantly compresses model size while retaining accuracy as much as possible. This efficient and compact characteristic makes it a perfect fit for resource-constrained edge and embedded devices, bringing unprecedented intelligent capabilities to these devices.
In fact, the future landscape of edge AI is gradually unfolding, and SLM will also be one of the key areas that MCU and MPU manufacturers will focus on.
For example, STMicroelectronics’ STM32N6, Infineon’s PSoC Edge latest generation MCUs, TI’s AM62A and TMS320F28P55x, NXP’s i.MX RT700 and i.MX 95, and ADI’s MAX7800X are all beginning to emphasize NPU.
Embedded AI was initially a feature of relatively expensive microprocessor-based products running on Linux systems. However, the market quickly realized that there is also space for AI in edge and endpoint devices—many of which are based on MCUs. Therefore, by the second half of 2025, advanced MCU manufacturers will incorporate products with AI capabilities into their product lines. These manufacturers’ NPU can be divided into Arm Ethos IP faction and self-developed faction. Currently, the latest Ethos-U85 has begun to support Transformer, and demonstrated the effects of SLM six months ago, with other manufacturers continuously following suit. In the future, SLM is expected to completely change the landscape of MCU and MPU.
References
[1]IBM：https://www.ibm.com/cn-zh/think/topics/small-language-models
[2]Hugging-Face：https://hugging-face.cn/blog/jjokah/small-language-model
[3]Alif：https://alifsemi.com/comparing-mcus-for-generative-ai-its-not-just-about-the-gops/
[4]Alif：https://alifsemi.com/who-wins-in-the-race-to-make-ai-mcus/
[5]Arm：https://newsroom.arm.com/blog/small-language-model-generative-ai-edge
· END ·
Follow the subscription account of EEWorld: “Robot Development Circle”
Reply “DS” to receive the complete version of “DeepSeek: From Beginner to Expert”
Scan the code to add the assistant and reply “robot”
to join the group and exchange experiences face-to-face with electronic engineers
