With the evolution of artificial intelligence (AI), executing AI workloads on embedded devices using small language models (SLM) has become a focal point in the industry. Small language models such as Llama, Gemma, and Phi3 have gained widespread recognition for their excellent cost-effectiveness, high efficiency, and ease of deployment on resource-constrained devices. Arm expects the number of such models to continue to grow by 2025.
Arm’s technology, with its significant advantages in high performance and low power consumption, provides an ideal operating environment for small language models, effectively enhancing operational efficiency and further optimizing user experience. To visually demonstrate the immense potential of endpoint AI in the Internet of Things (IoT) and edge computing, the Arm technology team recently created a technical demonstration. In this demonstration, when a user inputs a sentence, the system expands it to generate a children’s story. This demonstration was inspired by Microsoft’s “Tiny Stories” paper and Andrej Karpathy’s TinyLlama2 project, which trained a small language model to generate text using 21 million stories.
The demonstration is powered by the Arm Ethos-U85 NPU and runs small language models on embedded hardware. Although large language models (LLM) are more widely known, small language models are gaining attention for their ability to deliver excellent performance with fewer resources and lower costs, as well as being easier and cheaper to train.
Implementing Transformer-based Small Language Models on Embedded Hardware
Arm’s demonstration showcases the Ethos-U85 as a small, low-power platform capable of running generative AI, highlighting the outstanding performance of small language models in specific domains. The TinyLlama2 model is simplified compared to large models from companies like Meta, making it well-suited to demonstrate the AI capabilities of the Ethos-U85, serving as an ideal choice for endpoint AI workloads.
To develop this demonstration, Arm conducted extensive modeling work, including creating a fully integer INT8 (and INT8x16) TinyLlama2 model and converting it to a fixed-shape TensorFlow Lite format suitable for the constraints of the Ethos-U85.
Arm’s quantization method demonstrates a good balance between achieving high accuracy and output quality with fully integer language models. By quantizing activations, normalization functions, and matrix multiplications, Arm eliminates the need for floating-point operations. Since floating-point operations are costly in terms of chip area and energy consumption, this is a critical consideration for resource-constrained embedded devices.
The Ethos-U85 runs the language model at a frequency of 32 MHz on an FPGA platform, achieving a text generation speed of 7.5 to 8 tokens per second, comparable to human reading speed, while consuming only a quarter of the computational resources. In actual application systems on a system-on-chip (SoC), this performance can be improved by up to ten times, significantly enhancing the processing speed and energy efficiency of edge AI.
The children’s story generation feature uses the open-source version of Llama2 and combines it with the Ethos NPU backend, running the demonstration on TFLite Micro. Most of the inference logic is written in C++ at the application layer, and by optimizing the context window content, the coherence of the stories is improved, ensuring that the AI can narrate the stories smoothly.
Due to hardware limitations, the team needed to adapt the Llama2 model to ensure its efficient operation on the Ethos-U85 NPU, requiring careful consideration of performance and accuracy. The INT8 and INT16 mixed quantization techniques demonstrate the potential of fully integer models, encouraging the AI community to more actively optimize generative models for edge devices and promote the widespread application of neural networks on high-efficiency platforms like the Ethos-U85.
Arm Ethos-U85 Demonstrates Exceptional Performance
The Ethos-U85’s multiply-accumulate (MAC) units can scale from 128 to 2,048, achieving a 20% improvement in energy efficiency compared to the previous generation Ethos-U65. Additionally, a significant feature of the Ethos-U85 compared to its predecessor is its native support for Transformer networks.
Partners using the previous generation Ethos-U NPU can achieve seamless migration and fully leverage their existing investments in Arm architecture-based machine learning (ML) tools. With its outstanding energy efficiency and performance, the Ethos-U85 is increasingly favored by developers.
If configured with 2,048 MACs on-chip, the Ethos-U85 can achieve 4 TOPS of performance. In the demonstration, Arm used a smaller configuration, employing 512 MACs on the FPGA platform and running the TinyLlama2 small language model with 15 million parameters at a frequency of 32 MHz.
This capability highlights the potential of embedding AI directly into devices. Despite limited memory (320 KB SRAM for caching and 32 MB for storage), the Ethos-U85 can efficiently handle such workloads, laying the foundation for the widespread application of small language models and other AI applications in deeply embedded systems.
Bringing Generative AI to Embedded Devices
Developers need more advanced tools to tackle the complexities of edge AI. Arm is committed to meeting this demand by launching the Ethos-U85 and supporting Transformer-based models. As the importance of edge AI in embedded applications continues to grow, the Ethos-U85 is driving the realization of various new use cases, from language models to advanced visual tasks.
The Ethos-U85 NPU provides the exceptional performance and energy efficiency required for innovative frontier solutions. Arm’s demonstration shows significant progress in bringing generative AI to embedded devices and highlights the feasibility of deploying small language models on the Arm platform.
Arm is creating new opportunities for edge AI across a wide range of applications, making the Ethos-U85 a key driver in the development of the next generation of intelligent, low-power devices.