Open-Source Inference Engines

Currently, the mainstream and active open-source TinyML inference engines with over 1k stars on GitHub provide core support for implementing neural network model inference on MCUs.

• Arm CMSIS-NN/DSP (CMSIS-6)

• A function library designed specifically for Arm Cortex-M cores, providing efficient neural network (NN) and digital signal processing (DSP) core functions.
• https://github.com/ARM-software/CMSIS_6

• Google LiteRT (TensorFlow Lite)

• A lightweight runtime launched by Google, which is an important component of TensorFlow Lite.
• https://github.com/google-ai-edge/litert

• Google TensorFlow Lite for Microcontrollers (TFLite-Micro)

• The official version of TensorFlow Lite designed for MCUs, serving as a benchmark in the TinyML field.
• https://github.com/tensorflow/tflite-micro

• PyTorch ExecuTorch

• A next-generation inference solution for edge devices launched by PyTorch, featuring high portability.
• https://github.com/pytorch/executorch

• PyTorch QNNPack

• A high-performance function library designed for quantized neural networks, often integrated with PyTorch Mobile.
• https://github.com/pytorch/qnnpack

• Alibaba MNN

• A high-performance, lightweight deep learning inference engine open-sourced by Alibaba, widely supporting various devices.
• https://github.com/alibaba/MNN

• Alibaba TinyNeuralNetwork

• A toolchain focused on model conversion and deployment for TinyML.
• https://github.com/alibaba/TinyNeuralNetwork

• Tencent TNN

• A high-performance inference framework open-sourced by Tencent, also suitable for MCU and other edge scenarios.
• https://github.com/Tencent/TNN

• Tencent NCNN

• A high-performance neural network forward computation framework open-sourced by Tencent, deeply optimized for mobile platforms.
• https://github.com/Tencent/ncnn

• uTensor

• An extremely lightweight machine learning inference framework designed for MCUs.
• https://github.com/uTensor/uTensor

• Fraunhofer-IMS AIfES

• An AI engine from the Fraunhofer-IMS Institute in Germany, specifically providing support for the Arduino platform.
• https://github.com/Fraunhofer-IMS/AIfES_for_Arduino

• Sipeed TinyMaix

• A minimalist AI inference framework open-sourced by Sipeed, designed for MCUs with very small memory.
• https://github.com/sipeed/TinyMaix

• XiaoMi MACE

• A mobile deep learning prediction framework open-sourced by Xiaomi, emphasizing heterogeneous computing acceleration.
• https://github.com/XiaoMi/mace

• PaddlePaddle Paddle-Lite

• A lightweight inference engine launched by Baidu PaddlePaddle, supporting various hardware including Arm Cortex-M.
• https://github.com/PaddlePaddle/Paddle-Lite

• Eloquent EloquentMicroML

• A machine learning function library that is very friendly to Arduino developers.
• https://github.com/eloquentarduino/EloquentMicroML

• Kraiskil ONNX2C

• A utility that can directly convert ONNX models into C language code.
• https://github.com/kraiskil/onnx2c

• Huawei MindSpore

• A full-scenario AI framework open-sourced by Huawei, with a lightweight version supporting deployment on terminal devices.
• https://github.com/mindspore-ai/mindspore

• SenseTime OpenPPL (PPLNN)

• A high-performance AI inference computation function library open-sourced by SenseTime.
• https://github.com/OpenPPL/ppl.nn

• Open AI Lab Tengine

• An AIoT application inference framework open-sourced by OPEN AI LAB, designed for edge computing.
• https://github.com/OAID/Tengine

AI-Related Compilers

Compilers play a key role in converting high-level AI models into executable code for MCUs, maximizing hardware performance.

• MicroTVM

• A micro version of Apache TVM that can compile models to bare-metal environments without an operating system.
• https://tvm.hyper.ai/docs/topic/microtvm/

• Skymizer ONNC

• An open-source neural network compiler dedicated to converting ONNX models to various AI accelerators.
• https://github.com/ONNC/onnc

• Arm Ethos-U Vela Compiler

• An offline compiler designed for the Arm Ethos-U series NPU, used to optimize neural network models.
• https://pypi.org/project/ethos-u-vela/

MCU-Specific Tools & Platforms

Major MCU manufacturers have also launched their own AI development kits and tools to simplify the development process on their hardware platforms.

• Edge Impulse

• A leading online TinyML development platform that provides a complete workflow from data collection, model training to deployment.
• https://docs.edgeimpulse.com/docs

• STM32CubeAI (X-CUBE-AI)

• An official tool from STMicroelectronics (ST) that converts pre-trained models into optimized code for STM32 microcontrollers.
• https://stm32ai.st.com/stm32-cube-ai/

• NanoEdge AI Studio

• Another powerful tool from ST that allows direct training and inference of machine learning models on MCUs.
• https://stm32ai.st.com/nanoedge-ai/

• ST Edge AI Suite

• An integrated edge AI solution suite launched by ST.
• https://www.st.com/content/st_com/en/st-edge-ai-suite.html

• NXP eIQ

• NXP’s AI development environment that provides software, tools, and inference engines, supporting multiple processors.
• https://www.nxp.com/design/design-center/software/eiq-ai-development-environment:EIQ

• Renesas e-AI

• Renesas’ e-AI development environment that allows easy import of AI models into its MCU product line.
• https://www.renesas.com/en/key-technologies/artificial-intelligence/e-ai/e-ai-development-environment-microcontrollers