Ultra Lightweight! AI Model Compiler MegCC Open Source Reduces Inference Engine Size

Currently, there are quite a few mobile deep learning inference frameworks in the community (such as NCNN, MNN), which provide considerable convenience for community users to deploy deep learning on mobile devices. However, these inference frameworks share a common problem: as they continue to iterate and optimize performance, the runtime libraries gradually increase in size. Especially when different operators are fused, it leads to a large number of long-tail operators, making the App or SDK size bloated.

To address this issue, the MegEngine team has open-sourced MegCC, which innovatively uses a model pre-compilation scheme to generate the necessary code for model inference while removing code unrelated to model inference, thus significantly reducing the size of the inference engine. Its main approach isto move all necessary steps of traditional framework runtime, such as computation graph optimization, kernel selection, and memory allocation, into the compilation process, thereby minimizing the binary size during runtime and further optimizing performance based on model information.

• The size of the inference engine will no longer increase with the iteration of the framework.
• Operator fusion can generate corresponding code at compile time based on model information.
• The entire computation graph information can be obtained during model compilation for further extreme performance optimization.
• Can absorb community experiences in code generation to help generate code for MegCC.

1. Model Compilation: Compile the MegEngine model to generate the corresponding kernel and the optimized model.
2. Runtime Compilation: This stage compiles the runtime and the kernel generated in the previous step into a static library.
3. Integration into the Application: Call the interface of the static library compiled in the previous step for inference.

• Support ONNX models as input
• More kernel fusion
• Support more backend devices