On October 28, the high-performance heterogeneous inference framework KTransformers, co-open-sourced by Qujing Technology and the Tsinghua KVCache.AI team, announced the completion of comprehensive support for the Ascend NPU.
Through collaboration, this update has deeply optimized the CMake build system, stream management, and underlying operator implementation, enabling developers to efficiently run large models with hundreds of billions of parameters, such as single Ascend card + Kunpeng CPU.
Real-world testing on the Huawei Atlas 300I A2 inference card shows that when running the DeepSeek-R1 671B large model, the single card single concurrent Decode speed reaches 14.9 tokens/s.
Efficient Integration of Heterogeneous Collaborative Inference and Ascend Hardware
Through several system-level optimizations, KTransformers achieves significant performance improvements and resource usage optimizations in large model inference, reducing memory usage by over ninety percent:
CPU-NPU Heterogeneous Collaboration: Implementing precise load distribution strategies based on computational intensity, offloading parameters of routing expert layers with lower computational intensity in mixed expert models to the larger CPU memory, while retaining the highest density multi-layer potential attention layers on the Ascend NPU for execution;
NUMA Optimization: Optimizing local memory allocation and thread scheduling for multi-NUMA architectures to reduce access latency;
Specialized Acceleration for Mathematical Libraries: Combining the Kunpeng Mathematical Library (KML) for specialized acceleration of large model matrix multiplication;
Expert Delay Computation: Utilizing expert delay computation technology to effectively overlap communication and computation processes, enhancing hardware utilization.
KTransformers has previously gained widespread community attention on GitHub, and this update marks the first complete adaptation of the Ascend NPU computing solution, providing a high-performance, low-threshold inference solution for the AI software and hardware ecosystem, as well as offering developers a more flexible and efficient new choice for heterogeneous inference.
Deployment Documentation and ReportsDeployment Documentation:
https://github.com/kvcache-ai/ktransformers/blob/main/doc/zh/DeepseekR1_tutorial_zh_for_Ascend_NPU.md
Performance Verification Report:
https://github.com/kvcache-ai/ktransformers/pull/1525
