Recently, the hardware platform equipped with Arm Technology’s latest generation ‘Zhouyi’ NPU processor successfully ran the DeepSeek-R1 series models, delivering outstanding performance and excellent cost-effectiveness, providing users with a more efficient and convenient AI application experience. This innovative NPU processor adopts an architecture design optimized for large model characteristics, and its beta version has been made available for evaluation testing to early users by the end of 2024, receiving widespread recognition and positive feedback. It is expected that this highly anticipated NPU product will officially debut in the market in the first half of this year, bringing groundbreaking edge computing power experiences to more users.
Since its release, DeepSeek has quickly become the focus of the AI field due to its outstanding performance and low-cost training model. After the launch of the 1.5B and 7B distilled versions of DeepSeek-R1, Arm Technology’s new generation ‘Zhouyi’ NPU processor completed deployment and optimization on the Emulation platform in a short time, and successfully demonstrated end-to-end applications on the FPGA platform. After rigorous testing and validation, the new generation ‘Zhouyi’ NPU processor performed excellently when running the 1.5B and 7B distilled versions of DeepSeek-R1. In a testing environment with standard single batch input and a context length of 1024, its computing power utilization rate exceeded 40% during the first word computation phase, and the effective bandwidth utilization rate during the decoding phase reached over 80%. Its bandwidth utilization exhibited high linear characteristics, flexibly adapting to system bandwidth requirements from 16GB/s to 256GB/s. In the case of the 7B version with a context length of 1024, this processor can achieve a maximum processing speed of 40 tokens/s while fully ensuring model application accuracy, and it supports dynamic length model inference input. When facing complex AI tasks, the ‘Zhouyi’ NPU processor demonstrates significant technical advantages with its exceptional computing performance, high bandwidth utilization, and energy efficiency, providing strong ‘core’ power for the intelligent upgrade of terminal devices.
Demonstration of the new generation ‘Zhouyi’ NPU processor running the DeepSeek-R1 model
The efficient adaptation and tuning of the new generation ‘Zhouyi’ NPU processor for the latest large models like DeepSeek benefit from its software stack’s mature support for large models. The software stack provides efficient quantization compilation tools that can significantly compress model size while maintaining high-performance inference capabilities, and it meets diverse needs through flexible compilation options. Additionally, the software stack has undergone deep optimization for large models, including dynamic inference optimization and the exploration of hardware computing power potential, significantly enhancing inference speed and throughput. Currently, the software stack supports various mainstream large models such as Llama, Qwen, DeepSeek, ChatGLM, and MiniCPM, and provides a toolchain for integration with the Hugging Face model library, facilitating direct deployment of mainstream models by users. At the hardware level, the new generation ‘Zhouyi’ NPU processor breaks through traditional design limitations, increasing external bandwidth to 256GB/s, effectively solving the bandwidth bottleneck problem for large model computations. To meet the high precision requirements of edge large models, this NPU processor fully supports FP16 computation and provides a complete int4 soft and hard quantization acceleration solution, ensuring efficient and stable model operation. In response to the low first word latency requirements of edge models, the ‘Zhouyi’ NPU processor achieves efficient multi-core computing power expansion through hardware-software collaboration optimization. Under a 7nm process technology, single cluster computing power can reach up to 80 TOPS, easily handling deployment requirements for large models with over 16K context lengths while meeting the high computing power demands of multi-modal scenarios and chain-of-thought computations. In addition, this NPU processor also has strong multi-task parallel processing capabilities, achieving flexible switching between multiple tasks through fine-grained task scheduling and priority resource allocation, ensuring efficient collaboration between traditional voice, visual tasks, and large model applications, providing users with a smoother experience.
Currently, Arm Technology is actively expanding the application scenarios of the DeepSeek series models in edge acceleration cards, accelerating the commercialization process of edge AI applications by enhancing model inference performance. Relying on strong technological accumulation, ecosystem advantages, and forward-looking layout, Arm Technology will continue to promote the deep application and innovation of AI large model technology across multiple terminal fields such as PCs, smartphones, smart cars, robots, and wearable devices, comprehensively accelerating the construction and improvement of the edge AI ecosystem.
Follow 【Electronic Engineering Network】 for more information on the electronics industry.