Arm Technology held a product launch conference in Shanghai, officially introducing the new generation “Zhouyi” X3 NPU. This is the first heavyweight product under its Arm China “All in AI” strategy, targeting four major fields: infrastructure, smart vehicles, mobile terminals, and smart IoT, aiming to provide efficient large model inference and multi-scenario AI acceleration capabilities at the edge.
– Positioning and Architecture of X3 NPU
– Based on a DSP+DSA general architecture designed for large models, it accommodates both Convolutional Neural Networks (CNN) and Transformer models, addressing the challenges of running large models at the edge.
– A single cluster can be configured with up to 4 cores, with an FP8 total computing power range of 8 to 80 TFLOPS, and a single core bandwidth of up to 256 GB/s, providing high scalability and flexibility.
– Key Innovations
– Self-developed decompression hardware WDC: Lossless compression and decompression of large model weights, achieving an additional bandwidth improvement of 15-20%.
– Edge W4A8/W4A16 computing acceleration modes: Low-bit quantization of weights significantly reduces bandwidth consumption, supporting efficient migration from cloud to edge.
– AIFF and dedicated hardened scheduler: Achieves extremely low CPU load (approximately 0.5%) and low scheduling latency, supporting multi-tasking and high-priority scheduling.
– Multi-precision fusion computing: Supports various precisions including INT4/8/16/32, FP4/8/16/32, BF16, covering multi-scenario needs from mobile edge to data center level.
– Comparison with Previous Generation and Performance Highlights
– Compared to the Zhouyi X2, CNN performance has significantly improved, with multi-core computing power linearity reaching 70-80%; at the same computing power, AIGC large model capabilities have made a substantial leap.
– In actual tests (such as Llama2 7B), the X3 NPU achieved approximately 72% computing power utilization during the Prefill phase, and effective bandwidth utilization exceeded 100% during the Decode phase, demonstrating high throughput potential for large model inference at the edge.
– Role of the Compass AI Software Platform
– Provides an end-to-end software toolchain to enhance development and deployment efficiency. The NN Compiler component covers Parser, Optimizer, GBuilder, AIPULLM, etc., supporting mainstream frameworks (TensorFlow, ONNX, PyTorch, Hugging Face) and over 160 operators and 270 models.
– One-click deployment on Hugging Face, dynamic shape support, various quantization schemes, and high-performance support for models like LLM/VLM/MoE, lowering development barriers and accelerating implementation.
– Cross-platform and heterogeneous computing: Supports operating systems such as Android, Linux, RTOS, QNX, utilizing TVM/ONNX for SoC heterogeneous computing, coordinating CPU/GPU/NPU scheduling.
– Ecosystem and Implementation Scenarios
– Solutions cover AIoT, smart cockpits, ADAS, smart robots, AI PCs/mobile phones, smart gateways, smart security, etc., emphasizing efficient and low-latency inference at the edge.
– The product matrix across generations (such as Z1, Z2/Z3, X1, X2, X3) unfolds around different computing powers and application levels, forming a complete edge AI ecosystem.
– Market Significance
– As a domestic chip IP provider continuously advancing the boundaries of AI computing, the X3 NPU, together with the Compass platform, establishes a new standard for large model inference at the edge, promoting the realization of “edge-cloud collaboration” and localized intelligence.
– Risks and Considerations
– Attention is still needed regarding industry standardization, ecosystem compatibility, driver maturity, and stability of mass production supply. Before the official launch of the new product, the industry should focus on a comprehensive assessment of actual power consumption, heat dissipation, driver, and deployment experience.