The popularity of large-scale language models like ChatGPT and GPT-4 has quickly ignited public enthusiasm for artificial intelligence, drawing strong attention from the industry towards AI chips. Compared to general-purpose chips like CPUs and GPUs, NPUs (Neural Processing Units) can handle AI workloads with simpler control flows, higher efficiency, and lower power consumption. With the development of the AI technology stack and industry applications, NPUs support an increasing number of models, including the Transformer model used by GPT-4, and are expanding from AIoT to consumer electronics, autonomous driving, and more fields.
On March 28, Arm Technology released its latest generation NPU product “Zhouyi” X2, which, along with the previously announced “Zhouyi” NPU software open-source plan, will provide hardware and software support for the innovative applications of artificial intelligence, especially in high-computing, high-precision, and real-time scenarios like autonomous driving.
The main functional upgrades of the “Zhouyi” X2 NPU include:
Multi-core, high computing power, and multi-precision to empower AI applications across multiple scenarios.
As Moore’s Law slows down, general-purpose processor architectures struggle to meet the intensive computing demands of artificial intelligence. Data shows that from 2000 to 2004, the performance of chips improved at an annual rate of 48% per dollar spent, while after 2008, this rate has dropped to less than 10%. In this trend, dedicated chips are becoming a new approach to enhance computing power and energy efficiency for specific needs.
Among them, NPUs are dedicated acceleration chips designed for machine learning and artificial intelligence. Compared to CPUs and GPUs, NPUs simulate human neurons and synapses at the circuit level, optimizing for the vast number of neural network models involved in AI computation, enabling them to handle artificial neural networks, random forests, and deep learning models with higher efficiency and lower energy consumption.
Today, many smartphone manufacturers, including Apple, Samsung, and Oppo, have integrated NPUs for tasks such as facial recognition, object and environment recognition, and image processing, providing consumers with new experiences like 3D animated emojis, face unlocking, AI scene recognition, and lossless real-time RAW computation. As AI penetrates more terminals like tablets and desktops, and more fields such as home and automotive applications, the era of “Everything Connected” begins, which not only tests the computing power and precision of NPUs but also raises higher demands for the flexibility and compatibility of NPU architectures.
This time, Arm Technology’s “Zhouyi” X2 NPU focuses on enhancing overall performance while addressing the needs for high computing power, precision, flexibility, and compatibility across multiple terminals and cross-domain applications.
In terms of computing power, the “Zhouyi” X2 NPU not only enhances single-core performance but also introduces multi-core and multi-cluster architectures, supporting computing power schemes of up to 320 TOPS. Compared to the single-core architecture of the “Zhouyi” X1 and “Zhouyi” Z series, the “Zhouyi” X2 incorporates clusters composed of multiple NPU cores, achieving exponential increases in computing power compared to single cores. Multiple clusters form subsystems, further enhancing cost-effective computing power delivery for 320 TOPS high-performance products.
In terms of precision, the “Zhouyi” X2 NPU supports mixed-precision computing, accommodating integer calculations of 4bit, 8bit, 12bit, 16bit, 32bit, as well as floating-point calculations of 16bit and 32bit, allowing for a better balance of power consumption, computing power density, and calculation precision.
In terms of flexibility, the task scheduling speed of the “Zhouyi” X2 NPU reaches 100 nanoseconds. According to Yang Lei, product director of Arm Technology, a hardware-level task scheduling acceleration unit was designed for the “Zhouyi” X2 to support real-time task scheduling for multiple cores or computing units.
“When my car has 10 cameras and 10 streams of data coming in, the task scheduler identifies which NPU core is idle and can immediately assign the task to that core for computation, building a dynamic and real-time scheduling solution,” Yang Lei said.
In terms of compatibility, the “Zhouyi” X2 NPU supports custom operators to meet various model deployment needs and provides configuration schemes and specialized optimizations for applications such as ADAS, smart cockpits, tablets, desktops, and smartphones.
For consumer-grade terminals, the “Zhouyi” X2 NPU has been optimized for AI denoising, super-resolution, and frame interpolation in scenarios involving photography, videography, and video conferencing.
For scenarios such as autonomous driving, the “Zhouyi” X2 NPU can provide high-performance configuration schemes. First, the dedicated hardware acceleration task scheduler can better support the automotive needs for real-time response to front-target judgments. Secondly, the support for mixed precision aligns with the automotive scene’s pursuit of higher computational precision. Additionally, Arm Technology has specifically optimized the performance of the Transformer model for automotive algorithms, achieving a tenfold performance improvement over the previous generation “Zhouyi” Z2 at equal computing power.
Based on edge terminals and autonomous driving, the “Zhouyi” series will also expand to higher performance scenarios.
“In the past few years, the applications of the ‘Zhouyi’ series have expanded from the AIoT field to automotive and edge intelligent terminals, and will continue to develop towards higher-performance cloud and service sides. We hope ‘Zhouyi’ encompasses different application scenarios and combines with more diverse systems through various configurations,” said Liu Shu, executive vice president and head of product development at Arm Technology.
Open-source software addresses development pain points and aids local NPU ecosystem building.
In the process of landing in more industries and scenarios, the application development of NPUs has also encountered some pain points. The lack of a unified toolchain has led to hardware fragmentation on the inference side, increasing the costs and cycles of application development innovation. On the other hand, NPUs generate or involve vast amounts of data during the training and inference of AI models, making developers increasingly urgent in their demand for white-box software and toolchains.
To address these pain points, Arm Technology launched the “Zhouyi” NPU software open-source plan, which opens up source code to meet customers’ needs for more autonomous and flexible algorithm porting. According to the plan, Arm Technology will first open the NPU intermediate representation layer specifications, model parsers, model optimizers, drivers, and provide free software toolchains, including software simulators, debuggers, and C compilers.
Arm Technology’s “Zhouyi” NPU software open-source plan
“We have received a lot of user feedback, the most typical being in automotive application scenarios. On one hand, the algorithms and data generated during practical applications by Tier 1 are valuable resources. If issues arise during porting and development, Tier 1 wants to resolve them independently on their side, conducting white-box development and debugging. On the other hand, customers have many custom operator demands, which also strongly necessitate white-box software and toolchains. Based on this market feedback, we chose this time to launch the open-source plan,” Yang Lei stated in an interview with “China Electronics News”.
Currently, Arm Technology has open-sourced the front-end of the “Zhouyi” NPU software tool Compass, and in the first phase of the open-source plan, has opened the Compass parser, NPU Linux driver, Compass integration, and model repository.
Moreover, Arm Technology’s latest generation V3 architecture is free to license. According to calculations by Arm Technology’s R&D team, if developers participate in the “Zhouyi” NPU open-source project and are compatible with the “Zhouyi” architecture on hardware, they can save over 50% of the workload in both hardware and software development.
In the future, Arm Technology will gradually open more resources, such as source code for model quantization and operator implementations.
Open-sourcing software not only enhances development efficiency but also continuously incorporates developer feedback during usage, benefiting the healthy cycle of the industry ecosystem. On one hand, open-sourcing NPU software can improve user development efficiency, reduce time to market, and enhance system energy efficiency. On the other hand, open-source facilitates the broader application of NPUs, allowing NPU IP vendors to connect with upstream and downstream enterprises to jointly nurture the localized NPU ecosystem.
“NPUs possess strong software attributes and are not merely hardware accelerators, as users need to run various applications or algorithms on them. Each algorithm’s deployment or porting on hardware contributes to the ecosystem,” Yang Lei said.
Currently, Arm Technology has established open-source repositories for NPU software on code hosting platforms Gitee and GitHub, attracting the first batch of partners from AIoT, smart automotive, and intelligent operating system fields to join. According to Arm Technology, these partners have all expressed their intention to deepen cooperation with Arm Technology based on the NPU open-source plan to accelerate the construction of a localized intelligent computing ecosystem “circle of friends”.
Combining Arm’s heritage with localized innovation to strengthen the “ammunition depot” for chip design.
Since the establishment of Arm in 1990, the number of chips shipped based on the Arm architecture has reached 250 billion. The business model of IP licensing has become deeply ingrained in the industry, becoming an important aspect of vertical division in chip design.
As an independently operated, Chinese-funded joint venture, Arm Technology leverages the advantages of Arm technology and ecosystem while focusing on and integrating localized needs, forming a business strategy of “innovative development of self-researched IP technology in conjunction with Arm IP”. According to Arm Technology, at its inception, it signed a cross-licensing agreement with Arm, allowing it to provide integrated circuit IP licensing and technical services to partners based in China, while also possessing independent R&D rights to develop IP and standards based on Arm technology tailored to the Chinese market, with the intellectual property of self-developed IP belonging to Arm Technology.
Arm Technology’s self-developed IP product matrix and related services
Taking the “Zhouyi” NPU as an example, Arm Technology has attracted and trained a localized NPU engineer team while developing its hardware IP and software tools, providing localized support for company R&D and user development.
“After five years of effort, we have attracted and trained many engineers in Beijing, Shanghai, and Shenzhen, with the entire team comprising over 130 engineers engaged in full-stack R&D of NPUs. We have developed three generations of NPU products and architectures, focusing on IP and SDK development, as well as services and support for domestic customers,” said Sun Jinhong, senior director of NPU R&D at Arm Technology.
To date, Arm Technology has over 370 authorized customers in China, with cumulative chip shipments exceeding 30 billion units. In addition to the “Zhouyi” NPU, Arm Technology has also developed and launched localized IP products such as the “Xingchen” CPU, “Shanhai” SPU, “Linglong” ISP, and “Linglong” VPU, creating an IP matrix that includes CPU, NPU, information security, and multimedia, providing an “ammunition depot” for heterogeneous integration solutions for chip companies, which also enhances the flexibility of chip design.
For example, the two Wi-Fi and Bluetooth dual-mode SoC chips launched by Broadcom in 2022 integrated the “Xingchen” processor and the “Shanhai” information security core module.
The open-sourcing of the “Zhouyi” NPU software and corresponding ecological cooperation measures are also a continuation and supplement to Arm Technology’s “Ecosystem Partner Program”. This program was initiated in July 2022, relying on the Arm technology ecosystem and self-developed IP product matrix to co-build the upstream and downstream industrial ecosystem with ecological partners, jointly promoting the development of software and hardware, solutions, toolchains, industry standards, and community alliances across various fields. So far, several chip design companies, solution providers, and system platform companies have joined the ecosystem partner program.
Arm Technology’s Ecosystem Partner Program
IP, as the core asset of hardware design, plays a crucial role in chip design and product definition and has far-reaching significance for the underlying innovation of the chip industry. The iteration and innovation of IP need to be coupled with the development trends and common issues of critical nodes in the chip, software, ecosystem, and standards industries to drive the upward spiral of the chip industry.
“For the industry, the significance and role of an IP company lie in its ability to recognize the trends in ecosystem and technology development, helping clients and partners solve some issues of redundant investment. Generally speaking, once we resolve the underlying technical and innovation challenges, we can help clients and partners save one to two years in development cycles. Clients utilizing these IPs can launch mass-producible chips more efficiently, achieving application-level innovation, which is the positioning and role of IP as a cornerstone to help clients succeed,” Liu Shu said.
Author: Zhang Xinyi
Editor: Chen Bingxin
Design: Maria
Supervisor: Lian Xiaodong