The foundation of an intelligent future is “perception + computation”. AI vision will play a crucial role in the process of intelligence and has a very broad application prospect.
Author｜Li Yan Proofreader｜Sami
Jiwei Network · Ai Jiwei APP, available for download in major mainstream app stores
Jiwei Network reports that it is hard to imagine what an intelligent world without visual perception technology would look like. From intelligent security, smart transportation, industrial intelligence, smart cars to consumer electronics, AI-based visual perception technology has been integrated into various industries’ intelligence processes. It is precisely through the empowerment of AI that visual perception has undergone an evolutionary process of being visible, clear, and understandable, playing an irreplaceable role in many application scenarios.
As Liu Jianwei, co-founder of Aixin Yuan Zhi, stated in a recent article titled “Breaking the Memory Wall and Power Wall: The Present and Future of Domestic AI-NPU Chips”, the foundation of an intelligent future is “perception + computation”, and AI vision will play a very crucial role in the process of intelligence, with very broad application prospects.
Without NPU, how can we see “clearer” and “understand”?
A massive amount of visual data is a characteristic of the intelligent interconnected era. To fully exploit the information behind this data, AI technology is needed. Therefore, AI-based intelligent visual perception technology has become a key focus in industry development. According to relevant institutions, the market size of intelligent vision in the industrial, retail, and sports sectors reached 14.511 billion, 634 million, and 36.279 billion yuan respectively in 2021. Meanwhile, the application of intelligent vision accounted for 34.9% of the entire AI application field, becoming an important driving force for technological development.
Hardware determines the final effect presented by intelligent visual systems, and AI chips represented by NPU have thus become prevalent. As the industry increasingly emphasizes the deep integration of AI algorithms and hardware, NPUs are favored for their high computing power.
“If ISP chips allow intelligent visual systems to see clearly, NPU chips enable systems to see clearer and understand better,” Liu Jianwei stated in the article.
Figure | Technical Characteristics of AI Vision Chips
For example, images or videos captured by cameras at night often suffer from overexposure and loss of color detail. To see clearer, technology must be used for post-processing. If traditional techniques like filtering are used, a lot of noise will be generated in low light and wide dynamic range scenarios, making it difficult to achieve the desired effect. Switching to AI-ISP can solve this problem, but it requires AI algorithms to process video at full resolution and full frame rate. “To achieve full resolution and full frame rate processing for a 5-megapixel video stream, very high demands are placed on the computing power of the NPU,” Liu Jianwei stated.
In applications like vehicle detection and license plate recognition, license plates far away in video footage are often unrecognizable, and fast-moving vehicles may be missed. The solution is also to adopt a full-resolution, higher frame rate detection approach, which similarly requires high computing power from the NPU.
Not only is it about “seeing clearer”, but to “understand” also heavily relies on NPUs. A major trend in intelligent visual systems is the intelligence of front-end hardware. Taking the smart city sector as an example, more and more intelligent algorithms are moving from the backend to the front end. Front-end intelligence can provide high-quality, initially structured image data to the backend while greatly saving bandwidth and backend computing resources. Currently, many intelligent algorithms can run in real-time on the front end, such as face detection, entering/leaving areas, and boundary crossing. However, to enable front-end devices to work with these algorithms, high computing power from NPUs is essential.
Liu Jianwei pointed out in the article, “This is akin to embedding an expert into the AI chip. This expert system must be smart enough, corresponding to a large-scale network; a larger network means a larger brain capacity, which can maintain and store more weight values, thereby placing high demands on NPU computing power.”
In public safety monitoring, various abnormal behaviors must often be addressed, such as sudden acceleration, gathering, falling, and intelligent tracking of moving objects in complex scenes. These all require the continuous introduction of new algorithms to solve. Without high computing power NPUs, hardware resources will struggle to keep pace with algorithm upgrades.
On the other hand, to deploy algorithms trained on high computing power servers to front-end devices, model compression is necessary. If the front-end computing power is high, this process can be significantly shortened. In this regard, Liu Jianwei mentioned in the article, “Using some computing power to exchange for improved development efficiency accelerates the landing of AI, but this approach conversely raises the requirements for NPU computing power.”
In the context of massive data computation demands, to enable devices to “see clearer” and “understand”, high computing power NPUs are an indispensable part.
Hardware-Software Collaboration to Break the Memory Wall and Power Wall
The industry is continuously exploring ways to deeply integrate AI algorithms with hardware to improve chip utilization, achieve better visual information processing effects, and reduce chip power consumption for large-scale commercial use. In this process, the development of AI chips faces two major obstacles: the memory wall and the power wall.
Under the traditional von Neumann architecture, the computing and storage units are separated; the computing unit reads data from memory, completes the calculation, and writes it back to memory. However, AI algorithms are a large and complex network, containing a vast amount of image data and weight parameters, which generates a lot of data during the calculation process. The frequent movement of data between the computing and storage units creates the memory wall problem, where the speed of memory data access cannot keep up with the data processing speed of the computing unit, hindering the improvement of computing power.
At the same time, the frequent migration of data also leads to serious transmission power consumption issues. According to Intel’s research, in the 7nm semiconductor process, data transport power consumption reaches 35pJ/bit, accounting for 63.7%. Power loss caused by data transmission is increasingly becoming a limiting factor for chip development, forming the “power wall” problem.
Liu Jianwei pointed out in “Breaking the Memory Wall and Power Wall: The Present and Future of Domestic AI-NPU Chips” that the power wall issue mainly arises from two aspects: MAC units and DDR. “When the MAC units are stacked to increase computing power metrics, the total power consumption of the MAC units will increase, and high bandwidth support is also required. Expensive HBM can be used on the server side, which inevitably raises the power consumption required by DDR. However, on the edge side, due to cost considerations, there is currently no particularly good DDR solution.”
AI chip calculations are data-driven, and the challenges of data transport highlight the “memory wall” and “power wall” issues, which not only limit their computational performance but also restrict their entry into mobile and embedded devices that have strict power consumption requirements.
Integration of storage and computation and reducing data transport are common strategies used in the industry to break through these challenges. While integration of storage and computation is powerful, it is limited by process node bottlenecks and still has some distance to go before chip mass production. Reducing data transport can tap into the potential of existing chips and significantly lower development costs. Liu Jianwei stated that Aixin Yuan Zhi has reduced data transport through mixed precision technology, thereby mitigating the obstacles posed by the memory wall and power wall to a certain extent and improving the overall efficiency of the NPU.
Mixed precision refers to performing numerical calculations with floating-point/fixed-point numbers of different precisions. Industry experts have found that not all stages of the calculation process require high-precision numerical formats. By reasonably allocating numerical precision across different stages, overall calculation speed can be achieved while ensuring the accuracy of the final calculation result.
In this regard, Liu Jianwei also provided corresponding answers in the article: “In the entire neural network, the weight coefficients are relatively complex. The traditional NPU data representation formats are generally 8-bit, 16-bit, and floating-point numbers to meet the precision of AI algorithms, making the computation heavy. However, Aixin Yuan Zhi discovered that in practical applications, some information in AI networks is redundant, which means not all calculations require high-precision floating-point or high-precision 16-bit; using 8-bit or 4-bit low precision mixed operations is sufficient.”
Liu Jianwei stated that in Aixin Yuan Zhi’s AI-ISP applications, many intermediate layers in the network use INT4 precision. Compared to the original 8-bit network, the data transport volume may be reduced to half, and the computation volume reduced to one-fourth, equivalent to providing several times the effective computing power of traditional NPUs within the same area, while also lowering costs and power consumption, making it more favorable for AI deployment on the edge and at the front end.
Figure | Design Challenges and Driving Forces of AI-NPU
Confronted with the cost pressure brought by advanced processes and high-level packaging technology for AI chips, using mixed precision technology to address the “memory wall” and “power wall” issues is a practical approach.
However, on the edge side and at the front end, the coupling strength between AI chips and scenarios is relatively weak. To truly realize AI deployment, a close integration of algorithms and hardware is still needed.
Liu Jianwei stated in the article that the Aixin Yuan Zhi algorithm team will provide detailed information such as algorithm network structure, quantization requirements/operator requirements, and memory access requirements to NPU design architects early in the NPU design process. Hardware architects can adjust or optimize the entire NPU design based on these algorithm requirements. At the same time, hardware engineers will also provide hardware limitations to algorithm engineers, so that algorithm engineers can consider hardware limitations when designing algorithms and avoid some hardware shortcomings from the algorithm perspective.
“Only with true integration of both can we balance NPU hardware and software development, accelerate the efficiency of AI deployment, and achieve the ultimate goal of intelligent vision in seeing clearer and understanding better.”
Image Source｜Network
More Important News
Please click to enterAi Jiwei Mini Program
Or download the Ai Jiwei APP for reading