Embedded AI Briefing (2020-02-16)

Focus on model compression, low-bit quantization, mobile inference acceleration optimization, and deployment

Introduction: This content includes 20 items. ARM has released the Cortex-M55 and Ethos-U55 series, suitable for voice AI model inference. ARM also published a white paper on deploying machine learning with the Cortex-M series and deploying convolutional network models with Cortex-M combined with CMSIS-NN, etc. NXP engineers have implemented and released AI inference demos and tools on MCUs. The “open-source project” includes a face mask detection dataset and model, and Google’s experimental MLIR execution environment project is worth attention. The “blog” section includes video reviews and PPTs from two live broadcasts of MNN, as well as an overview of MLsys direction and a detailed explanation of the Winograd algorithm that are worth a look.

Note: Some links may not open, please click at the end of the article to read the original text for redirection

Industry News

• Arm launches the latest Cortex-M55 and Ethos-U55 processor cores: AI inference performance can be improved by up to 50 times, supporting custom instruction sets, suitable for voice inference | EETOP Summary: Arm has launched two new IPs (Cortex-M55 and Ethos-U55) to expand its AI-related products. The Cortex-M55 CPU: This CPU comes with many new features announced by Arm over the past year. The first new feature is support for custom instructions, and Arm also calls the Cortex-M55 the “most AI-capable Cortex-M processor” as it is the first processor to implement Helium functionality. Helium, also known as M-Profile Vector Extension (MVE), is an extension of the Armv8.1-M architecture that introduces new SIMD 128-bit vector operations designed to enhance DSP and ML applications and performance. It relies more on existing registers rather than vector registers like NEON and introduces new support for features such as lane prediction, loop prediction, and complex operations (like scatter-gather). More importantly, it achieves this by executing smaller data blocks (32 bits) to keep the silicon area of the IP minimal. For machine learning, Helium provides instructions such as VMLAV horizontal vector multiply-accumulate instructions. Ethos-U55: is a dedicated microNPU inference accelerator that can be used in conjunction with Cortex-M series CPUs, bringing performance and energy efficiency improvements specifically for NPU. For more information on Arm Helium, Armv8.1-M extended architecture, and improvements in machine learning and signal processing performance, see: https://learn.arm.com/introduction-armv8.1m-cn.html

• ARM Mali-G77 GPU wins Linley Group’s “Best Processor IP” award | EETOP Summary: On January 21, ARM announced that its Mali-G77 GPU won the “Best Processor IP” award from the Linley Group, a US chip market research and consulting firm. This is ARM’s latest generation mobile GPU, with a performance increase of 60%. High-end smartphones equipped with the Mali-G77 GPU will officially ship in Q1 of this year. The Mali-G77 is ARM’s latest generation mobile GPU released in May last year, designed with the Valhall architecture, which brings a new ISA bus and computing core design, addressing the main shortcomings of the previous Bifrost architecture. Among current mobile processor manufacturers, Qualcomm and Apple develop their own GPUs, while Huawei’s Kirin 990 series still uses the Mali-G76, and Samsung’s Exynos 980 also uses the Mali-G76. Only MediaTek’s Dimensity 1000 and Samsung’s Exynos 990 use the Mali-G77, but only the Dimensity 1000 has been officially launched. ARM’s mention of high-end models launching this quarter refers to smartphones based on the Dimensity 1000 processor, mainly the OPPO Reno3 series. The Dimensity 1000 features four ARM Cortex-A77 big cores with a clock speed of up to 2.6GHz and four ARM Cortex-A55 cores with a clock speed of 2.0GHz. The performance of the A77 architecture is a 20% improvement over the previous A76 architecture. In terms of GPU, it also uses the latest ARM Mali-G77MC9, with a clock speed of 836MHz, a 40% improvement over the previous generation Mali-G77. The Dimensity 1000 is also the world’s first chip to use the ARM Cortex-A77 architecture + Mali-G77 GPU.

• Intel scraps “first AI chip” and promotes recently acquired Habana | Quantum Bit Summary: Intel also stated that it will push harder for Habana Labs’ Gaudi and Goya processors—the former focuses on training while the latter focuses on inference—to replace the Nervana chip. Habana Labs, founded in 2016 in Israel, focuses on programmable deep learning accelerators aimed at data centers. In just three years, it has released chips for both training and inference markets. The training chip Gaudi competes with NVIDIA’s T4, boasting a 4x performance advantage, while the inference chip Goya has a 2x performance advantage. Of course, Intel’s Nervana released its first inference chip, NNP-I, in 2019, which has a performance 3.7 times that of NVIDIA’s T4. However, it should be noted that the last performance score for Goya was disclosed at the end of 2018, at which time Intel’s own products had no competitors. This may also be the reason why Intel ultimately decided to acquire it fully.

• Apple acquires startup Xnor.ai for $200 million | AI Report Summary: Xnor.ai, founded in 2017, has gained recognition in edge AI technology. In 2019, Xnor.ai developed a standalone AI chip that can run on solar or coin-sized batteries, capable of automatically monitoring grocery store shelves with Gizmo. In addition, Xnor.ai reached an agreement with Wyze Labs to integrate its edge-based person recognition technology into Wyze Labs’ low-cost security cameras. Products developed by Xnor.ai can run directly on devices such as smartphones, cameras, drones, and embedded low-power mobile CPUs without relying on cloud computing.

• Exploring the “hottest” AI temperature measurement industry in security: Daily supply of hundreds of devices, single unit priced over ten thousand | Machine Heart Summary: To expedite AI temperature measurement projects, many employees are living and working at the company, with some sleeping only 2-3 hours a day… Currently, companies like Baidu, SenseTime, Megvii, Geling Deep Vision, as well as companies like Gaode Infrared, Hikvision, Dahua, and Uniview have all launched AI infrared temperature measurement products. From the market solutions, the industry mostly adopts dual sensors of infrared/visible light, combining infrared thermal imaging and face recognition to achieve non-contact multi-target temperature screening, with temperature measurement accuracy generally around ±0.3℃. From a technical logic perspective, it requires AI algorithms for face recognition and tracking, accurately locating each face in the camera’s view, and then combining infrared temperature point arrays to calculate the temperature of the face (mostly the forehead). The details involved in recognizing faces with masks, ReID (person re-identification), and calibration between two cameras directly affect the effectiveness of each solution.

Research Papers

• Reversible Residual Networks Significantly Reduce GPU Memory Usage | I Love Computer Vision Title: The Reversible Residual Network: Backpropagation Without Storing Activations Summary: Aidan N.Gomez and Mengye Ren from the University of Toronto proposed a reversible residual neural network, where the activation results of the current layer can be calculated from the results of the next layer. This means that if we know the final result of the network layer, we can backtrack to obtain the intermediate results of each previous layer. Thus, we only need to store the network parameters and the final layer’s results, making the storage of activation results independent of the network’s depth, which will significantly reduce memory usage. Surprisingly, experimental results show that the performance of the reversible residual network does not significantly decline and is comparable to the previous standard residual network experimental results.

• Pruning Practice: How to Accelerate and Save Memory in Image Retrieval | Zhihu Title: Progressive Local Filter Pruning for Image Retrieval Acceleration Summary: The speed of image retrieval is one of the important indicators for evaluating application systems, but image retrieval networks are more sensitive to compression algorithms. A slight modification to the network may lead to significant changes in output features. To address this issue, the author proposes the following two solutions:

1. Adopt a local-based approach: Prioritize selecting and removing filters that have the closest local relationships, meaning if a filter is sufficiently similar to its adjacent filters, it can be safely removed. Since this method involves local modifications to filters, it is easy to maintain the overall distribution of filters.

2. Try to maintain the representational power of the pre-trained model. However, removing filters can easily reduce the network capacity, thereby affecting the output features. Soft pruning methods set the filters to be removed to zero first and give them a chance to recover weights during fine-tuning, which can alleviate the network capacity loss caused by directly removing filters to some extent. However, if the proportion of removed filters is too large, it will severely affect the network’s representational power. Therefore, we propose a filter weight decay strategy, gradually reducing the weights of filters to be removed during training, multiplying the weights by a decay factor γ until they converge to 0 or close to 0.

Open Source Projects

• google/IREE: An Experimental MLIR Execution Environment Summary: IREE (Intermediate Representation Execution Environment, pronounced as “eerie”) is an experimental compiler backend for MLIR that lowers ML models to an IR optimized for real-time mobile/edge inference against heterogeneous hardware accelerators. The IR produced contains the sequencing information required to communicate pipelined data dependencies and parallelism to low-level hardware APIs like Vulkan and embed hardware/API-specific binaries such as SPIR-V or compiled ARM code. As the IR is specified against an abstract execution environment, there are many potential ways to run a compiled model, and one such way is included as an example and testbed for runtime optimization experiments.

• zamhown/wear-a-mask: Wear a Mask for Your Avatar Summary: This project uses the face-api.js library based on TensorFlow.js. The face detection task uses the SSD MobileNet V1 model (trained on the WIDERFACE dataset), and the face landmark detection task uses the 68-point detection model based on CNN constructed by the author of face-api.js (trained on a dataset of approximately 35,000 facial images), with model weight data sourced from face-api.js.

• hikariming/virus-mask-dataset: Face Mask Detection Dataset Summary: Currently, there are over 200 images, primarily from data after the outbreak of the pandemic in January 2020. In the future, monitoring and hospital photos will be added, aiming to exceed 1,000 images in the dataset. The labeling method includes: labeling the entire head and upper shoulders of the person, with a total of three label categories: rightmask/this person is wearing a mask that prevents viruses, wrongmask/this person is wearing a mask that does not prevent viruses, nomask/not wearing a mask.

• jason9075/mask_checker: Face Mask Detection Model Summary: The model uses the pre-trained MobileNetV2 from tf.keras for training, with a dataset of 400 images of faces with and without masks sourced from the internet.

• Baidu announces the free open-source of the industry’s first mask face detection and classification model | Machine Heart Online demonstration: https://www.paddlepaddle.org.cn/hub/scene/maskdetect Summary: This model can effectively detect all faces wearing and not wearing masks in densely populated areas, while determining whether they are wearing masks. The face recognition and mask-wearing judgment deployed on mobile can be found at: https://github.com/PaddlePaddle/Paddle-Lite/tree/develop/lite/demo/cxx

• UoB-HPC/openmp-tutorial: Exercises and Solutions for “Programming Your GPU with OpenMP: A Hands-On Introduction” PPT: https://sc18.supercomputing.org/presentation/?id=tut138&sess=sess245 Summary: This is a hands-on tutorial that introduces the basics of targeting GPUs with OpenMP 4.5 through a series of worked examples. Starting with serial code, the tutorial takes you through parallelizing, exploring the performance characteristics, and optimizing the following small programs:

1. vadd – A simple vector addition program, often considered the “hello world” of GPU programming.

2. pi – A numerical integration program that calculates an approximate value of π.

3. jac_solv – A Jacobi solver.

4. heat – An explicit finite difference 5-point stencil code.

Blog Posts

• The Computer Vision Algorithms Behind Xiaomi CC9 | InfoQ Summary: Huang Ying, director of Xiaomi’s AI department, gave an excellent speech at AICon2019 Beijing on “From Research to Implementation: Practical Discussion on Xiaomi’s Computer Vision Algorithms.” Huang first briefly introduced Xiaomi’s business in the field of computer vision, and then illustrated how Xiaomi implements computer algorithms with examples such as magic sky change, face detection, and intelligent clip selection, as well as the difficulties encountered and solutions during implementation.

• Implementing Face Recognition and Gender Recognition on MCU | Selected Embedded News Summary: Running deep learning models on MCUs is significant as it combines the low power consumption, low cost, large volume, short development cycle, fast market launch, good real-time performance, and stable response characteristics of MCUs with the powerful capabilities of deep learning, which will undoubtedly unlock a huge market and make massive devices intelligent! NXP engineers implemented face recognition functionality based on the i.MX RT1050 crossover MCU, with a 30-layer deep learning model running in just 295ms. While developing these use cases, NXP engineers also prepared the necessary supporting tools and provided a test version for everyone to download and use, along with a comprehensive user guide of over ten thousand words.

• Wang Chuan: Why Moore’s Law Has Not Died, But People Will Continue to Predict Its Death | Silicon Valley Wang Chuan Summary: This article analyzes why Moore’s Law continues to develop, but people will continue to mistakenly predict its death. “The number of people predicting Moore’s Law will die doubles every two years.”

• MNN Two Live Broadcasts | Tao Series Technology First Live Broadcast: Exploring Edge Intelligence and MNN: Facing Future Mobile Development Video Replay: https://mudu.tv/watch/4308076 Online PDF: https://docs.qq.com/pdf/DVXVZdGdTeFlSRFVH Second Live Broadcast – In-Depth Discussion on MNN Principles, Why It Can Be So Fast | Tao Series Technology Video Replay: https://mudu.tv/watch/4397479 Online PDF: https://docs.qq.com/pdf/DVUNHb0VIc2FJVlJH

• Overview of MLsys Directions | Zhihu Summary: The author summarizes the current directions of MLsys from their perspective, covering distributed machine learning, deep learning model compression acceleration, framework system design, and using ML to optimize traditional system issues. Finally, they summarize the corresponding engineering and algorithm skills tree based on recruitment needs.

• Detailed Explanation of the Winograd Transformation Matrix Generation Principle | Zhihu Summary: There are already many articles explaining the Winograd algorithm from the perspective of mathematical principles online, but the author could not fully understand, especially how the Winograd transformation matrix is generated. Many details are not explained in detail, and the author describes this part in a more accessible way for readers.

On the smartphone manufacturer side, the Xiaomi 10 series and Samsung Galaxy S20 series have been released, both equipped with Snapdragon 865. Qualcomm is promoting its own SNPE and has partnered with TFLite and others to hold an AI innovation application competition. MediaTek has stated that due to the impact of the pandemic, Q1 revenue may decline by up to 15%.

Previous issues: See the main menu of the public account 【Historical Messages】

• Wechat: NeuralTalk | NeuroMem

• Editor: https://github.com/ysh329

• Project: https://github.com/ysh329/awesome-embedded-ai

This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.