Embedded AI Briefing (2020-03-19)

Focus on model compression, low-bit quantization, mobile inference acceleration optimization, and deployment

Abstract: The domestic epidemic situation has stabilized, and we are waiting for the open-source release of the Megvii framework at the end of March. Huawei Cloud has launched its 2020 “new flagship” – Kunpeng Cloud Phone. Meanwhile, the overseas epidemic is surging, and image sensors with built-in neural networks have been featured in Nature, achieving image classification in 40 nanoseconds. Georgia Tech has developed an OpenCL-compatible GPGPU based on the RISC-V instruction set architecture (Vortex), which will soon be open-sourced. Recently, UK AI chip startup Graphcore and US AI chip startup SambaNova have successfully completed new funding rounds of $150 million and $250 million, respectively. It is reported that Graphcore’s IPU chips are already configured in Microsoft’s and Cirrascale’s cloud computing products, as well as Dell’s DSS8440 IPU servers. This content includes 20 items, “Industry News” Apple’s A14, Huawei, MediaTek, and Unisoc’s new SoCs are worth noting. Google has open-sourced a mobile 3D object detection demo. In the “Papers” section, there is a paper from MLsys discussing training speed on CPUs being faster than on GPUs. The first paper from MLsys is MNN, which is worth a look. The review of binary networks also provides a detailed comparison. Previous issues featured BERT compression, and this time the idea of the Ship of Theseus is applied to BERT compression. “Open Source” includes a super lightweight Chinese OCR model of only 17M and a silent face liveness detection algorithm. In the “Blog” section, the first article on the best practices for lightweight network design in WeChat’s scan-to-identify feature and NCNN’s BF16 acceleration are both worth reading.

At the end of the article, “Sorry, because the previous code was poorly written, I can only continue to write poorly,” is dedicated to everyone.

Embedded AI Briefing (2020-03-19)

Note: Some links may not open, please click at the end of the article to read the original text for redirection

Industry News

  • Apple’s 5nm A14 has a clock speed of up to 3.1GHz, with single-core scores far exceeding the strongest Android chip | EETOP Summary: Recently, more leaks about Apple’s iPhone 12 have emerged, with a blogger revealing the suspected Apple A14 chip’s Beta 1 version Geekbench 5 scores. The single-core score of the Apple A14 processor Beta 1 version Geekbench 5 is 1658, and the multi-core score is 4612, with a clock speed of up to 3.1GHz. In comparison, the current A13 processor scores 1330 in single-core and 3435 in multi-core under the same benchmark, far exceeding the current strongest Android chip, Qualcomm Snapdragon 865, which scores 901 in single-core and 3315 in multi-core.

  • Huawei’s new flagship SoC exposed: to be released in September | AnTuTu Summary: Processor chip tape-out is divided into pre-tape-out verification and post-tape-out verification. Last September, @mobile chip expert indicated that the 5nm HiSilicon processor had officially completed tape-out, so it should be preparing for function verification on the development board before entering the engineering machine testing phase. Currently, the official naming of the next-generation Kirin flagship processor is not yet known, with rumors suggesting it will be named Kirin 1020 or Kirin 1000. According to convention, Huawei holds a new generation Kirin flagship processor launch event every September, followed by the Mate series debut. Therefore, it is expected that the Mate 40 series will be among the first models equipped with a 5nm process technology processor, giving Huawei a first-mover advantage over competitors, which is worth looking forward to.

  • MediaTek releases Helio P95 SoC: equipped with PowerVR GM 9446 | Imagination Tech Summary: On February 27, MediaTek quietly launched the Helio P95 page on its official website. From the naming convention, it can be seen that this SoC is an iterative upgrade based on Helio P90. MediaTek claims that this chip includes a new generation AI processing unit (APU 2.0). Helio P90 is still a 4G SoC and does not support 5G networks. It supports Wi-Fi 5 / Bluetooth 5.0, Cat 12 downlink / Cat 13 uplink rates, and 4×4 MIMO. In terms of CPU, Helio P95 adopts a “2 big cores and 6 small cores” combination of 2×Cortex A75 (2.2GHz) + 6× Cortex A55 (2.0GHz). In terms of GPU, this SoC integrates Imagination PowerVR GM9446, supporting a resolution of 2520×1080, and MediaTek claims that the GPU benchmark score has improved by 10% compared to the previous generation, which is a small improvement.

  • 6nm EUV Unisoc launches new 5G SoC T7520 | AnTuTu Summary: Leading global mobile communication and IoT core chip supplier Unisoc announced a series of major product releases last month, including several commercial 5G terminals equipped with Unisoc’s 5G chips: Unicom 5GCPE and Hisense’s first 5G phone F50, as well as the new 5G SoC mobile platform – Tiger T7520. The new 5G SoC mobile platform T7520 has been heavily released. T7520 integrates high-performance, energy-efficient NPU, maintaining about twice the energy efficiency ratio (FPS/w) compared to competitors’ flagships for typical models like ResNet/MobileNet.

  • Allwinner Technology and Arm China join forces to launch the first AI voice dedicated chip R329 | Allwinner Technology Summary: Allwinner Technology recently officially released the R329, a high-performance, low-power AI voice dedicated chip featuring Arm China’s new AI processing unit (AIPU). By integrating high-performance AIPU, DSP, and CPU, it will bring a new AI interaction experience to smart speakers and smart homes. Allwinner Technology’s R329 is equipped with Arm China’s “Zhouyi” AIPU, providing up to 0.256TOPS of computing power. The Zhouyi AIPU, as an AI core, has a theoretical AI computing power 25 times that of a single-core A7 at 1.2GHz and also 25 times that of a single-core HIFI4 at 600MHz. In addition, it adopts 2 cores with a clock speed of up to 1.5GHz Arm Cortex-A53, providing a more sufficient system computing power foundation for smart voice product applications.

  • Google open-sources mobile real-time 3D object detection, available for download on Android | PanChuang AI Summary: Google has launched MediaPipe Objectron, a mobile real-time 3D object detection pipeline suitable for everyday objects. It can detect targets in 2D images and estimate their pose and size using machine learning models trained on a newly created 3D dataset. Specifically, MediaPipe is a cross-platform open-source framework for building pipelines to process different modality perception data, and Objectron is implemented within MediaPipe, capable of calculating the oriented 3D bounding boxes of targets in real-time on mobile devices. MediaPipe is a multimedia framework released by Google last July, applicable on various platforms such as Android, iOS, and web, utilizing machine learning media models. Recently, MediaPipe released version 0.7, adding mobile 3D detection models. Currently, MediaPipe includes features such as face detection, hand detection, hair segmentation, and automatic video orientation switching. Framework: https://github.com/google/mediapipe/ Project: https://github.com/google/mediapipe/blob/master/mediapipe/docs/objectronmobilegpu.md

Papers

  • SLIDE: Accelerating Training on CPU Using Sparsity | Zhihu Title: SLIDE: In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems Link: https://proceedings.mlsys.org/static/paper_files/mlsys/2020/105-Paper.pdf Summary: This paper discusses how to speed up training on CPUs. When updating gradients during model training, it is unnecessary to look at all neurons; only those with high activations need to be considered. However, previous algorithms utilizing sparsity lacked optimization, and even if values were set to 0, matrix operations still had to be performed. This paper will use LSH + lookup table to accelerate the algorithm.

  • Latest Review on Binary Neural Networks | PaperWeekly Title: Binary Neural Networks: A Survey Link: https://www.sciencedirect.com/science/article/abs/pii/S0031320320300856 Summary: In this paper, the authors provide a comprehensive summary and overview of binary network methods, mainly divided into naive binarization methods for direct quantization and improved binarization methods using techniques such as minimizing quantization error, improving network loss functions, and reducing gradient errors. The authors also investigate other practical aspects of binary neural networks, such as hardware-friendly designs and training techniques. They then evaluate and discuss different tasks such as image classification, object detection, and semantic segmentation. Finally, they look ahead to the challenges that future research may face.

  • [CVPR2020] CARS: Huawei’s Neural Architecture Search Based on Evolutionary Algorithms and Weight Sharing, Only Half a Day on CIFAR-10 with a Single Card | Zhixun Title: CARS: Continuous Evolution for Efficient Neural Architecture Search Link: https://arxiv.org/abs/1909.04977 Summary: Nowadays, evolutionary algorithms, gradients, and reinforcement learning can all perform structure searches. Research has shown that evolutionary algorithms can find better models than reinforcement learning, but the search takes a lot of time, mainly due to the cumbersome training and validation of individuals. However, the weight-sharing strategy used in ENSA can be borrowed for validation acceleration. If directly applied to evolutionary algorithms, the supernet will be affected by poor search structures, so modifications are needed in the evolutionary algorithms used in current neural network search algorithms. To maximize the value of the knowledge learned from the last evolutionary process, the authors propose a continuous evolution architecture search method (CARS): first, initialize a supernet with a large number of cells and blocks, which generates individuals (subnets) in the evolutionary algorithm through several benchmark operations (crossover, mutation, etc.), using a non-dominated sorting strategy to select several excellent models of different sizes and accuracies, then train the subnets and update the corresponding cells in the supernet. In the next round of evolution, the process will continue based on the updated supernet and the non-dominated sorted solution set. Additionally, the paper proposes a protection mechanism to avoid the small model trap. An additional consideration for the growth rate of accuracy is added to the non-dominated sorting, and finally, the two sorts are combined for selection. This way, larger models with slower accuracy growth can also be retained.

  • BERT Compression via the Ship of Theseus: Compressing BERT by Progressive Module Replacing | Machine Heart Title: BERT-of-Theseus: Compressing BERT by Progressive Module Replacing Link: https://arxiv.org/abs/2002.02925 Summary: The authors propose a novel model compression method that effectively compresses BERT through progressive module replacement. First, the original BERT is divided into multiple modules, and more compact replacement modules are constructed. Then, the replacement modules are randomly substituted for the original modules, training the replacement modules to mimic the behavior of the original modules. During the training process, the researchers gradually increase the probability of module replacement, allowing for deeper interaction between the original model and the compact model, facilitating a smooth training process. Unlike explicitly using a distillation loss function to minimize the distance between the teacher model and the student model, this study proposes a novel model compression method. The researchers were inspired by the famous philosophical thought experiment “Ship of Theseus” (if the wood on the ship is gradually replaced until none of the original wood remains, is it still the same ship?), proposing Theseus Compression for BERT (BERT-of-Theseus), which progressively replaces the original modules of BERT with fewer parameters. The original model is referred to as the “predecessor,” and the compressed model is called the “successor,” corresponding to the teacher and student in KD. Recommendation: Compared to previous knowledge distillation methods used for BERT compression, this method only utilizes one loss function and one hyperparameter, freeing developers from the tedious parameter tuning process. This method outperforms existing knowledge distillation methods on the GLUE benchmark, opening a new direction for model compression.

Open Source Projects

  • ouyanghuiyu/chineseocrlite: Practical ultra-lightweight Chinese OCR open-source project, total model only 17MAddress: https://github.com/ouyanghuiyu/chineseocrlite Summary: The currently more commonly used Chinese OCR open-source project is chineseocr, which is based on YOLO V3 and CRNN for Chinese natural scene text detection and recognition, and this project has already garnered 2.5K stars. This article introduces a Chinese OCR project, improved based on chineseocr, which is an ultra-lightweight Chinese character recognition project. The chineseocrlite project indicates that compared to chineseocr, it adopts a lightweight backbone network PSENet, a lightweight CRNN model, and a line text direction classification network AngleNet. Despite aiming to achieve multiple capabilities, the overall model of chineseocrlite is only 17M. Currently, chineseocr_lite supports text detection in any direction and automatically determines the line text direction during recognition.

  • zeusees/HyperFAS: HyperFAS based on deep learning face silent liveness algorithmAddress: https://github.com/zeusees/HyperFAS Summary: Face liveness verification is an important part of the face recognition process, mainly used to distinguish between real faces and fake face images, capable of recognizing deceptive behaviors through printed paper, screen captures, 3D models, etc. During the algorithm design phase, the project author tried different methods, including SVM, LBP, deep learning, etc. For a single scene or camera, good results can be achieved, but a liveness algorithm adaptable to various cameras has not been obtained. The author has made one of the trained models available, but the performance under backlight conditions is not very good, which can serve as a reference for everyone.

  • ShiqiYu/libfacedetection: High-speed face detection library libfacedetection upgraded to version 3, adding five-point detectionAddress: https://github.com/ShiqiYu/libfacedetection Summary: Among many face detection open-source libraries, the libfacedetection developed by Professor Shiqi Yu from Southern University of Science and Technology is renowned for its speed on CPUs and has gained high attention in the developer community due to its BSD license allowing commercial use. The library has quietly released version 3, with the new feature being face five-point detection, which is essential in many face recognition applications. The author claims that the computational cost has almost no increase.

  • fossfreedom/indicator-sysmonitor: Ubuntu system parameter display tool – indicator-sysmonitorAddress: https://github.com/fossfreedom/indicator-sysmonitor Summary: indicator-sysmonitor is a desktop open-source tool for displaying system parameters on Ubuntu, capable of showing CPU temperature, memory, network speed, CPU usage, network IP, and network connection status on the desktop. It supports Unity, Xubuntu, Gnome-Shell Linux desktops and follows the GPL open-source protocol.

  • google/trax: Trax — your path to advanced deep learningAddress: https://github.com/google/trax Summary: Trax code is structured in a way that allows you to understand deep learning from scratch. We start with basic maths and go through layers, models, supervised and reinforcement learning. We get to advanced deep learning results, including recent papers such as Reformer – The Efficient Transformer, selected for oral presentation at ICLR 2020.

Blog Posts

  • Revealing the best practices for lightweight network design in WeChat’s “Scan to Identify” feature | Tencent Technology Engineering Summary: The article title has been modified. WeChat’s “Scan to Identify” feature has been online for some time. Compared to related competitors’ “Take a Photo” feature, the characteristic of “Scan” brings a more convenient user experience. “Scan” relies on efficient mobile object detection. This article will reveal the model design selection to the final implementation.

  • How are the face slimming and skin smoothing beauty functions achieved during live streaming? | AI Technology Camp Summary: The purpose of beauty is to make people look more beautiful, including delicate, fair, and smooth skin, with detailed adjustments to facial features and shapes. Through makeup adjustments, a quick makeup effect can be achieved, ultimately attracting users to increase platform revenue. This article explains the implementation of four key steps from a technical perspective. Ultimately, in terms of performance, on mid-range devices like the iPhone 6P, real-time face beautification can be achieved at 720p 24fps; in terms of effect, skin processing can make the face appear fair and delicate, while the host can adjust any facial features according to their preferences.

  • Using bf16 to accelerate ncnn | Zhihu Summary: In simple terms, it means cutting off the last 16 bits of float, sacrificing effective digits. To represent a number, bf16 uses half the memory space of fp32. The cache size of mobile CPUs is limited, so halving it is significant! Even if fp32 operations require shifting conversions, it can still benefit from fewer reads and higher cache hit rates. @Circle Worm: This bf16 is different from fp16. bf16 supports early armv7a and armv8.1 CPUs, not fp16 on GPUs. That is, Cortex-A7/A9/A15/A17/A53/A72/A73 can achieve a speed increase of 15%-30%.

  • Detailed explanation of PaddlePaddle’s automatic mixed precision technology: one line of code doubles training speed | Machine Heart Summary: PaddlePaddle AMP (Automatic Mixed Precision) technology can help users convert single-precision training models to automatic mixed precision training with just one line of code. At the same time, it ensures training stability through black-and-white lists and dynamic Loss Scaling, avoiding INF or NAN issues. PaddlePaddle AMP can fully leverage the computational performance advantages of Tensor Cores in the new generation of NVIDIA GPUs, with training speeds for models like ResNet50 and Transformer increasing by 1.5 to 2.9 times compared to single-precision training.

  • Sorry, because the previous code was poorly written, I can only continue to write poorly | Brother Code Summary: This is a management issue, not just a developer issue. Often, the timeline is very tight, and there isn’t enough time; it’s not that developers want to refactor. If given time, then writing poor code would indeed reflect a lack of skill. However, often there is no time. For a small requirement, if it involves 10 interfaces and is a modification based on the original, the leader thinks it can be done in a day. Developers are so busy they can’t even drink water, how can they stay up all night to refactor? And even if they do, they may not complete it. Once they fail to complete it, the leader says that code should prioritize business implementation. What reason can the leader not use? And why should employees be made to stay up all night to refactor? Employees can have technical pursuits and can write good code in their own projects, but without time, not modifying the company’s poor code is understandable; the key is to provide time, and if time is given, everything can be discussed. Otherwise, just saying that employees do not modify the previous poor code is meaningless.

Previous issues: See the main menu of the public account 【Historical Messages】

  • Wechat: NeuralTalk | NeuroMem

  • Editor: https://github.com/ysh329

  • Project: https://github.com/ysh329/awesome-embedded-ai

This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.

Leave a Comment