↑ ClickBlue TextFollow the Jishi platform
Author丨Pai Pai XingSource丨CVHub
Jishi Introduction
This article presents a simple yet efficient modern inverted residual mobile module designed for mobile applications. The proposed efficient model (Efficient MOdel, EMO) achieves excellent overall performance on the ImageNet-1K, COCO2017, and ADE20K benchmarks, surpassing the SOTA models based on CNN/Transformer at the same computational power level. >>Follow the public account and reply ‘Jishi Dry Goods’ in the background to get the latest collection of CV knowledge content.
Title: Rethinking Mobile Block for Efficient Neural Models
Author: Jiangning Zhang et al. (Tencent Youtu, Zhejiang University, Peking University, Wuhan University)
Paper: https://arxiv.org/pdf/2301.01146
Github: https://github.com/zhangzjn/EMO
Introduction
This article rethinks the efficient inverted residual block in MobileNetv2 and the essence of effective Transformers in ViT, abstracting the general concept of Meta Mobile Block. Inspired by this phenomenon, the authors designed a simple yet efficient modern inverted residual mobile block (Inverted Residual Mobile Block, iRMB) for mobile applications, which absorbs the efficiency of CNNs to simulate short-range dependencies and the dynamic modeling capability of Transformers to learn long-range interactions. The proposed efficient model (Efficient MOdel, EMO) achieves excellent overall performance on ImageNet-1K, COCO2017, and ADE20K benchmarks, surpassing SOTA models based on CNN/Transformer at the same computational power level, while effectively balancing model accuracy and efficiency.
Motivation
In recent years, with the increasing demand for mobile applications that are constrained by storage and computing resources, many lightweight models with fewer parameters and lower FLOPs have emerged. For example, during the Inceptionv3 era, asymmetric convolutions were proposed to replace standard convolutions. Later, MobileNet introduced depth-wise separable convolutions to significantly reduce computational load and parameters, becoming a classic in lightweight networks. Building on this, MobileNetv2 proposed an efficient inverted residual block (IRB) based on Depth-Wise Convolution (DW-Conv), which has become one of the standard efficient module representatives. However, due to the influence of the inductive bias of static CNNs, the accuracy of pure CNN models remains relatively low, resulting in a lack of truly groundbreaking work in the subsequent lightweight path.
With the rise of Transformers in the CV field, many high-performance networks have emerged, such as Swin Transformer, PVT, Eatformer, and EAT. Thanks to their dynamic modeling and lack of inductive bias, these methods have achieved significant improvements over CNNs. However, due to the quadratic limitations of multi-head self-attention (MHSA) parameters and computational load, Transformer-based models often consume a large amount of resources, leading to complaints about their practicality.
To address this drawback of Transformers, some solutions have been proposed:
- Designing variants with linear complexity, such as FAVOR+ and Reformer;
- Reducing the spatial resolution of query/value features, such as Next-vit, PVT, Cvt;
- Rearranging channel ratios to reduce the complexity of MHSA, such as Delight;
However, these small modifications have not made a significant impact, leading to the emergence of many hybrid models that combine lightweight CNN designs to achieve efficient performance, outperforming CNN-based models in terms of accuracy, parameters, and FLOPs, such as Mobilevit, MobileViTv2, and Mobilevitv3. However, these methods often introduce complex structures or even directly adopt multiple hybrid modules like Edgenext and Edgevits, which is not conducive to optimization.
In summary, there is currently no efficient block based on Transformers or hybrids that is as popular as the IRB based on CNNs. Therefore, inspired by this, the authors reconsidered the Inverted Residual Block in MobileNetv2 and the MHSA/FFN modules in Transformers, abstracting a universal Meta Mobile Block that instantiates different modules, namely IRB, MHSA, and Feedforward Network (FFN), using a parameter expansion ratio λ and efficient operator F.
Based on this, this article proposes a simple and efficient module—the inverted residual mobile block (iRMB), by stacking different levels of iRMB, thereby designing a lightweight network model for mobile applications—EMO, which can surpass CNN/Transformer-based SOTA models with relatively low parameters and FLOPs, as shown in the figure below:
Method
The above figure is the overall framework diagram, with the left side showing an example of the iRMB module. Let’s further break down this network structure diagram.
Meta Mobile Block
As mentioned above, by abstracting the Inverted Residual Block in MobileNetv2 and the core MHSA and FFN modules in Transformers, the authors propose a unified Meta Mobile (M2) Block to uniformly represent the above structures, instantiating different modules using the parameter expansion rate λ and efficient operator F.
Inverted Residual Mobile Block
Based on the inductive M2 block, this article designs an inverted residual mobile block (iRMB), which absorbs the efficiency of CNN architectures to model local features and the dynamic modeling capability of Transformer architectures to learn long-range interactions.
In specific implementations, F in iRMB is modeled as a cascade of MHSA and convolution operations, which can be abstracted as follows. Here, two main issues need to be considered:
- λ is usually greater than the intermediate dimension, which will be a multiple of the input dimension, leading to a quadratic increase in parameters and computation.
- The FLOPs of MHSA are proportional to the square of the total image pixels.
A simple comparison of parameters can be seen in the following table:
Therefore, the authors naturally considered combining W-MHSA and DW-Conv along with a residual mechanism to design a new module. Additionally, this cascading method can enhance the expansion rate of the receptive field while effectively reducing the model’s MPL.
To evaluate the performance of iRMB, the authors set λ to 4 and replaced the standard Transformer structure in DeiT and PVT. As shown in the following table, we can see that iRMB can improve performance with fewer parameters and computations under the same training settings.
EMO
To better measure the performance of lightweight models for mobile applications, the authors define the following four criteria:
- Usability. That is, a simple implementation without complex operators, easy to optimize for applications.
- Simplicity. That is, using as few core modules as possible to reduce model complexity.
- Effectiveness. That is, good classification and dense prediction performance.
- Efficiency. That is, fewer parameters and computational accuracy trade-offs.
The following table summarizes the differences between this method and several other mainstream lightweight models:
We can observe the following phenomena:
- The performance of the CNN-based MobileNet series now seems slightly lower, and its parameters are slightly higher than its peers;
- The recently proposed MobileViT series has achieved better performance, but they have high FLOPs and poor efficiency;
- The main issue with EdgeNeXt and EdgeViT is that the design is not elegant, and the modules are relatively complex;
Based on the above criteria, the authors designed an efficient model similar to ResNet composed of multiple stacked iRMB modules—EMO, which primarily exhibits the following advantages:
1) For the overall framework, EMO consists only of iRMBs, without diversified modules, which can be considered a design philosophy of simplicity;
2) For specific modules, iRMB consists only of standard convolutions and multi-head self-attention, without any other complex operators. Additionally, benefiting from DW-Conv, iRMB can adaptively downsample through stride operations without requiring any positional embeddings to introduce positional bias into MHSA;
3) For the network’s variant settings, the authors adopt gradually increasing expansion rates and channel numbers, as detailed in the table below.
Since MHSA is more suitable for modeling deeper semantic features, EMO only employs it in the 3rd and 4th stages. To further enhance EMO’s stability and efficiency, the authors also introduce a combination of BN and SiLU in the 1st and 2nd stages, while replacing it with LN and GeLU in the 3rd and 4th stages, which is also the preferred configuration for most CNN and Transformer models.
Experiments
Parameter Comparison
First, let’s look at the relevant hyperparameter comparison between EMO and other lightweight networks:
It can be seen that EMO does not use a large amount of strong DataAug and Tricks, which also fully reflects the effectiveness of its module design.
Performance Metrics
Overall, EMO performs strongly across the three foundational tasks of image classification, object detection, and semantic segmentation, achieving more competitive results with fewer computations and parameters.
Visualization Results
From the above visualization results, it can be clearly observed that the proposed method performs better in detail segmentation.
To better illustrate the effectiveness of the proposed method, the authors further employed the Grad-CAM method to highlight the relevant areas of different models. As shown in the above figure, the CNN-based ResNet tends to focus on specific objects, while the Transformer-based MPViT pays more attention to global features. In contrast, EMO can more accurately focus on salient objects while maintaining the ability to perceive global areas. This somewhat explains why EMO achieves better results across various tasks.
As mentioned earlier, the cascading of Convolution and MHSA operations can effectively enhance the expansion speed of the receptive field. To verify the effectiveness of this design, the similarity of diagonal pixels with different compositions in the 3rd Stage is visualized, specifically visualizing DW-Conv and EW-MHSA as well as the combination of both modules.
It can be seen that, whether from quantitative or qualitative experimental results, when only using DW-Conv, features often have short-range correlations, while EW-MHSA brings more long-range correlations. In contrast, when both are used simultaneously, the network has modules with a larger receptive field, i.e., better modeling of long-distance contextual information.
Finally, the ablation experiments of this article are presented. Overall, the experimental section is quite substantial. Interested friends can refer to the original text; due to time constraints, we will analyze it here.
Conclusion
This article explores the design of efficient architectures for mobile applications. By rethinking the efficient Inverted Residual Block in MobileNetv2 and the essence of effective Transformers in ViT, the authors introduce a universal concept called Meta Mobile Block, which leads to the derivation of a simple yet efficient modern iRMB module. Specifically, this module contains two core components, namely DW-Conv and EW-MHSA, which can fully utilize the efficiency of CNNs to model short-range dependencies while combining the dynamic modeling capability of Transformers to learn long-range interactions. Finally, by stacking iRMB modules of different scales, an efficient ResNet-like architecture—EMO—is constructed, which has been proven through extensive experiments on the ImageNet-1K, COCO2017, and ADE20K benchmarks to outperform other CNN or Transformer-based SoTA methods.
Reply ‘Dataset’ in the public account background to get 100+ resources sorted in various directions of deep learning.
Jishi Dry Goods
Technical Column: Detailed interpretation column on multimodal large models｜ Understanding the Transformer series｜ ICCV2023 paper interpretation｜ Jishi LiveJishi Perspective Dynamics: Welcome to apply for the Jishi Perspective 2023 Ministry of Education Industry-University Cooperation Collaborative Education Project｜New Horizons + Intelligent Brain, ‘Drones + AI’ become good helpers for intelligent road inspection!Technical Review: A detailed explanation of Neural ODE: Using neural networks to describe non-discrete state changes｜ What are the details of transformers? 18 questions about Transformers!
Click to read the original text and enter the CV community
Gain more technical dry goods