↑ ClickBlue Text Follow the Jishi Platform
Author | Yumu Linfeng Source | AICV and Frontiers Editor | Jishi Platform
Jishi Introduction
The article proposes a new Superpixel Token Interaction Network (SPIN). This method uses superpixels to cluster locally similar pixels, forming interpretable local regions and achieving local information interaction through attention within superpixels. >> Join the Jishi CV technology group to stay at the forefront of computer vision
Article link: https://openaccess.thecvf.com/content/ICCV2023/papers/Zhang_Lightweight_Image_Super-Resolution_with_Superpixel_Token_Interaction_ICCV_2023_paper.pdf
Project link: https://github.com/ArcticHare105/SPIN
00 | Introduction
Transformer-based methods have shown impressive results in Single Image Super Resolution (SISR) tasks. However, when applied to whole images, the computational cost of self-attention mechanisms is very high.
Current Situation:
The current approach is to split the low-resolution input image into small patches, which are processed separately and then fused to generate a high-resolution image. However, this conventional patch division is too coarse and lacks interpretability, leading to artifacts and interference from dissimilar structures during the attention operation.
Solution:
A new Superpixel Token Interaction Network (SPIN) is proposed. This method uses superpixels to cluster locally similar pixels, forming interpretable local regions (since only similar areas complement each other, while dissimilar areas are excluded) and achieves local information interaction through attention within superpixels.
A superpixel cross-attention module is designed to facilitate information propagation through superpixel replacement. Extensive experiments show that the proposed SPIN model outperforms state-of-the-art SR methods in terms of accuracy and lightweight design.
Innovations
-
A new super-resolution model is proposed that combines superpixel clustering with transformer structures, resulting in a more interpretable framework.
-
The Intra-Superpixel Attention (ISPA) and Superpixel Cross Attention (SPCA) modules are introduced, which operate within and between superpixels, maintaining the ability to capture long-range dependencies while allowing computation in irregular regions.
01 | Method
The proposed SPIN model architecture is shown in Figure 2, primarily consisting of the proposed Superpixel Interaction (SPI) blocks. Before the SPI blocks, a 3 × 3 convolutional encoder is used to embed the low-resolution image ILR into a high-dimensional feature space. Given the encoder, shallow features xemb can be obtained as:
Then, K SPI blocks are stacked on top of the encoder to extract deeper features containing rich low-level and high-level information from the input image. Each SPI block consists of four parts: Superpixel Aggregation (SPA), Superpixel Cross Attention (SPCA), Intra-Superpixel Attention (ISPA), and Local Attention. The input features of each block are first aggregated into superpixels through the SPA module. Then, the ISPA module captures the dependencies and interactions of pixels within each superpixel, while the SPCA module captures dependencies and interactions between distant pixels. To enhance interactions among pixels within local regions, a local attention module is used after the ISPA and SPCA modules, which employs window-based attention to strengthen interactions within local areas. We use overlapping patches to enhance feature interaction. Formally, for the i-th SPI block, the entire process can be expressed as:
Where si represents the features of the superpixel in the i-th SPI block, and f(·) denotes the function for each individual component. In previous works, residual connections have been used to simplify the entire training process. After K SPI blocks, a 3×3 convolution layer and pixel shuffle operation are employed to obtain global residual information, which is then added to the upsampled image of ILR to solve for the high-resolution image ISR.
1) SPA Module
Unlike previous methods that divided input images or features into regular blocks, we propose to partition the input features into superpixels. Compared to conventional patches that easily cut connected regions into different patches, superpixel partitioning can perceptually group similar pixels together, depicting more accurate boundaries and reducing the risk of generating blurred and inaccurate boundaries.
Specifically, during the superpixel aggregation process, we used a soft k-means based superpixel algorithm in SSN. Given visual labels (where the number of visual labels is L), it is assumed that each label belongs to one of the K superpixels, thus it is necessary to compute the association between visual labels and superpixel labels.
Formally, the superpixel aggregation process is similar to an expectation-maximization process, which includes a total of T iterations. First, as shown in Figure 3, we sample the initial superlabels by averaging the labels in a regular grid, referred to as Patchify. Assuming the grid size is G, the number of supertokens is S. For the t-th iteration, the association mapping is calculated as:
Where A is the association mapping, and the value at the i-th row and j-th column. Note that superpixel aggregation only calculates the association mapping from each token to surrounding superpixels, ensuring the locality of superpixels, making it computationally and memory efficient. The superpixels can then be obtained as a weighted sum of visual labels, defined as:
Where N denotes the normalization term along the columns. After T iterations, the final association mapping AT can be obtained.
2) SPCA Module
Since super-resolution only captures the locality and interconnectivity of pixels in local regions, it may lack the ability to capture long-range dependencies of super-resolution. Here, we utilize the self-attention paradigm through superpixel replacement to enhance long-range communication, which helps leverage the complementarity between features to generate high-quality super-resolution images. Since pixel features are highly similar to their corresponding superpixel features, superpixels serve as promising alternatives for propagating information as much as possible among pixels.
As shown in Figure 4, given superpixel features S, where P represents the number of superpixels, and planar pixel features P. The attention mechanism is first employed to propagate pixel information to superpixels. Specifically, linear projections are used to compute queries: Q, keys: K, values: V as:
Where Wq, Wk, and Wv are the weight matrices based on query, key, and value respectively. The output can be obtained by first calculating the similarity between queries and keys and using it as weights to aggregate values, which can be expressed as:
Where √D is the scaling factor to avoid gradient vanishing, and su is the updated superpixel features. Note that unlike superpixel aggregation, this process does not consider neighbor constraints, ensuring the propagation of long-range information.
Once information is propagated from pixels to superpixels, it needs to be redistributed back to pixels for information propagation among pixels. Here, the attention mechanism is further applied. Specifically, another weight matrix Wq x is used to obtain queries from pixel features. To reduce the number of parameters, the superpixel features q are directly used as keys, and the updated superpixel features are used as values, mapping the updated superpixel features back to pixel level using cross attention.
Similar to the Transformer block, we also adopt a feed-forward network (FFN) after the above process. The FFN contains a layer normalization layer, followed by feature gating to modulate input features and channel attention to extract global information. Finally, two fully connected layers and a GELU activation function are used.
3) ISPA Module
With the association graph, an intuitive way to improve the quality of super-resolution images is to leverage the complementarity of similar pixels within the same superpixel. To achieve this, it is necessary to obtain the pixels corresponding to each superpixel. However, different superpixels may contain different numbers of pixels, making parallel processing challenging and leading to unexpected memory consumption, as there are always some superpixels containing a large number of pixels.
To address this issue, as shown in Figure 5, we use the association graph A and select the top-N pixels most similar to each superpixel. Assuming that the pixels associated with a superpixel are P, where N is the number of selected pixels. We perform intra-superpixel attention using the standard self-attention mechanism, i.e., Eq. 5 and Eq. 6, which includes weight matrices Wq and Wk for queries, keys, and values projections. After superpixel interaction, the fine pixel features are dispersed back to their respective positions in the image using the indices generated during the top-N selection process.
The top-N selection may lead to some “ignored” pixels, which are not included in any superpixel. For those “ignored” pixels, we use the value projection V to project them to obtain updated features, and then integrate them with the pixels that have been updated through intra-superpixel interaction. Similar to the SPCA module, the same FFN is adopted after the ISPA module.
02 | Experimental Results
Comparison of average PSNR/SSIM with other state-of-the-art CNN and transformer-based SISR models. The best and second-best results are highlighted and underlined, showing that SPIN achieves SOTA results.
Qualitative comparison of state-of-the-art classical and lightweight transformer SR models for ×4 upscaling tasks. Compared to other models, our (SPIN) can restore more accurate and clearer details.
References
https://openaccess.thecvf.com/content/ICCV2023/papers/Zhang_Lightweight_Image_SuperResolution_with_Superpixel_Token_Interaction_ICCV_2023_paper.pdf
Reply “Dataset” in the public account backend to obtain a collection of 100+ resources in various directions of deep learning
Jishi Essentials
Technical Column: Detailed Interpretation Column of Multi-modal Large Models｜ Understanding Transformer Series｜ ICCV2023 Paper Interpretation｜ Jishi LiveJishi Perspective Dynamics: Welcome university teachers and students to apply for the Jishi Perspective 2023 Ministry of Education Industry-University Cooperation Collaborative Education Project｜New Horizons + Smart Brain, “Drones + AI” become good helpers for intelligent road inspection!Technical Review: 40,000-word detailed explanation of Neural ODE: Using neural networks to describe non-discrete state changes｜ What are the details of transformers? 18 Questions about Transformers!
Click to read the original text and enter the CV community
Gain more technical insights