A Synergistic CNN-Transformer Network with Pooling Attention Fusion for Hyperspectral Image Classification

Title: A synergistic CNN-transformer network with pooling attention fusion for hyperspectral image classification

Paper link: https://github.com/chenpeng052/SCT-Net

1. HPA (Hybrid Pooling Attention) Hybrid Pooling Attention: Divides channels into groups and performs global average pooling and global max pooling in both horizontal and vertical directions, followed by a 1×11 imes11×1 convolution and Sigmoid to obtain channel attention; simultaneously, it performs cross-spatial information aggregation (GroupNorm + global pooling + Softmax multiplication), enhancing pixel-level attention without reducing channels. This design establishes local and non-local dependencies without adding a large number of parameters.

2. Cascaded Transformer Encoder: Converts features after HPA into tokens (including position encoding and learnable[CLS][CLS][CLS], using multi-head self-attention to model long-range dependencies and enhance the expression of global spectral-spatial relationships.

3. CFF (Cross-layer Feature Fusion): Concatenates inputs with outputs from each layer encoder based on learnable weights and feeds them into the last layer, reducing information loss during deep propagation and enhancing discriminability.

4. End-to-end HSI Classification Pipeline: Uses PCA for dimensionality reduction + 3D patches as input, with a fully connected layer at the end of the network for classifying the center pixel of the patch; outperforms or matches SOTA on five representative datasets.

Overall Structure

The method first uses PCA to reduce the dimensionality of hyperspectral data and cuts it into 3D patches. It then extracts spectral (3D convolution) and spatial (2D convolution) features in parallel through two-level TBFE and fuses them along the channel dimension; subsequently, it models channel and spatial attention simultaneously using HPA Hybrid Pooling Attention without reducing channels, obtaining weighted tokens; these tokens, along with position encoding and [CLS][CLS][CLS] are input into the cascaded Transformer to model global dependencies, and with the help of CFF Cross-layer Fusion, shallow/deep semantic features are adaptively aggregated into the final representation, and finally, a fully connected head classifies the center pixel of the patch.

1. Input and Preprocessing: Perform PCA on the original HSI to reduce the spectral channels fromCCC toBBB, then generate 3D patches using a window ofP×PP imes PP×P with the center pixel as the label.

2. Two-Branch Feature Extraction (TBFE) ×2: First, use1×11 imes11×1 point convolution to adjust channels, then split into two branches:

   3D convolution branch: Expands to 4D, uses1×1×31 imes1 imes31×1×3 for spectral convolution, then squeezes back to 3D;

   2D convolution branch:3×33 imes33×3 spatial convolution; the outputs of both branches are concatenated along the channel dimension, with activation and normalization applied.

3. HPA Module: Performs grouped average/max 1D global pooling on the concatenated features (along H and W directions), merging to obtain channel attention; and generates spatial attention through cross-space aggregation, ultimately recalibrating input features through element-wise multiplication.

4. Transformer Encoder Stack: Adds position encoding and[CLS][CLS][CLS], using multi-head self-attention + MLP + LayerNorm for global relationship modeling.

5. Cross-layer Feature Fusion (CFF): Connects inputs with outputs from each layer encoder based on learnable weights, feeding into the last layer encoder to form a fused representation.

6. Classification Head: Fully connected layer reads the [CLS][CLS][CLS]/fused representation, outputting the class for the center pixel of the patch.

Applicable Scenarios

1. Hyperspectral Image Classification (HSI Classification): The DSP module is specifically designed for the spectrum-spatial joint modeling of hyperspectral images, particularly suitable for scenarios requiring simultaneous capture of local spectral correlations (fine-grained relationships between bands) and spatial contextual information (texture, edges, etc.).

2. Multimodal or Multi-channel Image Tasks: In medical imaging (MRI, PET), remote sensing images (multispectral), or even video tasks, if the data dimensions are high and exhibit significant cross-channel correlations, the DSP module can also serve as a pluggable unit to enhance the diversity and globality of feature extraction.

3. Enhancement of Transformer or Convolutional Networks: For existing CNN or Transformer backbones, DSP can be inserted as an intermediate feature enhancement module to compensate for the limitations of a single structure (e.g., CNN excels at local features, while Transformer focuses on global features).

2. Function

1. Joint Extraction of Spectral and Spatial Features: Achieves complementary enhancement through dual branches (3D CNN captures spectral features, 2D CNN captures spatial features), addressing the issue of insufficient features from a single path.

2. Introducing Attention Mechanism to Enhance Discriminability: DSP contains Pooling Attention, utilizing average pooling and max pooling to generate attention weights from horizontal/vertical directions, dynamically adjusting feature channels and spatial distribution, improving the utilization of effective information.

3. Improving Network Generalization and Transferability: As a plug-and-play module, it does not rely on specific downstream structures and can be embedded in different HSI classification frameworks or other visual models, enhancing overall classification performance.

4. Reducing Computational and Memory Overhead: Compared to directly stacking Transformers or large convolution kernels, DSP is designed to be lightweight (1×1 convolutions, grouped pooling, etc.), improving performance while avoiding computational bottlenecks.

In summary: DSP, as a plug-and-play module, is primarily used for spectrum-spatial joint modeling of hyperspectral and multi-channel images, enhancing feature expressiveness through dual-branch convolution + pooling attention and can be inserted into CNN or Transformer architectures to enhance classification and recognition effects.

import torch
from torch import nn


class TBFE(nn.Module):
    def __init__(self, input_channels, reduction_N = 32):
        super(TBFE, self).__init__()
        self.point_wise = nn.Conv2d(input_channels,reduction_N,kernel_size=1,padding=0,bias=False) 
        self.depth_wise = nn.Sequential(nn.Conv2d(reduction_N, reduction_N, kernel_size=(3, 3),padding=1),nn.BatchNorm2d(reduction_N),nn.ReLU(),)

        self.conv3D = nn.Conv3d(in_channels=1, out_channels=1, kernel_size=(1,1,3),padding=(0,0,1),stride=(1,1,1),bias=False)
        self.bn = nn.BatchNorm2d(reduction_N)
        self.relu = nn.ReLU()
        
    def forward(self,x):
        x_1 = self.point_wise(x) 
        x_2 = self.depth_wise(x_1) 
        x_2=x_1+x_2
        
        #DSC
        x_3 = x_1.unsqueeze(1)
        x_3 = self.conv3D(x_3)
        x_3 = x_3.squeeze(1)
        x = torch.cat((x_2,x_3),dim=1)
        
        return x
    
if __name__ == "__main__":
    # Module parameters
    batch_size = 1    # Batch size
    channels = 32     # Number of input feature channels
    height = 256      # Image height
    width = 256        # Image width

    model = TBFE(input_channels = channels)
    print(model)
    
    # Generate random input tensor (batch_size, channels, height, width)
    x = torch.randn(batch_size, channels, height, width)
    # Print input tensor shape
    print("Input shape:", x.shape)
    # Forward propagation to compute output
    output = model(x)
    # Print output tensor shape
    print("Output shape:", output.shape)

For more analysis, see the original text