A Synergistic CNN-Transformer Network with Pooling Attention Fusion for Hyperspectral Image Classification

Title:A synergistic CNN-transformer network with pooling attention fusion for hyperspectral image classification

Paper Link:

https://www.sciencedirect.com/science/article/abs/pii/S1051200425000922

1. Two-Branch Feature Extraction Module (TBFE): Utilizes 2D and 3D convolutions in parallel to extract spatial and spectral features, effectively fusing multidimensional information.

2. Hybrid Pooling Attention Module (HPA): Combines average pooling and max pooling to achieve information aggregation across spatial dimensions, enhancing attention expression capabilities.

3. Cross-Layer Feature Fusion Module (CFF): Introduces skip connections between Transformer encoders to effectively mitigate information loss during inter-layer transmission.

4. Achieved superior performance compared to state-of-the-art methods on five mainstream datasets (Salinas, PaviaU, Houston2013, WHU-Hi-HanChuan, Houston2018).

Overall Structure

The proposed model uses PCA-reduced hyperspectral image blocks as input, first employing a two-branch feature extraction module to extract spectral and spatial features in parallel using 3D and 2D convolutions, followed by a hybrid pooling attention module for cross-spatial attention modeling, then utilizing multi-layer Transformer encoders for global feature modeling, and finally fusing encoded information from different layers through the cross-layer feature fusion module, ultimately completing the classification task for each pixel through a fully connected layer.

1. • PCA Dimensionality Reduction: Performs principal component analysis on the original hyperspectral image to reduce spectral dimensions.

     Patch Extraction: Generates 3D image blocks based on the reduced data, extracting training samples centered on pixels.

     TBFE (Twin-Branch Feature Extraction) Module:

   1. One branch uses 3D convolution to focus on extracting spectral features.

   2. The other branch uses 2D convolution to extract spatial features.

   3. The results from both branches are concatenated along the channel dimension.

   HPA (Hybrid Pooling Attention) Module:

   1. Uses 1D average pooling and max pooling to encode spatial attention in horizontal and vertical directions, respectively.

   2. Incorporates GroupNorm and Sigmoid activation function to reconstruct the spatial attention map.

2. Transformer Encoder:

1. Models global features from the fused characteristics.

2. Processes token sequences using multi-head self-attention mechanisms.

3. Introduces positional encoding and class tokens.

3. CFF (Cross-layer Feature Fusion) Module:

1. Aggregates intermediate layer outputs from multiple Transformer encoders, fusing with the current input to enhance information integrity.

4. Fully Connected Layer: Outputs the final classification results.

Applicable Scenarios

1. Processing Multi-source Heterogeneous Remote Sensing Data

   The module can be flexibly inserted into existing CNN, Transformer, Mamba, and other network architectures to process remote sensing images with different spatial or spectral resolutions.

2. Complex Object Category Recognition

   In scenarios where object types are similar and distribution is complex (e.g., urban landscapes, crop classification), plug-and-play modules (like HPA) can enhance spatial feature recognition capabilities.

3. Data Sample Scarcity Scenarios

   The module has a lightweight structure and fewer parameters, making it suitable for situations with limited training samples (e.g., classification tasks with only 0.5%-5% labeled samples).

4. Model Expansion or Ensemble Learning

   When combining multiple feature extractors or attention mechanisms to build a more powerful network, such modules can be inserted as standard interfaces into other backbone networks (e.g., ViT, ResNet, ConvNext, etc.).

5. Real-time or High-Efficiency Scenarios

   Compared to fully reconstructing network structures, this module adds with lower computational overhead, improving performance without significantly increasing parameter count, suitable for resource-constrained platforms (e.g., drones, edge devices).

In Summary:

These plug-and-play modules are suitable for hyperspectral image classification tasks that require a balance of spectral-spatial feature extraction, attention enhancement, and information fusion, significantly improving the model’s discriminative ability and generalization performance while maintaining low computational overhead.

import torch
from torch import nn
class TBFE(nn.Module):
    def __init__(self, input_channels, reduction_N = 32):
        super(TBFE, self).__init__()
        self.point_wise = nn.Conv2d(input_channels,reduction_N,kernel_size=1,padding=0,bias=False) 
        self.depth_wise = nn.Sequential(nn.Conv2d(reduction_N, reduction_N, kernel_size=(3, 3),padding=1),nn.BatchNorm2d(reduction_N),nn.ReLU(),)

        self.conv3D = nn.Conv3d(in_channels=1, out_channels=1, kernel_size=(1,1,3),padding=(0,0,1),stride=(1,1,1),bias=False)
        self.bn = nn.BatchNorm2d(reduction_N)
        self.relu = nn.ReLU()
        
    def forward(self,x):
        x_1 = self.point_wise(x) 
        x_2 = self.depth_wise(x_1) 
        x_2=x_1+x_2
        
        #DSC
        x_3 = x_1.unsqueeze(1)
        x_3 = self.conv3D(x_3)
        x_3 = x_3.squeeze(1)
        x = torch.cat((x_2,x_3),dim=1)
        
        return x
    
if __name__ == "__main__":
    # Module parameters
    batch_size = 1    # Batch size
    channels = 32     # Input feature channels
    height = 256      # Image height
    width = 256        # Image width

    model = TBFE(input_channels = channels)
    print(model)    # Generate random input tensor (batch_size, channels, height, width)
    x = torch.randn(batch_size, channels, height, width)
    # Print input tensor shape
    print("Input shape:", x.shape)
    # Forward pass to compute output
    output = model(x)
    # Print output tensor shape
    print("Output shape:", output.shape)

For more analysis, see the original text.