A Synergistic CNN-Transformer Network with Pooling Attention Fusion for Hyperspectral Image Classification

Title: A Synergistic CNN-Transformer Network with Pooling Attention Fusion for Hyperspectral Image Classification

Paper Link:https://github.com/chenpeng052/SCT-Net

1. Collaborative Architecture: The “Synergistic CNN-Transformer” framework is proposed, utilizing the local spatial representation strengths of CNNs and the global spectral dependency modeling capabilities of Transformers for hyperspectral pixel-level classification.

2. TBFE Dual-Branch Feature Extraction: A parallel 3D convolution branch (1×1×3 kernel, expansion/compression mechanism) and a 2D convolution branch (3×3 kernel) synchronously extract spectral and spatial information in the early stages, adjusting channels and reducing complexity through 1×1 pointwise convolution.

3. HPA Hybrid Pooling Attention: After channel grouping, 1D global “average pooling + max pooling” is performed in parallel along horizontal/vertical directions to combine cross-spatial attention weights; this is further refined with GroupNorm and global pooling for precise recalibration of channel and spatial responses.

4. Cascaded Transformer Encoder: Multi-head self-attention and MLP mapping are applied to high-level semantic tokens to learn long-range spectral-spatial dependencies.

5. CFF Cross-Layer Feature Fusion: Outputs from multiple layers of encoders are adaptively concatenated/weighted with inputs to reduce information loss during inter-layer propagation and enhance inter-layer interaction.

Overall Structure

This method first reduces dimensionality using PCA and constructs P×P×B voxel blocks for HSI, followed by two consecutive TBFE dual-branch (parallel 3D/2D convolutions) to collaboratively extract spectral and spatial features; then, HPA hybrid pooling attention is used to merge “average + max” pooling information in both horizontal and vertical directions for precise feature recalibration; the weighted tokens are then input into the cascaded Transformer encoder for global dependency modeling, and multi-layer information is aggregated through CFF cross-layer fusion, ultimately classifying the central pixel using a fully connected layer.

1. PCA Dimensionality Reduction: The spectral dimension of the original HSI is reduced from C to B, alleviating overfitting and computational burden.

2. Patch Construction: Extract 3D small blocks using a window of P×P×B, with the central pixel label as supervision.

3. Two Consecutive TBFE Modules: 1×1 convolution adjusts channels → parallel 3D (1×1×3) and 2D (3×3) convolutions → concatenation and fusion, followed by lightweight 2D convolution and normalization/activation.

4. HPA Module: After grouping, global average pooling and max pooling are performed in horizontal/vertical directions to generate cross-space-channel attention, recalibrating the feature map.

5. Stacked Transformer Encoder: A learnable cls token and positional encoding are added, executing multi-head self-attention and MLP.

6. CFF Cross-Layer Fusion: Outputs from several previous encoders are adaptively fused with the current input into the last layer encoder input.

1. Applicable Scenarios

• As afront-end stem/feature extraction layer to replace or supplement the first few layers of existing models: one branch of 3D convolution models along the spectral dimension (1×1×3 small kernel + expand/squeeze to reduce complexity), while the other branch of 2D convolution captures local spatial textures, suitable for hyperspectral pixel-level classification tasks with strong “spectral-spatial” coupling.

• Hyperspectral dimensions, redundant multi data (e.g., WHU-Hi-HanChuan with 274 spectral bands) can achieve stronger early representation under low parameters after being reduced to an appropriate dimension via PCA before entering TBFE.

Module Function

• Using 1×1 pointwise to first adjust channels and reduce computational power, then 3D + 2D parallel to extract spectral/spatial features, finally concatenating and fusing, which is more robust than a single path; in ablation studies, using only TBFE significantly improved OA (+2.66%) compared to basic parallel convolution, indicating that its structural design (lightweight + parallel complementarity) is effective.

import torch
from torch import nn


class TBFE(nn.Module):
    def __init__(self, input_channels, reduction_N = 32):
        super(TBFE, self).__init__()
        self.point_wise = nn.Conv2d(input_channels,reduction_N,kernel_size=1,padding=0,bias=False) 
        self.depth_wise = nn.Sequential(nn.Conv2d(reduction_N, reduction_N, kernel_size=(3, 3),padding=1),nn.BatchNorm2d(reduction_N),nn.ReLU(),)

        self.conv3D = nn.Conv3d(in_channels=1, out_channels=1, kernel_size=(1,1,3),padding=(0,0,1),stride=(1,1,1),bias=False)
        self.bn = nn.BatchNorm2d(reduction_N)
        self.relu = nn.ReLU()
        
    def forward(self,x):
        x_1 = self.point_wise(x) 
        x_2 = self.depth_wise(x_1) 
        x_2=x_1+x_2
        
        #DSC
        x_3 = x_1.unsqueeze(1)
        x_3 = self.conv3D(x_3)
        x_3 = x_3.squeeze(1)
        x = torch.cat((x_2,x_3),dim=1)
        
        return x
    
if __name__ == "__main__":
    # Module parameters
    batch_size = 1    # Batch size
    channels = 32     # Input feature channels
    height = 256      # Image height
    width = 256        # Image width

    model = TBFE(input_channels = channels)
    print(model)
    
    # Generate random input tensor (batch_size, channels, height, width)
    x = torch.randn(batch_size, channels, height, width)
    # Print input tensor shape
    print("Input shape:", x.shape)
    # Forward pass to compute output
    output = model(x)
    # Print output tensor shape
    print("Output shape:", output.shape)

For more analysis, see the original text.