Title:A synergistic CNN-transformer network with pooling attention fusion for hyperspectral image classification
Paper link: https://doi.org/10.1016/j.dsp.2025.105070
1. Proposed Synergistic CNN-Transformer Network Structure: Combines the local spatial feature extraction capabilities of CNNs with the global spectral modeling capabilities of Transformers to comprehensively extract spatial-spectral features from hyperspectral images (HSI).2. Twin-Branch Feature Extraction Module (TBFE): Parallel combination of 3D and 2D convolutions, used to extract spectral and spatial features respectively, designed reasonably, lightweight structure, capable of enhancing feature representation ability.3. Hybrid Pooling Attention Module (HPA): Combines average pooling and max pooling to mine local and global spatial attention information while maintaining the channel dimension, thereby enhancing spatial attention representation capability.4. Cross-Layer Feature Fusion Module (CFF): Reduces information loss by connecting the intermediate layer outputs of multiple Transformer encoders, improving the coherence and accuracy of deep semantic representation.
5. Excellent Performance Validation: Extensive experiments on five representative datasets show that the model outperforms various advanced methods in terms of overall accuracy, average accuracy, and Kappa coefficient.
Overall Structure
The model proposed in this paper is a synergistic network architecture that integrates Convolutional Neural Networks (CNN) with Transformers, specifically designed for hyperspectral image classification. First, PCA is used to reduce the dimensionality of hyperspectral images and extract 3D image blocks as input. Then, the input goes through a Twin-Branch Feature Extraction Module (TBFE), which captures spectral and spatial features using parallel 3D and 2D convolutions. The extracted features enter the Hybrid Pooling Attention Module (HPA), which integrates global information along the spatial dimension and enhances attention representation. Subsequently, the fused features are sent to a Vision Transformer structure composed of multiple stacked encoders for high-order modeling, and the Cross-Layer Feature Fusion Module (CFF) reduces information loss. Finally, the classification head outputs the land cover class for each pixel, achieving high-precision spatial-spectral joint classification.
1. Twin-Branch Feature Extraction Module (TBFE)
Structure: Parallel 3D convolution branch (for extracting spectral features) and 2D convolution branch (for extracting spatial features). Process: First, use 1×1 convolution for channel adjustment. The resulting feature maps are sent to the 3D and 2D convolution branches to extract deeper features. Finally, the outputs of the two branches are concatenated along the channel dimension.
2. Hybrid Pooling Attention Module (HPA)
Structure: A dual-branch attention mechanism that includes average pooling and max pooling. Function: Extracts channel attention in different directions along the spatial dimension. Adjusts the weights of different spatial regions of the image, enhancing feature representation capability. Uses GroupNorm and Softmax to improve training stability and generalization ability.
3. Vision Transformer Encoder
Input: Feature maps from the HPA module. Processing: The feature maps are embedded as token sequences and position encoding is added. High-order modeling is performed using Multi-Head Self-Attention (MSA) and Feed-Forward Networks (MLP). The final classification decision is represented by the [CLS] token.
Applicable Scenarios:
1. Hyperspectral images have high-dimensional spectral data, which can easily lead to overfitting and information redundancy, especially when training samples are limited.
This network is designed to accurately identify the land cover class of each pixel in remote sensing applications such as urban planning, agricultural monitoring, and geological disaster warning.
2. Complex model architecture combining CNN and Transformer
Suitable for scenarios that require both local feature extraction (CNN) and long-distance dependency modeling (Transformer).
3. Multi-scale and multi-dimensional information fusion tasks
Suitable for tasks that require joint processing of spatial and spectral information, such as few-shot learning and multi-class fine-grained classification in remote sensing image classification.
Role:
The DSP module essentially embodies the three core modules proposed in the paper, which can serve as plug-and-play modules embedded independently into other neural network structures to enhance the model’s expressive capability and classification performance:
1. TBFE (Twin-Branch Feature Extraction)
Uses parallel 2D and 3D convolutions to extract spatial and spectral features respectively.
Can be embedded into any front-end feature extraction structure to enhance the capture of local and spectral dependencies.
2. HPA (Hybrid Pooling Attention)
Integrates global average pooling and max pooling to generate spatial attention weights, enhancing inter-channel interaction.
Can be used in other networks to strengthen the performance of spatial attention mechanisms, especially suitable for high-resolution remote sensing scenarios.
3. CFF (Cross-layer Feature Fusion)
Cross-layer fusion of features from different Transformer encoders, reducing loss during information propagation.
Can be inserted into any Encoder-Decoder structure to enhance the integration of deep semantic information.
In Summary:
The DSP exists as a modular component in this paper, aimed at enhancing spatial and spectral feature extraction, attention modeling, and cross-layer fusion capabilities in hyperspectral image classification, applicable to various remote sensing tasks that integrate CNN and Transformer structures.
- Suitable for tasks that require joint processing of spatial and spectral information, such as few-shot learning and multi-class fine-grained classification in remote sensing image classification.
-
Uses parallel 2D and 3D convolutions to extract spatial and spectral features.
Can be embedded into any front-end feature extraction structure to enhance the capture of local and spectral dependencies.
-
Integrates global average pooling and max pooling to generate spatial attention weights, enhancing inter-channel interaction.
Can be used in other networks to strengthen the performance of spatial attention mechanisms, especially suitable for high-resolution remote sensing scenarios.
-
Cross-layer fusion of features from different Transformer encoders, reducing loss during information propagation.
Can be inserted into any Encoder-Decoder structure to enhance the integration of deep semantic information.
import torch
from torch import nn
class TBFE(nn.Module):
def __init__(self, input_channels, reduction_N = 32):
super(TBFE, self).__init__()
self.point_wise = nn.Conv2d(input_channels,reduction_N,kernel_size=1,padding=0,bias=False)
self.depth_wise = nn.Sequential(nn.Conv2d(reduction_N, reduction_N, kernel_size=(3, 3),padding=1),nn.BatchNorm2d(reduction_N),nn.ReLU(),)
self.conv3D = nn.Conv3d(in_channels=1, out_channels=1, kernel_size=(1,1,3),padding=(0,0,1),stride=(1,1,1),bias=False)
self.bn = nn.BatchNorm2d(reduction_N)
self.relu = nn.ReLU()
def forward(self,x):
x_1 = self.point_wise(x)
x_2 = self.depth_wise(x_1)
x_2=x_1+x_2
#DSC
x_3 = x_1.unsqueeze(1)
x_3 = self.conv3D(x_3)
x_3 = x_3.squeeze(1)
x = torch.cat((x_2,x_3),dim=1)
return x
if __name__ == "__main__":
# Module parameters
batch_size = 1 # Batch size
channels = 32 # Input feature channels
height = 256 # Image height
width = 256 # Image width
model = TBFE(input_channels = channels)
print(model) # Generate random input tensor (batch_size, channels, height, width)
x = torch.randn(batch_size, channels, height, width)
# Print input tensor shape
print("Input shape:", x.shape)
# Forward pass to compute output
output = model(x)
# Print output tensor shape
print("Output shape:", output.shape)For more analysis, see the original text
