Title: A synergistic CNN-transformer network with pooling attention fusion for hyperspectral image classification
Paper Link: https://github.com/chenpeng052/SCT-Net
-
Twin-Branch Feature Extraction Module (TBFE): Parallel combination of3D Convolution (for spectral features) and2D Convolution (for spatial features), achieving spectral-spatial joint modeling early on.
-
Hybrid Pooling Attention Module (HPA): Combines average pooling and max pooling to capture spatial dependencies through horizontal/vertical global pooling, enhancing cross-channel and cross-spatial interaction capabilities.
-
Cross-layer Feature Fusion Module (CFF): Improves information interaction between Transformer encoding layers, reducing information loss through skip connection feature concatenation.
-
Collaborative CNN-Transformer Design: CNN is responsible for local spatial feature modeling, while Transformer handles global spectral dependency modeling, forming complementary advantages.
-
Comprehensive Experimental Validation: Outperforms existing SOTA methods on five typical datasets (Salinas, Pavia University, Houston2013, WHU-Hi-HanChuan, Houston2018).
Overall Structure
The model first reduces the dimensionality of hyperspectral data using PCA, then utilizes a twin-branch convolution module to extract spectral and spatial features in parallel, and achieves cross-channel and cross-spatial feature fusion through a hybrid pooling attention mechanism. Next, the Transformer encoder models global spectral dependencies while introducing cross-layer feature fusion to reduce deep information loss. Finally, the fused features are output through a fully connected layer to achieve precise classification of hyperspectral images.
-
Data Preprocessing: PCA dimensionality reduction is applied to the original hyperspectral data to retain the main spectral components, reducing dimensionality and overfitting risk.
-
Twin-Branch Feature Extraction (TBFE): Parallel use of 3D CNN and 2D CNN to extract and concatenate spectral and spatial features.
-
Hybrid Pooling Attention (HPA): After grouping the feature maps, average pooling and max pooling are performed horizontally/vertically to form attention weights that enhance cross-spatial and cross-channel interaction.
-
Vision Transformer Encoder: After embedding position encoding and classification token, global spectral-spatial dependencies are modeled through multi-head self-attention and MLP layers.
-
Cross-layer Feature Fusion (CFF): Fuses the outputs of the earlier encoder layers with the inputs to enhance cross-layer information interaction and reduce information loss.
-
Classification Layer: Classifies the central pixel through a fully connected layer to obtain land cover category labels.
1. Main Applicable Scenarios
-
Remote Sensing and Hyperspectral Image Processing: Preprocessing hyperspectral images (e.g., PCA dimensionality reduction, noise suppression, feature extraction) to provide clean and effective data input for subsequent deep models.
-
Audio and Speech Processing: In speech recognition, noise reduction, and acoustic modeling, DSP modules can quickly implement filtering, Fourier transforms, spectral analysis, and other operations.
-
Image and Video Analysis: In video encoding/decoding, image enhancement, edge detection, and other tasks, DSP is often inserted as a front-end or intermediate module to improve feature quality.
-
Communication Systems: Used for signal modulation/demodulation, channel equalization, and bit error rate detection, enhancing the reliability and speed of wireless communication.
-
Medical Imaging: Noise reduction, filtering, and feature extraction for signals such as EEG, ECG, and MRI, assisting in disease diagnosis.
2. Main Functions
-
Feature Enhancement: Enhances useful information through filtering, frequency domain transformation, and attention weighting, while weakening noise and redundancy.
-
Dimensionality Reduction and Compression: Techniques such as PCA, FFT, and wavelet transforms help reduce input dimensions and lower the computational complexity of subsequent models.
-
Cross-Domain Adaptation: As a plug-and-play module, DSP can be easily embedded into deep networks like CNN and Transformer, achieving feature supplementation across spatial, spectral, and temporal dimensions.
-
Efficiency Optimization: DSP operations are typically mathematically simple (convolution, transformation, filtering), which can reduce model computation and improve inference speed.
-
Robustness Improvement: By denoising, normalizing, and enhancing signal features, subsequent classification/recognition models become more robust.
In summary: DSP, as a plug-and-play module, is most suitable for scenarios involvingtime-frequency domain signals (such as hyperspectral images, speech, remote sensing, and medical signals), serving the purpose ofefficient preprocessing, feature enhancement, and dimensionality reduction, thereby improving the performance and robustness of deep learning models.
import torch
from torch import nn
# Hybrid pooling attention (HPA)
class HPA(nn.Module):
def __init__(self, channels, c2=None, factor=32):
super(HPA, self).__init__()
self.groups = factor
assert channels // self.groups > 0
self.softmax = nn.Softmax(-1)
self.agp = nn.AdaptiveAvgPool2d((1, 1))
self.map = nn.AdaptiveMaxPool2d((1, 1))
self.pool_h = nn.AdaptiveAvgPool2d((None, 1)) #Y avg
self.pool_w = nn.AdaptiveAvgPool2d((1, None)) #X avg
self.max_h = nn.AdaptiveMaxPool2d((None, 1)) #Y avg
self.max_w = nn.AdaptiveMaxPool2d((1, None)) #X avg
self.gn = nn.GroupNorm(channels // self.groups, channels // self.groups)
self.conv1x1 = nn.Conv2d(channels // self.groups, channels // self.groups, kernel_size=1, stride=1, padding=0)
self.conv3x3 = nn.Conv2d(channels // self.groups, channels // self.groups, kernel_size=3, stride=1, padding=1)
def forward(self, x):
b, c, h, w = x.size()
group_x = x.reshape(b * self.groups, -1, h, w) # b*g,c//g,h,w --->2048,2,11,11
x_h = self.pool_h(group_x) #2048,2,11,1
x_w = self.pool_w(group_x).permute(0, 1, 3, 2) #2048,2,1,11--->2048,2,11,1
hw = self.conv1x1(torch.cat([x_h, x_w], dim=2)) #2048,2,22,1
x_h, x_w = torch.split(hw, [h, w], dim=2) #2048,2,11,1
x1 = self.gn(group_x * x_h.sigmoid() * x_w.permute(0, 1, 3, 2).sigmoid()) #2048,2,11,11
x2 = self.conv3x3(group_x) #2048,2,11,11
y_h = self.max_h(group_x) #2048,2,11,1
y_w = self.max_w(group_x).permute(0, 1, 3, 2)
yhw = self.conv1x1(torch.cat([y_h, y_w], dim=2)) #2048,2,22,1
y_h, y_w = torch.split(yhw, [h, w], dim=2) #2048,2,11,1
y1 = self.gn(group_x * y_h.sigmoid() * y_w.permute(0, 1, 3, 2).sigmoid()) #2048,2,11,11
y11 = y1.reshape(b * self.groups, c // self.groups, -1) # b*g, c//g, hw 2048,2,121
y12 = self.softmax(self.map(y1).reshape(b * self.groups, -1, 1).permute(0, 2, 1)) #2048,1,2
x11 = x1.reshape(b * self.groups, c // self.groups, -1) # b*g, c//g, hw 2048,2,121
x12 = self.softmax(self.agp(x1).reshape(b * self.groups, -1, 1).permute(0, 2, 1)) #2048,2,1,1-->2048,2,1--->2048,1,2
x21 = x2.reshape(b * self.groups, c // self.groups, -1) # b*g, c//g, hw #2048,2,121
x22 = self.softmax(self.agp(x2).reshape(b * self.groups, -1, 1).permute(0, 2, 1)) #2048,2,1,1-->2048,2,1--->2048,1,2
weights = (torch.matmul(x12, y11) + torch.matmul(y12, x11)).reshape(b * self.groups, 1, h, w)
return (group_x * weights.sigmoid()).reshape(b, c, h, w)
if __name__ == "__main__":
# Module parameters
batch_size = 1 # Batch size
channels = 32 # Number of input feature channels
height = 256 # Image height
width = 256 # Image width
model = HPA(channels=channels)
print(model)
# Generate random input tensor (batch_size, channels, height, width)
x = torch.randn(batch_size, channels, height, width)
# Print the shape of the input tensor
print("Input shape:", x.shape)
# Forward propagation to compute output
output = model(x)
# Print the shape of the output tensor
print("Output shape:", output.shape)For more analysis, see the original text
