Title:A synergistic CNN-transformer network with pooling attention fusion for hyperspectral image classification
Paper Link:https://doi.org/10.1016/j.dsp.2025.105070
-
Collaborative CNN-Transformer Architecture Design A synergistic CNN-Transformer network is proposed, combining the local spatial feature extraction capability of CNNs with the global modeling capability of Transformers, effectively achieving joint modeling of spectral and spatial information in hyperspectral images (HSI).
-
Two-Branch Feature Extraction Module (TBFE) A dual-branch structure is introduced, using 2D convolution to extract spatial features and 3D convolution to extract spectral features, fully integrating spectral-spatial information and enhancing the expressiveness in the early feature extraction stage.
-
Hybrid Pooling Attention Module (HPA) A pooling attention mechanism combining average pooling and max pooling is designed to enhance information interaction between channels and spatial dimensions, strengthening the global receptive field and detail expression of feature maps.
-
Cross-Layer Feature Fusion Module (CFF) A simple yet effective cross-layer fusion mechanism is proposed, which performs weighted fusion of outputs from different Transformer encoder layers to reduce information loss and enhance the expressiveness of deep semantic features.
-
End-to-End Optimization Design and Extensive Experimental Validation Systematic experiments were conducted on five hyperspectral remote sensing datasets (such as Salinas, Houston2013, WHU-Hi-HanChuan, etc.), and ablation experiments validated the effectiveness of each module, outperforming various state-of-the-art models (such as MASSFormer, SSFTT, SS-Mamba, etc.).
Overall Structure This paper proposes a collaborative CNN-Transformer network structure for hyperspectral image classification. The model first preprocesses the original data through PCA dimensionality reduction and patch segmentation, then employs the Two-Branch Feature Extraction Module (TBFE) to utilize 2D and 3D convolutions to extract spatial and spectral features, and enhances features through the Hybrid Pooling Attention Module (HPA) in both channel and spatial dimensions. Subsequently, the fused features are input into multiple layers of Vision Transformer encoders to extract deep semantic information, and the Cross-Layer Feature Fusion Module (CFF) integrates features from different encoding layers, finally using a fully connected layer to achieve classification predictions for each pixel.
Data Preprocessing Module (PCA + Patch)
-
Principal Component Analysis (PCA) is used to reduce the spectral dimension of high-dimensional HSI data from
C B
2. Hybrid Pooling Attention Module (HPA)
The feature maps output from TBFE are grouped, and average pooling + max pooling is used to generate attention weights in the spatial dimension. The attention is used to enhance local and global channel interactions and spatial information integration.
3. Vision Transformer Encoder
The feature maps output from HPA are flattened into tokens, with position encoding and classification tokens added. Multiple layers of Transformer encoders are used to model high-level semantics and long-range dependencies.
4. Cross-Layer Feature Fusion Module (CFF)
The outputs from multiple previous Transformer encoding layers are weighted and fused with the initial input to reduce deep information loss. Finally, a fully connected layer outputs the classification results for the center pixel of each patch. This model achieves joint learning of spatial, spectral, local, and global information throughout the structure, enhancing expressiveness and stability through attention mechanisms and skip connections.
In deep learning and image processing tasks, “ DSP (Digital Signal Processing) Plug-and-Play Module refers to signal processing components or mechanisms that can be easily integrated into existing network structures, typically provided in a modular way, allowing usage without major modifications to the original architecture.
1. The DSP Plug-and-Play Module is mainly suitable for the following scenarios:
1. Image and Hyperspectral Image Enhancement
For example: denoising, deblurring, artifact removal, enhancing edge clarity. DSP modules can perform traditional processing such as frequency domain filtering, wavelet transforms, and edge enhancement, effectively improving the quality of input images.
2. Preprocessing Stage Before Feature Extraction
Adding methods such as PCA, Fourier transforms, and wavelet decomposition before data is fed into deep networks can reduce redundant information, highlight key features, and accelerate convergence speed.
3. Data Compression and Accelerated Inference
Using signal processing methods (such as sparse coding, quantization) to compress feature maps or the model itself achieves model lightweighting and reduces computational costs, especially suitable for deployment on edge devices.
4. Multimodal/Multi-source Fusion Tasks
DSP modules can be used for unified alignment, filtering, and synchronization of signals from different modalities (such as LiDAR, SAR, infrared), which is very practical in fields like remote sensing and autonomous driving.
5. Few-Shot Learning and Sample Augmentation
Utilizing traditional signal transformations (such as frequency domain perturbations, filtering perturbations) to generate diverse samples enhances the model’s generalization ability for small sample data.
In summary:
The greatest value of the DSP plug-and-play module lies in its ability to enhance the robustness, efficiency, and expressiveness of the model using traditional signal processing techniques without compromising the main network structure, especially suitable for scenarios sensitive to noise, data scarcity, or requiring cross-modal processing.
import torch
from torch import nn# Hybrid pooling attention (HPA)
class HPA(nn.Module):
def __init__(self, channels, c2=None, factor=32):
super(HPA, self).__init__()
self.groups = factor
assert channels // self.groups > 0
self.softmax = nn.Softmax(-1)
self.agp = nn.AdaptiveAvgPool2d((1, 1))
self.map = nn.AdaptiveMaxPool2d((1, 1))
self.pool_h = nn.AdaptiveAvgPool2d((None, 1)) #Y avg
self.pool_w = nn.AdaptiveAvgPool2d((1, None)) #X avg
self.max_h = nn.AdaptiveMaxPool2d((None, 1)) #Y avg
self.max_w = nn.AdaptiveMaxPool2d((1, None)) #X avg
self.gn = nn.GroupNorm(channels // self.groups, channels // self.groups)
self.conv1x1 = nn.Conv2d(channels // self.groups, channels // self.groups, kernel_size=1, stride=1, padding=0)
self.conv3x3 = nn.Conv2d(channels // self.groups, channels // self.groups, kernel_size=3, stride=1, padding=1)
def forward(self, x):
b, c, h, w = x.size()
group_x = x.reshape(b * self.groups, -1, h, w) # b*g,c//g,h,w --->2048,2,11,11
x_h = self.pool_h(group_x) #2048,2,11,1
x_w = self.pool_w(group_x).permute(0, 1, 3, 2) #2048,2,1,11--->2048,2,11,1
hw = self.conv1x1(torch.cat([x_h, x_w], dim=2)) #2048,2,22,1
x_h, x_w = torch.split(hw, [h, w], dim=2) #2048,2,11,1
x1 = self.gn(group_x * x_h.sigmoid() * x_w.permute(0, 1, 3, 2).sigmoid()) #2048,2,11,11
x2 = self.conv3x3(group_x) #2048,2,11,11
y_h = self.max_h(group_x) #2048,2,11,1
y_w = self.max_w(group_x).permute(0, 1, 3, 2)
yhw = self.conv1x1(torch.cat([y_h, y_w], dim=2)) #2048,2,22,1
y_h, y_w = torch.split(yhw, [h, w], dim=2) #2048,2,11,1
y1 = self.gn(group_x * y_h.sigmoid() * y_w.permute(0, 1, 3, 2).sigmoid()) #2048,2,11,11
y11 = y1.reshape(b * self.groups, c // self.groups, -1) # b*g, c//g, hw 2048,2,121
y12 = self.softmax(self.map(y1).reshape(b * self.groups, -1, 1).permute(0, 2, 1)) #2048,1,2
x11 = x1.reshape(b * self.groups, c // self.groups, -1) # b*g, c//g, hw 2048,2,121
x12 = self.softmax(self.agp(x1).reshape(b * self.groups, -1, 1).permute(0, 2, 1)) #2048,2,1,1-->2048,2,1--->2048,1,2
x21 = x2.reshape(b * self.groups, c // self.groups, -1) # b*g, c//g, hw #2048,2,121
x22 = self.softmax(self.agp(x2).reshape(b * self.groups, -1, 1).permute(0, 2, 1)) #2048,2,1,1-->2048,2,1--->2048,1,2
weights = (torch.matmul(x12, y11) + torch.matmul(y12, x11)).reshape(b * self.groups, 1, h, w)
return (group_x * weights.sigmoid()).reshape(b, c, h, w)
if __name__ == "__main__":
# Module parameters
batch_size = 1 # Batch size
channels = 32 # Number of input feature channels
height = 256 # Image height
width = 256 # Image width
model = HPA(channels=channels)
print(model) # Generate random input tensor (batch_size, channels, height, width)
x = torch.randn(batch_size, channels, height, width)
# Print the shape of the input tensor
print("Input shape:", x.shape)
# Forward propagation to compute output
output = model(x)
# Print the shape of the output tensor
print("Output shape:", output.shape)For more analysis, see the original text
