Title:A synergistic CNN-transformer network with pooling attention fusion for hyperspectral image classification
Paper Link:
https://www.sciencedirect.com/science/article/abs/pii/S1051200425000922?via%3Dihub
-
TBFE (Twin-Branch Feature Extraction): A parallel fusion of 2D and 3D convolutions, used to extract spatial and spectral features respectively.
-
HPA (Hybrid Pooling Attention): A designed attention mechanism that combines average pooling and max pooling to enhance channel representation in the spatial dimension.
-
CFF (Cross-layer Feature Fusion): Reduces information loss between different Transformer encoding layers through cross-layer feature fusion.
-
Experiments conducted on multiple datasets (such as Salinas, PaviaU, Houston2013, etc.) show significant improvement over existing methods (such as HybridSN, SpectralFormer, SS-Mamba, etc.)
Overall Structure
The model first reduces the dimensionality of hyperspectral images using PCA, then inputs them into the Twin-Branch Feature Extraction module (TBFE) composed of parallel 2D and 3D convolutions to extract spatial-spectral joint features. Next, the Hybrid Pooling Attention module (HPA) enhances the spatial attention distribution, followed by multiple Transformer encoders for global feature modeling, and utilizes the Cross-layer Feature Fusion module (CFF) to alleviate information loss in deep features, ultimately completing pixel classification through a fully connected layer.
-
Data Preprocessing: Dimensionality reduction using PCA to obtain hyperspectral images with fewer spectral channels.
-
TBFE Module: Input patches are sent in parallel to 2D and 3D convolution branches to extract spatial and spectral features, which are then concatenated.
-
HPA Module: Performs hybrid pooling attention operations on the fused features to enhance spatial attention modeling capabilities.
-
Transformer Encoder × N: After several cascaded Transformer encoders, further modeling of long-range dependency semantic features occurs.
-
CFF Module: Outputs from the previous encoder layers are fused into the last layer of the Transformer encoder through learnable weights, enhancing cross-layer information flow.
-
MLP Head: The output from the Transformer is sent to the MLP to predict the class of the central pixel in the patch.
TBFE is a lightweight structure with clear semantics and strong spatial-spectral decoupling modeling capabilities, suitable for scenarios requiring spatial-spectral joint modeling of high-dimensional multi-channel data, enhancing the model’s basic perception ability and downstream learning efficiency.
Applicable Scenarios
-
Tasks with highly coupled spatial and spectral (or channel) information
-
For example: hyperspectral image classification, medical imaging (such as multi-channel input for MRI, CT), remote sensing image analysis, etc.
-
Input is a 3D structured data block (Patch)
-
For example, a patch of size
P × P × B P B -
Desire to model local spatial information and local/neighborhood spectral information simultaneously in the shallow stage
-
To enhance the model’s initial perception ability and avoid weak modeling of a certain dimension in subsequent networks.
-
Need for efficient feature extraction in lightweight models but with limited computational budget
-
For instance, in edge deployment or situations with limited data, the pointwise convolution and branching structure of TBFE help reduce parameters and computational load.
Roles Played
-
Combining the advantages of 3D and 2D convolutions
-
3D convolution branch is better at modeling the contextual relationships in the **spectral dimension (between channels)**.
-
2D convolution branch captures spatial local patterns such as textures, edges, etc.
-
Effectively extracting spatial-spectral joint features in the shallow stage
-
Through early dual-path perception, it helps downstream modules (such as attention or transformer) to better utilize these foundational features.
-
Can be seamlessly inserted as a preprocessing encoding module into other networks
-
The module structure is independent and clear, maintaining the input-output shape, suitable for integration into CNN, Transformer, Mamba, and other backbone architectures.
-
Enhancing the model’s initial modeling capability and reducing the burden on the backbone
-
Helps the backbone network (such as transformer) focus more on high-level semantic relationship modeling rather than early processing of basic textures or local structures.
import torch
from torch import nn
class TBFE(nn.Module):
def __init__(self, input_channels, reduction_N = 32):
super(TBFE, self).__init__()
self.point_wise = nn.Conv2d(input_channels,reduction_N,kernel_size=1,padding=0,bias=False)
self.depth_wise = nn.Sequential(nn.Conv2d(reduction_N, reduction_N, kernel_size=(3, 3),padding=1),nn.BatchNorm2d(reduction_N),nn.ReLU(),)
self.conv3D = nn.Conv3d(in_channels=1, out_channels=1, kernel_size=(1,1,3),padding=(0,0,1),stride=(1,1,1),bias=False)
self.bn = nn.BatchNorm2d(reduction_N)
self.relu = nn.ReLU()
def forward(self,x):
x_1 = self.point_wise(x)
x_2 = self.depth_wise(x_1)
x_2=x_1+x_2
#DSC
x_3 = x_1.unsqueeze(1)
x_3 = self.conv3D(x_3)
x_3 = x_3.squeeze(1)
x = torch.cat((x_2,x_3),dim=1)
return x
if __name__ == "__main__":
# Module parameters
batch_size = 1 # Batch size
channels = 32 # Number of input feature channels
height = 256 # Image height
width = 256 # Image width
model = TBFE(input_channels = channels)
print(model)
# Generate random input tensor (batch_size, channels, height, width)
x = torch.randn(batch_size, channels, height, width)
# Print the shape of the input tensor
print("Input shape:", x.shape)
# Forward pass to compute output
output = model(x)
# Print the shape of the output tensor
print("Output shape:", output.shape)For more analysis, see the original text.
