CrossFuse: A Method for Infrared and Visible Light Image Fusion Based on Cross-Sensor Visual Alignment

Click the blue text above to follow, and remember to subscribe 🌟

Paper link:

https://arxiv.org/pdf/2502.14493

Introduction

Infrared and visible light image fusion (IVIF) is increasingly applied in critical fields such as video surveillance and autonomous driving systems. Deep learning-based fusion methods have made significant progress; however, these models often encounter out-of-distribution (OOD) scenarios in real-world applications, severely affecting their performance and reliability. Therefore, addressing the OOD data challenge is crucial for the safe deployment of these models in open-world environments. Unlike existing studies, this paper focuses on the challenges posed by OOD data in practical applications and aims to enhance the robustness and generalization ability of the models. This paper proposes a multi-view enhanced infrared-visible fusion framework. In terms of external data augmentation, a Top-k selective visual alignment technique is employed, effectively alleviating distribution shifts between datasets by transforming the RGB dimensions of visible light images. This strategy significantly increases the number of augmented samples and enhances the model’s adaptability to complex real-world scenarios. For internal data augmentation, a weak-strong augmentation combination is used to establish a self-supervised learning mechanism, allowing the model to learn more robust and general feature representations during the fusion process, thereby improving robustness and generalization performance. Extensive experiments demonstrate that the proposed method exhibits outstanding performance and robustness under various conditions and environments, significantly enhancing the reliability and stability of IVIF tasks in practical applications.

Research Motivation

Distribution Drift Challenge: Existing fusion models are typically trained in idealized, static closed environments, where the test data and training data come from the same labeled feature space distribution (ID).
Deployment Difficulties: In practical applications, models face data from unknown domains or distributions (OOD data), leading to performance degradation and affecting safe deployment.
Severe Performance Degradation: As shown in Table 1, when the test distribution deviates from the training data, existing methods experience varying degrees of performance degradation.

To address the above challenges, the paper adopts a data centering strategy to enhance model performance through cross-dataset learning algorithms: External Perspective: Using RGB channel alignment combined with Top-K selection to reduce the distribution gap between external data and target data. Internal Perspective: Introducing weak-strong data augmentation to establish self-supervised learning and enhance model robustness.

CrossFuse Model

The CrossFuse framework mainly consists of three core components:

External Data Augmentation: Top-k selective channel alignment
Internal Data Augmentation: Weak-strong augmentation self-supervised learning
Frequency-Aware Fusion Network: Multi-scale feature extraction and fusion

The overall framework is shown in Figure 1. Through a multi-view data augmentation strategy, the model can obtain more general and robust feature representations.

CrossFuse: A Method for Infrared and Visible Light Image Fusion Based on Cross-Sensor Visual Alignment Figure 1. The overall framework of the proposed method. External data augmentation employs Top-K selective channel alignment technology to effectively expand the training data scale; internal data augmentation further utilizes weak-strong augmentation strategies for self-supervised learning. The frequency-aware fusion network achieves more comprehensive and detailed feature extraction and fusion.

External Data Augmentation: Top-k Selective Channel Alignment

To better address the challenges posed by out-of-distribution (OOD) data in real-world applications, a multi-view data augmentation method is proposed. In the external view, the Top-k selective channel alignment technique is introduced to process auxiliary datasets. This augmentation strategy aligns external data with target data through gamma transformation of the RGB channels, thereby effectively increasing the sample size. In the internal view, a weak-strong augmentation strategy is employed to establish a self-supervised learning paradigm. By performing contrastive learning on different augmented views, our model can generate robust fusion results. This multi-view data augmentation strategy significantly enhances the model’s adaptability and generalization ability to OOD scenarios in real-world applications.

As shown in Figure 2, the same cropping operation (64×64 image blocks) is first applied to both the target dataset and the external dataset, and gamma transformation is used to align the RGB channels:

Finally, Top-K selection is performed based on pixel distribution similarity:

Internal Data Augmentation: Self-Supervised Learning

Internal view enhancement for self-supervised learning. To improve the model’s cross-domain generalization ability and enhance fusion quality, a weak-strong augmentation strategy is adopted from the internal data perspective for self-supervised learning. As shown in Figure 5, a progressive weak-strong augmentation technique generates associated multi-level augmented views for the model to perform contrastive learning. First, weak augmentation operations are applied to the input to generate corresponding weakly augmented views for fusion results:

Weak Augmentation: Random cropping generates weakly augmented views

Strong Augmentation: Gaussian blur generates strongly augmented views

Contrastive Learning: Learning robust feature representations by comparing different augmented views

Frequency-Aware Fusion Network

This paper constructs a frequency-aware fusion network to extract and integrate global information and local features at multiple scales. As shown in Figure 1, this fusion network includes multi-scale feature extraction and fusion modules. This structure allows the fused image to retain complete global structural information and local texture details from a frequency perspective.

Multi-scale feature extraction and fusion: Generally, the symbiotic features between modalities (such as common backgrounds and large-scale environmental characteristics) are primarily reflected in the low-frequency domain, while high-frequency information emphasizes the detailed features within each modality. The proposed fusion network effectively enriches the information content of the fused image and enhances the representation capability of scene targets by extracting and integrating multi-scale modal features.

As shown in Figure 6, different feature extractors are used to extract diverse feature representations from the source images. First, shallow features are extracted from infrared and visible light inputs using Restormer blocks. Then, a long-scale feature extractor is constructed based on Transformer blocks to capture long-distance dependencies. Finally, a short-scale feature extractor is built using convolutional blocks to capture local features. The fusion process is as follows:

(1) Extract shallow features

(2) Extract low-frequency and high-frequency features separately – Low-frequency:. High-frequency:

(3) Fuse features of different scales:

(4) Generate the final fused image:

Loss Function Design

The paper adopts a two-stage training strategy:

First Stage (Feature Reconstruction):

Second Stage (Feature Fusion):

Where the self-supervised loss:

Experimental Results Analysis

Dataset Settings

Training: RoadScene dataset (181 pairs of training samples, 40 pairs of test samples)
Auxiliary: M3FD dataset (102 pairs of images)
Testing: MSRS, TNO datasets for OOD evaluation

Quantitative Results

The evaluation results on three test datasets show:

Qualitative Results Analysis

The fusion results can highlight foreground targets in dark areas, effectively retaining the rich texture information of visible light images and the thermal radiation features of infrared images, with background structures and texture details exhibiting clearer edge contours.

Ablation Experiments

Validated the effectiveness of key modules:

Top-k Selective Channel Alignment: Improved EN and SCD metrics, enhancing data consistency.
Internal Data Augmentation: Removal of this component led to significant declines in MI and SSIM values.
Multi-Scale Feature Extraction: Replacing any component would lead to declines in all metrics.

Thanks to the authors!Please cite the source!For detailed implementation processes and specific explanations of the paper, please read the original paper～❤️❤️ / Welcome to submit