Infrared and visible image fusion is vital for advanced visual tasks like object detection and autonomous driving, integrating complementary information from the two modalities. Visible images provide rich textures but are sensitive to illumination changes and occlusions, while infrared images highlight thermal targets regardless of lighting but suffer from low resolution and weak textures. Existing methods often fail to balance thermal structure preservation and texture retention, especially under extreme conditions (e.g., low light, smoke), leading to key information loss and poor visual quality. A novel framework is thus needed to enhance target focusing, preserve domain-specific features, and enable deep cross-modal interaction.
A fusion framework called rectangular self-calibration and cross-domain interaction fusion (RSCFusion) was proposed, comprising three core modules: rectangular self-calibration module (RCM), thermal sense map convolution module (TSMC), and cross-domain interaction module (CIM) (Fig.1). For visible images, RCM captured axial global context via horizontal/vertical pooling, fused features through broadcast addition, and generated adaptive attention maps to focus on foreground targets. For infrared images, TSMC introduced conditional position encoding (CPE) and deep convolution to preserve thermal structures and reduce edge blurring. CIM integrated local spatial attention (LSA) and global interactive semantic attention (GISA) for deep cross-modal complementarity. A comprehensive loss function (content, gradient, semantic) guided network training. Experiments were conducted on MSRS and M3FD datasets, using six metrics for comparison with seven state-of-the-art methods.
Qualitative results showed that RSCFusion generated naturally bright fused images with clear thermal targets and intact details on MSRS (Fig.5) and handled extreme conditions effectively on M3FD (Fig.6). Quantitative evaluations on 20 image pairs revealed outstanding performance: on MSRS, entropy (6.649), standard deviation (8.053), peak signal-to-noise ratio (PSNR=64.405), and mean squared error (0.026) improved by 11.2%, 11.8%, 0.7%, and 13.2% (Table 1). On M3FD, entropy (7.498), correlation coefficient (0.599), standard deviation (10.305), and PSNR (64.516) increased by 14.7%, 13.3%, 5.4%, and 4.8% (Table 2). Ablation experiments confirmed module synergy, with the full configuration achieving the highest entropy (7.310) and PSNR (63.558) (Table 3).
RSCFusion effectively addresses existing limitations via self-calibration and cross-domain interaction, generating fused images with richer information and stronger robustness under extreme conditions. Future work will focus on lightweight design and optimizing cross-modal alignment to improve real-time performance and handle extreme occlusions.
DownLoad: