SSDFusion: A Semantic Segmentation Driven Framework for Infrared and Visible Image Fusion

Fusing infrared images with visible images facilitates obtaining more abundant and accurate information content. However, existing infrared and visible image fusion methods often lack attention to the semantic information and global context information in the original images. To address these issues, we propose a novel deep learning framework for infrared and visible image fusion, which is named Semantic Segmentation Driven Infrared and Visible Image Fusion Framework (SSDFusion). Within the fusion framework, the Local Global Feature Extraction Fusion Module is employed, complemented by the decoder. Furthermore, under the guidance of semantic segmentation, SSDFusion achieves a better understanding of complex scene region information, enhancing fusion task performance. Finally, an adaptive loss function is implemented throughout SSDFusion to fine-tune the balance between the semantic segmentation task and the image fusion task by adjusting their proportional contributions. This approach aids in more accurately preserving the semantic information in the image, thereby enhancing the performance of the fusion framework. We conducted comparative experiments on the MSRS dataset with existing advanced fusion methods. The experimental results show that SSDFusion performs best in both qualitative and quantitative metrics. Analysis of the public datasets indicates that our algorithm can improve the entropy (EN), spatial frequency (SF), standard deviation (SD), mutual information (MI), visual information fidelity (VIF), and edge-based similarity measure (Q<inline-formula> <tex-math notation="LaTeX">${}_{\text {AB/F}}$ </tex-math></inline-formula>) metrics with about 15.33%, 91.55%, 17.09%, 93.39%, 66.94%, and 122.56% gains, respectively. The ablation study further demonstrates that the local global feature fusion module, the adaptive fusion loss function, and the integration of semantic segmentation and image fusion have significant effects on improving the model performance. SSDFusion also exhibits excellent performance in terms of computational efficiency and parameter count. Furthermore, we have also verified the good generalization ability of SSDFusion on the RoadScene and M3FD datasets.