Task-driven infrared and visible image fusion via detail and semantic dual injection.

Abstract

Recently, introducing task-related information into the fusion of infrared and visible images has improved the practicality of fused images in real scenarios. However, due to the heterogeneity of fused features and task-related features, it is difficult to integrate task information into fused images, resulting in the quality degradation. To alleviate this issue, we propose a task-driven visible and infrared image fusion network called TDFuse, which implements dual injection of detail and semantic information from the downstream task into fused images. Specifically, the detail injection module (DIM) and semantic injection module (SIM) are designed to progressively distill the task-related detail and semantic information into the features of fused images, respectively. In DIM, the task detail enhancement block is constructed to combine fusion features with the detail information from the task network. The task semantic enhancement block in SIM is responsible for the injection of the semantic information in the task network. Meanwhile, a detail-semantic dual constraint loss is devised to ensure that DIM and SIM can adaptively learn the corresponding detail and semantic information from the task network. Through the dual injection manner, the fused images contain more detail and semantic information from the downstream task. Finally, extensive experiments on the M³FD, MSRS, LLVIP and downstream tasks demonstrate that our method not only ensures the visual effect of the fused images but also enhances the performance of downstream tasks. The code is available at https://github.com/Fullness-1/TDFuse.