GText-IF: Leveraging Text-Driven Semantics for Degradation-Aware Image Fusion.

Abstract

Image fusion aims to aggregate information from multiple source images to generate the fused image that provides a comprehensive representation. In the presence of degradations in source images, current image fusion methods struggle to adaptively accommodate them, resulting in residual degradations in the fused images and thereby impairing information representation. To address this issue, we propose an innovative degradation-aware image fusion framework via leveraging text-driven semantics, termed GText-IF. It extends the field of image fusion to the domain of text and image fusion and explores the feasibility of leveraging deep text and image coupling to guide degradation-aware fusion. First, by leveraging the powerful comprehension capability of vision-language models together with human prior knowledge, textual information is incorporated into degradation-aware image fusion as the semantic guidance. In this manner, text serves as a shared medium connecting humans and machines while providing foundational semantic cues. Second, semantic segmentation is ingeniously introduced as a bridge between textual and visual modalities, exploiting their intrinsic semantic consistency. The spatial representational properties of content information described by text are consistent with semantic segmentation, while the effective representation of semantic features in image content is also achieved through semantic segmentation. By integrating multiple decoders with a degradation prompt module, GText-IF can adaptively mitigate degradations in the fused results with the assistance of textual prompts and semantic guidance. Extensive experiments demonstrate that GText-IF can perform effective degradation-aware fusion in real degraded image scenarios, exhibiting remarkable advancement, robustness, and practicality. The code is available at https://github.com/XunpengYi/GText-IF.