From external to internal: Step-wise feature enhancement network for image-text retrieval.

Abstract

Image-Text Retrieval (ITR) is a challenging task due to the inherent inconsistency in feature representations across different modalities, commonly referred to as the "heterogeneity gap". To bridge this gap, establishing stronger associations between images and texts by capturing semantic cues as comprehensively as possible is an effective approach. However, existing ITR methods cannot completely capture semantic cues derived from a large-scale image-text corpus beyond a single image-text pair. Therefore, we propose a two-layer Step-wise Feature Enhancement (SFE) Network to establish a semantic propagation pathway, guiding semantic information flow progressively from the external layer to the internal layer. In Step 1, External Semantic Cues (ESC) are captured from visual and textual semantic concepts based on patch-level, instance-level, and neighbor-level co-occurrences within an image-text corpus. Then, visual and textual features are enhanced in the external layer with ESC by mining co-occurrences at the patch, instance, and neighbor levels. Note that Instance-level and Neighbor-level co-occurrence belong to cross-modal ESC, which can significantly facilitate modality interaction in the external layer. In step 2, SFE first fuses semantic information propagated from step 1, and then enhances visual and textual features in the internal layer by mining Internal Semantic Cues (ISC) through cross-modal context. Specifically, visual and textual features are concatenated with their corresponding cross-modal contextual features to further enhance modality interaction within the internal layer. Experimental results demonstrate the superiority of the proposed SFE network over state-of-the-art ITR methods.