Pseudo-Text Guided Robust Learning for Noisy Correspondence in Cross-Modal Retrieval.

Abstract

Noisy Correspondence (NC), caused by mismatched pairs in multimedia datasets, poses major challenges for cross-modal retrieval, especially under high noise levels. Existing solutions often suffer from substantial performance degradation as noise levels increase. To address this issue, we propose Pseudo-Text guided Robust Learning (PTRL), a novel framework designed to identify noisy pairs and enhance model robustness. Specifically, PTRL leverages pseudo-text as explicit supervision signals and introduces a new data division criterion to accurately distinguish between clean and noisy pairs. Instead of discarding or directly using noisy data, PTRL proposes a pseudo-text replacement strategy to maintain semantic consistency of the training set, thereby facilitating more reliable learning. In addition, pseudo-text-image pairs serve as a form of data augmentation, enriching data diversity and improving model generalization. To further stabilize training and mitigate overfitting, PTRL incorporates a robust InfoNCE loss that is particularly effective in the presence of noise. Extensive experiments demonstrate that PTRL achieves state-of-the-art performance and robustness, with an RSum improvements of +60.1% on Flickr30K and +22.6% on MS-COCO at an 80% noise level, significantly outperforming existing methods. The datasets and source code are available at https://github.com/shidan0122/PTRL.git.