Twin contrastive interventional-cause hashing for unsupervised cross-modal retrieval.

Abstract

Most unsupervised deep cross-modal hash retrieval (UDCMH) methods measure multimedia instances using similarity loss, while contrastive cross-modal hash retrieval (CMH) methods introduce contrastive loss. However, whether based on contrastive learning or not, these approaches are fundamentally similar, in which the contrastive learning module is plug-and-play. Meanwhile, they also have a problem of only using hashing as a semantic representation, lacking deeper analysis, and not integrating it into model construction or feature learning. In response to this dilemma, this paper presents twin contrastive interventional-cause hashing (TCICH) for unsupervised cross-modal retrieval. This represents the first attempt to incorporate both contrastive learning and hashing into model design, utilizing contrastive knowledge for group-wise causal reasoning. The binary opposite of the hash space can generate contrastive samples, and contrastive learning enhances the performance of hash generators. By employing the hash binary opposite value, we develop a twin contrast interventional-cause framework that uses contrastive dual sampling for data augmentation and proposes a new strategy for constructing an intrinsically interpretable UDCMH model. This model conducts interventions during training to create multiple interventional knowledge. Experiments on three baseline datasets demonstrate that the effectiveness of the proposed scheme surpasses that of most UDCMH methods. Given its comprehensive performance and innovation, our model design is well-suited for cross-modal retrieval tasks.