Zero-shot sketch-based remote sensing image retrieval based on cross-modal fusion.

Abstract

It is efficient for users to retrieve remote images by hand-drawn sketches when there is no example at hand but images in mind. So we address the challenges inherent in zero-shot sketch-based image retrieval (ZS-SBIR) for remote sensing data, i.e. zero-shot sketch-based remote sensing image retrieval (ZS-SBRSIR). The current progress in this area is slow. We believe there are two reasons: (1) remote sensing sketch data is scarce and difficult to obtain, consistent with the reality simulated by the task and (2) the domain gap between sketches and remote sensing images makes it difficult to find effective common cross-domain representations. To address this gap, our study introduces a novel zero-shot cross-modal fusion network for remote sensing images, leveraging extra readily available multi-modal information to bridge the gap and fuse the different modalities. On the one hand, we extract readily available edge feature information from remote sensing images. The edge image is a modality closer to sketches, and we use them as a bridge to fuse the two modalities. The edge image can assist with scarce sketches and enables the model to have better zero-shot generalization ability. On the other hand, we use existing image labels as simple semantic information and perform the same contrastive training as images. From comprehensive experiments, we verify the efficacy of the proposed model on ZS-SBRSIR.