PSL: Prototype swapping learning for modality imbalance.

Abstract

Multimodal learning (MML) utilizes the complementary information of different modalities to better handle downstream tasks compared to the uni-modal scenario. However, existing MML methods use all modalities incompatibly: Learning heavily from one dominant modality, while ignoring the others. This problem is called modality imbalance, which results in deteriorating MML performance. The existing solutions are to slow down the learning of dominant modality or speed up that of slow-learning modality. Such approaches usually involve explicitly estimating an imbalance rate to distinguish between dominant and slow-learning modalities. In this paper, we introduce a novel Prototype Swapping Learning (PSL) framework, which addresses modality imbalance from a mutual learning perspective. By simply swapping prototypes of two modalities, PSL uses the prototype of dominant modality to promote the slow-learning one, while applies the prototype of slow-learning modality to slow down the dominant one. Note that our PSL does not require any imbalance rate calculation for modality distinction, providing a self-adaptive remedy to the modality imbalance problem. In addition, our method can be seamlessly integrated with existing MML methods to further improve their performance. The experiments on multiple benchmark datasets demonstrate the effectiveness and superiority of our PSL, compared to the state-of-the-art MML approaches. The code is publicly available at https://github.com/zanchenyi/PSL.