Multimodal learning rebalanced: Negative correlation ensembles for improved performance.

Abstract

Multimodal learning aims to integrate information from different modalities to overcome the limitations of single-modal information. Recent research has shown that multimodal learning methods often focus on optimizing a dominant modality, leading to incomplete model performance development, and sometimes even inferior to single-modal models, a phenomenon referred to as the modality imbalance problem. To overcome this issue, some studies adaptively adjust the gradients or loss functions based on the design of identifying the dominant modality. However, while enhancing the convergence capability of the non-dominant modalities, they often result in a decreased ability to utilize the information from the dominant modality. Therefore, we treat each modality in our model as a basic classifier and address the modality imbalance problem from the perspective of ensemble learning, thus fully leveraging the information from each modality. In addition, we introduce the concept of negative correlation learning to ensure the diversity of information encoding across different modalities. Through experiments carried out across multiple datasets, using various late fusion techniques, and across a variety of tasks, we validated the superior performance of the proposed method, as evidenced by significant improvements in accuracy compared to existing approaches.