MSB-VQA: Overcoming multiple source biases for robust visual question answering.

Abstract

Recent studies have found that many VQA models are influenced by biases and cannot effectively utilize multimodal information for reasoning. Models that perform well on standard VQA datasets perform poorly on bias-sensitive VQA-CP datasets. Although there have been many studies focusing on mitigating biases in VQA models, most of them only consider language bias and fail to achieve satisfactory results. To address this issue, we propose a novel method that targets various sources of bias. Specifically, to eliminate multimodal shortcut biases, we design a bias detector, which can be trained by generative adversarial networks and knowledge distillation to use unimodal information to imagine another modality, effectively simulating the process of bias formation in humans. To combat distributional bias, we use a cosine classifier to obtain a cosine feature branch from the base model. We then use adaptive angular margin loss and supervised contrastive loss to address bias caused by uneven sample distributions in terms of frequency, difficulty, and answer. During the prediction phase, we fuse the predictions of the cosine classifier with those of the base model, balancing the model's performance on ID and OOD datasets. Finally, we conduct extensive experiments on the VQA-CPv2, VQAv2, and VQA-CE datasets, demonstrating that our MSB-VQA method outperforms other methods in bias reduction significantly, without using any data balancing and augmentation.