Causal gradient intervention for debiased and evidence-grounded medical visual question answering.

Abstract

Medical Visual Question Answering (Med-VQA) aims to answer clinically relevant questions based on medical images. However, existing methods often struggle to provide visual evidence that is consistent with the query and verifiable. Under scarce evidence-level supervision and biases induced by imbalanced data distributions, models tend to rely on shortcut signals from language priors and visual priors, replacing critical visual evidence with prior-driven cues and thereby distorting both answers and evidence. To address this issue, we propose DE-CaGI, a causal gradient intervention framework that achieves debiased learning and evidence grounding at the optimization level. First, DE-CaGI constructs auxiliary branches to characterize shortcut learning, explicitly estimates bias gradients driven by language and visual priors, and suppresses shortcut-related gradient components when updating the backbone representation module, thus weakening the pull of shortcut directions on model updates. Then, building on the debiased updates, DE-CaGI introduces visual evidence gradients induced by multitask evidence supervision and imposes evidence-consistency constraints on the backbone update direction, so that the model can move toward representations consistent with annotated evidence while suppressing shortcut effects. Experiments on VQA-RAD and SLAKE demonstrate stable improvements on both open-ended and closed-ended questions and yield competitive overall accuracy. Further analyses show that DE-CaGI more effectively reduces reliance on language and visual priors, while qualitative results indicate better evidence alignment. The source code is publicly available at https://github.com/cloneiq/DE-CaGI.