Multimodal Contrastive Prototype Learning for Resilient Brain Tumor Segmentation With Missing Modalities.

Abstract

Multimodal fusion is an effective solution for holistic brain tumor diagnosis; however, it faces challenges under missing modalities. Traditional multi-encoder architectures can easily capture modality-specific features, while single-encoder architectures readily obtain modality-shared features. The reverse, however, is challenging. In this paper, we propose a two-stage dual-view prototype learning framework to extract the modality-specific feature and the class-specific feature simultaneously. In the first stage, we utilize the Transformer decoder to learn the modality-prototypes that are used to optimize the modality reconstruction task. A masked autoencoder is introduced to generate shared features of incomplete modalities. The learned modality-prototypes that contain modality-specific features are blended with the modality-share features for the reconstruction process. In the second stage, we learn the class-prototypes through the Transformer decoder to generate a segmentation mask through voxel-to-prototype comparison. A masked modality strategy is introduced to handle random modality absence during training. Furthermore, modality-view and class-view contrastive learning strategies are developed to enhance prototype learning. We conduct experiments on BraTS2020 and BraTS2018; the experimental results demonstrate the superior performance of our model under various missing modality scenarios. On BraTS2020, our model achieves DSC improvements of 5.9% for ET, 0.5% for TC, and 0.2% for WT compared to state-of-the-art methods. Notably, in the challenging T1C modality missing scenario, our model achieved clinically significant gains of 9.5% for ET and 1.8% for TC. The code is available at https://github.com/Xiheran/MCPL.