MG-3D: Multi-grained knowledge-enhanced vision-language pre-training for 3D medical image analysis.

Abstract

3D medical image analysis is pivotal in diagnostic, prognostic, and therapeutic applications. However, the development of current AI methods for 3D medical image analysis is hindered by limited generalization capabilities across diverse clinical scenarios and high costs associated with task-specific data annotations, such as segmentation masks and classification labels. Foundation models (FMs) built via large-scale pre-training are expected to advance extensive clinical tasks without extensive task-specific annotations. In clinical practice, 3D medical images are often accompanied by radiology reports, which can serve as supervision signals providing rich semantics for building FMs. However, these narrative descriptions remain underutilized in pre-training. Specifically, the insufficient investigation into multi-grained radiology semantics and their correlations across patients has left the potential of large-scale 3D volume-report data untapped. To overcome these limitations, we propose MG-3D, a vision-language pre-training (VLP) method based on large-scale 3D Computed Tomography (CT) data (47.1K), addressing these challenges by two aspects: 1) Establishing robust intra-patient correspondences between 3D volume semantics and multi-grained report knowledge for each patient. This is achieved via cross-modal global alignment and complementary modality-guided local reconstruction, promoting multi-grained visual semantic understanding; 2) Correlating inter-patient visual semantics based on fine-grained report correlations across patients, and maintaining sensitivity to global individual differences via contrastive learning. Furthermore, we delve into the scaling law to investigate potential performance improvements. Comprehensive evaluations across ten clinical tasks are conducted to assess model efficacy in both internal and external datasets. MG-3D exhibits superior transferability, scalability, and generalization in 3D medical image analysis tasks, including those involving 3D CT and Magnetic Resonance Imaging (MRI). Codes and pre-trained models are available: https://github.com/Xuefeng-Ni/MG-3D.