MT-SAM: A Mamba-Transformer Enhanced SAM with Prior-guided Prompting for Multi-modal Prostate Cancer Delineation.

Abstract

Clinically, bi-parametric MRI (bp-MRI), including T2-weighted imaging, diffusion-weighted imaging, and apparent diffusion coefficient map, offers essential prior localization of biopsy and focal therapy for suspicious clinically significant prostate cancer (csPCa), and accurate csPCa delineation from bp-MRI is crucial for better outcomes. However, due to the complexity and high variability in appearance, size, shape, and indistinct boundaries, delineating csPCa remains challenging, time-consuming, and heavily relies on the clinician's experience. To address these issues, we propose MT-SAM, a novel framework that enhances SAM with higher-quality feature extraction and a prior-guided automatic prompting strategy. Specifically, we introduce a mamba-transformer network to extract multi-stage multi-modal features from bp-MRI and fuse them into the SAM encoder via cross-mamba modules. Moreover, we propose a prior-guided pyramid-mamba prompting strategy to strengthen the model's attention on the targets. We extensively evaluate our method on both public and private datasets, and the experimental results show that our method achieves up to 5.6-34.1% higher Dice scores than state-of-the-art methods. Code is available at https://github.com/LuckLT/MT-SAM.