Multi-modal medical image synthesis via dual-branch wavelet encoding and deformable feature interaction.

Abstract

Multi-modal medical imaging is essential in disease diagnosis and treatment planning, as it provides complementary anatomical and pathological information. However, the limitations of scan time, image corruption, and differences in imaging protocols often result in certain modalities being missing or unavailable, severely limiting the application of multi-modal data. Although existing medical image synthesis methods have made notable progress, it remains challenging to synergistically extract local details and global contextual information from multi-modal inputs, as well as to effectively fuse complementary features across modalities. To address these problems, this paper proposes a novel dual-branch wavelet encoding and deformable feature interaction generative adversarial network (DWFI-GAN), for synthesizing missing modalities from available ones. Specifically, in the generator, we design a dual-branch wavelet encoder and introduce a wavelet multi-scale downsampling (Wavelet-MS-Down) module, in order to efficiently extract multi-scale local features and global contextual information at the downsampling stage. To better fuse information across modalities, we design the deformable cross-attention feature fusion (DCFF) module to interactively fuse features from different source images at multiple scales. Additionally, at the bottleneck of the generator, we design an episodic bottleneck structure. This structure employs intermittent injection of frequency-space enhanced (FSE) modules, which enriches the scale diversity of the fusion features from both frequency and spatial domain perspectives. Experiments on two medical imaging datasets show that DWFI-GAN outperforms several state-of-the-art methods in both qualitative and quantitative comparisons, with ablation studies further validating each module's contribution to fusion and detail enhancement. The code is publicly available at https://github.com/xuefengjia227/DWFI-GAN.