Reliability-aware modality completion with cross-modal distillation for federated learning with missing modalities.

Abstract

Multimodal federated learning (MFL) enables multiple clients to collaboratively train a global model from decentralized data without sharing local privacy-sensitive information. However, practical MFL is challenged by cross-client heterogeneity and missing modalities, which jointly induce client drift, incomplete semantic representations, and degraded global generalization. To address these issues, we propose FCKMD, a robust multimodal federated learning framework for heterogeneous and incomplete-modality settings. Specifically, FCKMD introduces a heterogeneity-adaptive modality expert encoding mechanism, in which a sample-wise router dynamically selects suitable expert paths for different client data and adopts a bypass strategy for missing modalities. To compensate for incomplete observations, FCKMD further employs a cross-modal reconstruction module together with a reliability-aware constraint strategy that adjusts the supervision strength. To further improve prediction robustness, a cross-view consistency transfer scheme is developed to distill discriminative knowledge from the fused multimodal branch into unimodal branches. These components are jointly optimized under a unified objective that integrates classification, reconstruction, and distillation losses. Experimental results on CREMA-D, Crisis-MMD, and UCI-HAR demonstrate that FCKMD outperforms representative baselines and achieves strong robustness and generalization under different missing rates, heterogeneity levels, and client participation ratios.