CALM: Fine-grained federated unlearning via CLIP-assisted LoRA-MoE memory editing.

Abstract

Federated learning (FL) enables decentralized clients to collaboratively train a global model without sharing raw data, inherently improving data confidentiality during training. With regulatory frameworks such as the General Data Protection Regulation (GDPR) mandating the right to be forgotten, Federated Unlearning (FU) has emerged as a key technique for removing data-related influence from trained models upon request. Existing FU methods, however, predominantly rely on costly full-model retraining or coarse parameter rollback, often leading to substantial overhead, utility degradation, and incomplete forgetting, especially under heterogeneous client distributions. To address these challenges, we propose CLIP-Assisted LoRA-MoE Memory Editing (CALM), a fine-grained FU framework that integrates Contrastive Language-Image Pretraining (CLIP), Low-Rank Adaptation (LoRA), and Mixture-of-Experts (MoE). Specifically, CALM injects modular LoRA-MoE adapters into a frozen global model, enabling targeted knowledge isolation and editing without full-model retraining. In addition, CLIP-based cross-modal regularization mitigates semantic drift during unlearning by aligning model representations with high-level semantic supervision. Experiments on CIFAR10, CIFAR100, and Caltech101 show that CALM achieves effective forgetting while keeping retained accuracy within 2-5 percentage points of retraining across the main unlearning settings; on CIFAR10-ResNet18 client-level unlearning, it obtains 66% retained accuracy (4 percentage points below retraining) and 3% forgetting accuracy, while reducing communication cost by 90.8% compared with retraining.