When Federated Prompt Learning Fails: Understanding and Mitigating Cross-Domain Prompt Interference.

Abstract

Federated prompt learning (FPL) offers a parameter-efficient way to adapt large-scale vision-language models by collaboratively optimizing lightweight prompts across distributed clients. Existing approaches typically assume that prompts learned on heterogeneous domains can be safely aggregated. We show that this assumption breaks down in cross-domain federated settings with domain and model heterogeneity: naive global averaging induces cross-domain prompt interference and can even underperform simple intra-domain collaboration. Through geometric and optimization analyses, we find that prompts from different domains are semantically separated and exhibit low gradient alignment, which explains their poor compatibility under global aggregation. Motivated by these observations, we propose DaFPL, a domain-aware FPL framework that decomposes each client prompt into two complementary parts: a domain-transferable component that captures domain-invariant semantics and a domain-specific component that preserves domain-dependent characteristics. We introduce a domain discriminator to enforce the intended functional roles of the two components. In addition, we design a domain-aware aggregation strategy that aggregates domain-transferable prompts globally through an entropy-based invariance gate, while aggregating domain-specific prompts only within each domain. We further provide a non-convex convergence analysis of DaFPL. Extensive experiments on four multi-domain benchmarks demonstrate that DaFPL consistently outperforms strong FPL baselines.