Privacy-preserving personalized federated prompt learning for vision-language models.

Abstract

Visual-language models (VLMs) have shown great potential in capturing multimodal information. However, applying prompt learning to VLMs in a federated setting poses two major challenges. Firstly, the non-IID distribution of client data may significantly degrade model performance. Secondly, the transmission of prompts in plaintext may expose sensitive information of users. To address these issues, we propose Privacy-Preserving Personalized Federated Prompt Learning (PPFPL) for VLMs, which introduces a multi-metric personalization weighting algorithm to enhance prompt aggregation, enabling each client to better extract multimodal features while preserving strong generalization. Additionally, it ensures privacy protection against semi-honest servers by distributing sensitive information across two non-colluding entities, and neither of the servers can individually reconstruct the private data. Experimental results show that, under conditions of high heterogeneity, PPFPL improves local task accuracy by up to 9.12 % and achieves an average generalization performance gain of 4.32 % on unseen tasks, compared with standard prompt learning methods.