Negative prompt-guided optimization: Enhancing soft prompt generalization in vision-language models.

Abstract

Prompt learning has recently emerged as a pivotal approach for effectively adapting vision & language models to a wide range of downstream tasks. However, existing approaches are prone to significant overfitting on the training data, leading to substantial performance degradation when evaluated on unseen classes within the same domain. Our empirical analysis demonstrates that existing methods fail to effectively align 'negative prompts' for unseen classes, further substantiating the overfitting problem. To address this issue, we propose a novel approach, termed Negative Prompt-Guided Optimization (NPGO), which is based on adversarial training by using prompts containing negative text. To mitigate overfitting and improve the representation capacity of prompt, we use negative adversarial loss. The proposed loss aims to enable the logits between the commonly used prompt, "aphotoofthe{class}" and its corresponding negative text to infer a uniform probability distribution. Our proposed approach significantly alleviates the misalignment in existing methods and shows remarkable improvements in generalization performance for unseen classes. Through extensive experiments conducted on 11 datasets, we demonstrate that the proposed method achieves superior representation learning and inference stability compared to prior soft prompting approaches.