Prompt-Enhanced: Leveraging language representation for prompt continual learning.

Abstract

Continual learning enables models to learn from an evolving stream of data without forgetting the previously acquired skills. Traditional methods often rely on the rehearsal buffers or extended network structures to retain the previous knowledge, which incurs additional memory overhead and potential privacy concerns. Prompt-based methods have emerged as a promising alternative with the advent of large-scale pre-trained vision transformers. These methods effectively preserve the prior knowledge by retrieving the task-specific prompts. However, directly applying prompts for downstream tasks may result in a generalized information loss, which leads to the diminished performance in long-term continual learning. To address this limitation, we propose Prompt-E, a lightweight, plug-and-play method that enhances prompt-based continual learning by incorporating language-guided regularization. Prompt-E utilizes language representation to enhance the effectivity of prompt features dynamically, thus ensuring the task-specific relevance and stability across continual tasks. Our method aims to handle the challenges of prompt conflict and catastrophic forgetting by constraining both the CLS token and the prompts within a continual learning framework. Sufficient experiments on three challenging benchmarks demonstrate the advantages of Prompt-E over representative prompt-based methods, including significant accuracy gains in class-incremental tasks. Notably, Prompt-E improves the accuracy performance of L2P on DomainNet by 3.29% with only additional 1.89M parameters. Our ablation studies further validate the effectiveness of language-guided prompt regularization.