Multiple semantic prompt for rehearsal-free continual learning.

Abstract

Deep neural networks have achieved impressive achievements in many applications while suffering from catastrophic forgetting, which refers to drastic degradation in performance on former tasks when training on novel ones. Continual learning aims to address the issue. A simple but effective solution is to replay a subset of previous data, while it increases memory costs and may violate data privacy. Recently, prompt-based methods have seen a recent explosion of interest because they can achieve good performances without the rehearsal buffer. In this work, we link prompts and textual labels to learn semantic information and combines prompts according to the similarity between visual and textual features. So we call it the Multiple Semantic Prompt (MSP). The combined prompt captures semantic information from learned tasks and can instruct the pre-trained model to perform better in zero-shot transfer. Extensive experiments show the superiority of our method over SOTA prompt-based methods by over 3 % in average final accuracy. Our method also has higher zero-shot classification accuracy than the vanilla vision language model by over 15 % on the established benchmark. Our code is available at https://github.com/EthenWillson/Multiple-Semantic-Prompt-for-CL.