Branch-Tuning: Balancing Stability and Plasticity for Continual Self-Supervised Learning.

Abstract

The self-supervised learning (SSL) has emerged as an effective paradigm for deriving general representations from vast amounts of unlabeled data. However, as real-world applications continually integrate new content, the high computational and resource demands of SSL necessitate continual learning (CL) rather than complete retraining. This poses a challenge in balancing between stability and plasticity when adapting to new information. In this article, we employ centered kernel alignment (CKA) for quantitatively analyzing model stability and plasticity, revealing the critical roles of batch normalization (BN) layers for stability and convolutional layers for plasticity. Motivated by this, we propose branch-tuning (BT), an efficient and straightforward method that achieves a balance between stability and plasticity in continual SSL. BT consists of branch expansion and compression and can be easily applied to various SSL methods without the need of modifying the original methods, retaining old data or models. We validate our method through experiments on various benchmark datasets, demonstrating its effectiveness and practical value in real-world scenarios. We hope our work offers new insights for future continual SSL research. The code will be made publicly available.