Exploration and exploitation in continual learning.

Abstract

Continual learning (CL) has received a surge of interest, particularly in parameter isolation approaches, aiming to prevent catastrophic forgetting by assigning a disjoint parameter set to each task. Despite their effectiveness, existing approaches often neglect the task-specific differences, depending on predetermined allocation ratios of parameters. This can lead to suboptimal performance as it disregards the unique requirements of individual task traits. In this paper, we propose a novel Exploration-Exploitation approach to address this issue. Our goal is to adaptively distribute resources between acquiring new information (Exploration) and retaining previously learned knowledge (Exploitation) as new tasks emerge. This allows a continual learner to adaptively allocate parameters for every consecutive task by enabling them to compete for resources using exploration and exploitation. To achieve this, we introduce an allocation learner that learns the intricate interplay between exploration and exploitation across all layers of the continual learner. We demonstrate the proposed method under popular image classification benchmarks for diverse CL scenarios, including domain-shift task-incremental learning. Experimental results show that the proposed method outperforms other competitive continual learning approaches with an average margin of 5.3% across all scenarios.