Similarity-based prototype reconstruction and feature reorganization for non-exemplar class incremental learning.

Abstract

Although deep learning (DL) has achieved great success in many computer vision tasks, it still suffers from catastrophic forgetting when data comes in incremental form. Non-exemplar class incremental learning (NECIL) can mitigate catastrophic forgetting without storing the original samples of old data. In NECIL, the mean representations of classes, i.e. prototypes, are usually memorized to replace original samples. Therefore, how to utilize the memorized prototypes to alleviate the catastrophic forgetting of model has become a key issue in NECIL. In this paper, we propose a novel similarity-based prototype reconstruction and feature reorganization (SPRR) method for NECIL. In this method, a feature reorganization mechanism is designed, which continuously updates the prototypes to adapt to the changes of mapping from the original data space to the feature space caused by the continuous model updating in the incremental learning process. In order to maintain the decision boundary of previous tasks, we also propose a similarity-based prototype reconstruction, by which appropriate features of new data are selected to reconstruct the features of old data based on the similarity between the features of new data and the updated prototypes. To enhance the stability of the model, we further introduce a simple yet effective knowledge integration strategy for classifier training so as to make the classifier match the previous feature space. Finally, the performance of our method is verified for NECIL on three benchmark datasets, including CIFAR-100, TinyImageNet and ImageNet-Sub. Experimental results show the effectiveness of the proposed method in NECIL.