Logits -with-correlation-based distillation for class incremental learning with limited initial classes.

Abstract

Class Incremental Learning (CIL) has attracted growing attention recently because it learns new class data incrementally while alleviating forgetting of old class knowledge. Knowledge Distillation (KD) is the most widely used strategy to alleviate forgetting, and it is categorized into logit- and feature-based KD by distilled knowledge. However, the performance of KD-based CIL methods, especially for feature-KD, heavily depends on the number of classes for initial training. In practice, it is difficult to collect sufficient classes in advance, especially for disease diagnosis. If there are limited initial classes, the performance of feature-KD-based CIL methods will be greatly reduced. In this work, an innovative Logits-with-Correlation-based Distillation (LCD) method with our newly designed Inter-class Semantic Correlation-based less-forgetting Constraint and Intra-class Consistency Loss is proposed. It provides: i) high accuracy as our LCD can effectively alleviate forgetting and learn new class data; and ii) less sensitivity to initial classes as our LCD operates in logit space but achieves intra-class compactness and inter-class separation in feature space. Experiments on five natural and four medical image datasets with multiple incremental scenarios demonstrate that our LCD achieves superior performance compared to previous state-of-the-art methods, especially when initial classes are limited. Our code is available at https://github.com/dujie-szu/LCD.