Towards class-incremental open-set scenarios for video-based action recognition.

Abstract

Recent advances in action recognition have achieved strong results in both class-incremental and open-set learning settings. However, Class-Incremental Open-Set Action Recognition (COAR) remains challenging because the model must simultaneously preserve previously learned action knowledge, learn newly arriving action categories, and reject unseen actions that do not belong to any known class. In this work, we address COAR by developing an incremental recognizer that classifies closed-set samples while effectively distinguishing open-set samples at each stage. We identify two major limitations of existing solutions: training only with closed-set data weakens the separation between known and unknown action representations, and insufficient transfer of stage-wise discriminative ability aggravates forgetting. To overcome these issues, we propose a Pearson-distance-based framework consisting of discriminability enhancement and logit alignment. The discriminability enhancement module constrains the Pearson distances between closed-set action representations and their class centroids, thereby improving inter-class separability and reducing open-space risk. The logit alignment module transfers batch-level and class-level relationships between adjacent stages, preserving recognition and rejection capability during incremental learning. Comprehensive experiments on UCF101, HMDB51, and Something-Something V2 demonstrate the effectiveness of the proposed method. For example, under the five-stage COAR setting using UCF101 as known classes and HMDB51 as unknown classes, the proposed method achieves 73.26% ACC, 82.40% AUROC, and 64.52% AUPR, outperforming competitive baselines under the same protocol.