Spatiotemporal Context-Aware Prompting With Low-Rank Dynamic Routing for Exemplar-Free Video Class-Incremental Learning.

Abstract

Video class-incremental learning (VCIL) aims to progressively recognize novel action categories while preserving spatial-temporal knowledge of previous tasks. Unlike image class-incremental learning (CIL), VCIL requires simultaneously capturing spatial semantics and temporal dynamics, which makes catastrophic forgetting more significant. Prompt-based learning has recently made remarkable advancements in VCIL. However, they primarily optimize a predefined static prompt for all action sequences in a group-level manner, which overlooks the diverse spatial-temporal characteristics across frames and exhibits limited generalization capability for future action categories. To address the above limitations, we propose a novel exemplar-free VCIL framework termed STCP-low-rank dynamic routing (LRDR) that consists of two crucial innovations, i.e., the spatiotemporal context-aware prompting (SCAP) and LRDR. The former dynamically generates instance-level prompts based on the input video. Specifically, the frame-level prompt is developed to adaptively emphasize fine-grained details in each frame by leveraging the attention-guided spatial activation module. Meanwhile, we also design the cross-frame prompt to capture the differential importance of sequences, allowing the model to focus on key frames and facilitate its ability to explore temporal dependencies. Furthermore, we introduce a parameter-efficient LRDR that achieves dynamic scalability by incorporating mixture-of-experts low-rank spatial-temporal adapters, which can maintain old prompt knowledge and cross-task collaboration as the trainable SCAP also suffers from catastrophic forgetting when learning from incremental tasks. Finally, we propose a novel prompt correction mechanism (PCM) that prevents the proposed SCAP from acquiring ineffective class-wise spatiotemporal representations via the discriminative class-level prompt knowledge. Extensive experiments are conducted on four video benchmarks, and our approach consistently achieved substantial gains over state-of-the-art methods in VCIL.