Cross-category spatiotemporal consensus and discriminative networks for weakly-supervised temporal action localization.

Abstract

Weakly-supervised temporal action localization is a practical task that localizes different action instances from untrimmed videos without frame-level annotations. Current approaches either enhance the discriminative features of action snippets to reduce confusion with the background, or focus on less informative snippets to guide the model to explore non-salient regions. However, they seldom explicitly consider similar sub-processes across different actions, known as cross-category consensus relationships, which can provide complementary information for exploring the more comprehensive localization results. Moreover, previous methods mostly overlooked class-level higher-order dynamics, which can provide finer-grained motion relationships to help the model capture subtle discriminative features. To alleviate the above problems, we investigate a simple yet effective method termed the STCD network, which leverages superclass-level semantics and high-order dynamics for spatiotemporal consensus and discriminative learning. Specifically, we leverage the high-order encoding module based on Koopman theory to explicitly explore the discriminative class-wise dynamics. Meanwhile, we adopt superclass-level semantics to capture the consensus relationships among various actions due to the similar sub-actions of diverse categories are essential to mine more comprehensive action snippets. Finally, we argue that snippets with high entropy in their category distribution typically demonstrate significant uncertainty and possess ambiguous representations in their feature space. From the perspective of information theory, we further propose an effective loss function to further enhance the discriminative features of each action snippet, i.e., selecting the Top-k categories with the highest predicted probability for each segment and reducing the uncertainty by minimizing their information entropy. Experimental results on three datasets, i.e., THUMOS14, ActivityNet v1.2 and ActivityNet v1.3, demonstrate that our method is superior to the state-of-the-art.