TDP-DETR: Temporal dynamics perception framework for video moment retrieval and highlight detection.

Abstract

Video Moment Retrieval (VMR) and Highlight Detection (HD) aim to localize query-relevant temporal segments and evaluate clip-level saliency within untrimmed videos. Accurate temporal boundary perception is essential for VMR and HD. While current models have made significant progress, they still struggle to achieve precise action semantic alignment with temporally dynamic video content and are prone to boundary perception bias when action-related visual semantic cues experience fluctuations in specific frames. In this paper, we propose a Temporal Dynamics Perception DEtection TRansformer (TDP-DETR) that models action temporal dynamics from two complementary perspectives: temporal persistence and temporal progression. For temporal persistence, we introduce a dynamic masking strategy for action duration-aware temporal modeling, enabling the model to infer action persistence from query semantics and incorporate it as a temporal prior for boundary prediction. For temporal progression, we design an action state difference perception module that captures frame-to-frame action state variations, allowing the model to perceive action progression speed and thereby improve anticipation of action boundaries. Extensive experiments on three MR/HD benchmarks demonstrate that our method consistently outperforms existing state-of-the-art approaches. Our code will be public soon.