Spatial-Temporal Scene Graph Generation for Open-Vocabulary Multiple Object Tracking.

Abstract

Open-vocabulary multiple object tracking (MOT) aims to track arbitrary objects in the real world. Although significant progress has been achieved in object classification by leveraging the knowledge from large vision-language models, advances in data association for open-vocabulary MOT remain limited. Existing methods primarily rely on appearance cues to establish associations. However, these cues are often unreliable in the face of occlusions and ambiguous object appearances, resulting in suboptimal tracking performance in complex scenarios. In this paper, we propose a novel open-vocabulary MOT method, Spatial-temporal Scene Graph Tracker (SSGTrack), which introduces a fundamentally different approach to data association by building a Spatial-temporal Scene Graph (SSG) that captures rich semantic and spatial relationships between objects across adjacent frames. Specifically, SSGTrack constructs proposal-level relationships by extracting diverse contextual information from the multi-head self-attention layers of the Transformer decoder. These relationships, derived from the keyframe and reference frame, are compressed into the compact SSG, where nodes represent detected objects, and edge weights denote frame-level connectivity. Furthermore, to address the challenge of differentiating visually similar objects and background distractors, we propose a Context-aware Contrastive Learning (CCL) strategy. By identifying background features that significantly differ from positive samples and incorporating them as negative samples, CCL enhances the ability of the model to learn discriminative representations, thus improving tracking robustness. Extensive experiments conducted on several challenging MOT benchmarks demonstrate the effectiveness of our method, which achieves superior tracking performance.