ProPy : Building interactive and efficient prompt pyramids upon CLIP for partially relevant video retrieval.

Abstract

Partially Relevant Video Retrieval (PRVR) is a practical yet challenging task that involves retrieving videos based on queries relevant to only specific segments. While existing works follow the paradigm of developing models to process unimodal features, powerful pretrained vision-language models like CLIP remain underexplored in this field. To bridge this gap, we propose ProPy, a model with systematic architectural adaption of CLIP specifically designed for PRVR. Drawing insights from the semantic relevance of multi-granularity events, ProPy introduces three key innovations: (1) A Prompt Pyramid framework that organizes event prompts in a pyramid structure to capture semantics at multiple granularity levels, (2) A Layer-then-Segment two-stage event sampling strategy to reduce memory usage while preserving the representation capability and (3) An Ancestor-Descendant Interaction Mechanism built on the pyramid that enables dynamic semantic interaction among events. With these designs, ProPy achieves SOTA performance on three public datasets, outperforming previous models by significant margins. Code is available at https://github.com/BUAAPY/ProPy.