TMMSAM2: Tracker-Aided Multitemporal Memory SAM2 for Hyperspectral Object Tracking.

Abstract

Segment Anything Model 2 (SAM2) demonstrates outstanding performance in prompt-based visual segmentation. However, directly applying it to hyperspectral object-tracking tasks still faces numerous challenges. This article proposes TMMSAM2, a novel SAM2-based framework for hyperspectral object tracking that requires no additional training, comprising a multitemporal memory bank (MMB) and a three-dimensional constraint error mechanism (TCEM). Specifically, the MMB overcomes limitations of traditional strategies based on fixed historical frames by collaboratively integrating memory information across short-, medium-, and long-term temporal scales. Combined with feature similarity calculations and redundant frame filtering mechanisms, it constructs historical memory with optimal spatiotemporal diversity. The TCEM implements real-time quality assessment of tracking results by modeling physical constraints in three dimensions: velocity, size, and motion direction. When anomalies are detected, the system automatically triggers correction using a ViPT-based hyperspectral tracker, thereby establishing an end-to-end tracking quality assurance system. Extensive experiments on three public datasets demonstrate that our method exhibits superior performance in complex scenarios involving rapid target movement and frequent occlusions.