CAPTR-GTP: Class-aware prompting and token refinement with graph token propagation for few-shot ViTs.

Abstract

Vision Transformers (ViTs) play a central role in modeling long-range dependencies. However, in few-shot learning, they face persistent obstacles. Dense self-attention amplifies background interactions in cluttered scenes. Reliance on a final-layer [CLS] token or static prototypes can miss fine-grained cues. Uniform patch pooling ignores token reliability under scarce labels. Many ViT-based adapters rely on token-similarity matching or merging, which often discard or overcompress tokens and cause information loss. To address these issues, we propose CAPTR-GTP, Class-Aware Prompting and Token Refinement with Graph Token Propagation, a ViT-based framework for episodic few-shot learning that preserves and propagates token information rather than discarding it. A ViT encoder is pretrained with masked image modeling and then kept mostly frozen to provide stable patch embeddings. An uncertainty-aware module uses Monte Carlo dropout to estimate token reliability, retaining high-confidence tokens and summarizing noisy patches with a variance-consistent gate. Class-Aware Prompt Refinement maps per-class prototypes and dispersion from filtered supports into learnable prompts that align attention with class semantics and adapt to intra-class variation. A bi-level hierarchical attention module refines tokens within clusters and aggregates context across clusters, combining local structure with global information for support-conditioned prediction. On a sparse top-k token graph with per-episode fixed neighbors, Graph Token Propagation refreshes edge weights, propagates features and labels, mitigates over-smoothing, and preserves rare cues. We evaluate CAPTR-GTP on four in-domain benchmarks and four cross-domain transfer benchmarks. Extensive experiments under the 5-way 1-shot and 5-shot protocols demonstrate that CAPTR-GTP achieves state-of-the-art or consistently competitive performance against strong baselines.