CLASH-CTTA: Class-Wise Shift-Aware Hierarchical Continual Test-Time Adaptation.

Abstract

The domain shift between training and test data has emerged as a critical obstacle to the generalization of deep models. Continual Test-Time Adaptation (CTTA), which aims to leverage online test data stream to adapt to continuously evolving target distributions, has become a promising and practical solution to address real-world domain shift issues. Existing CTTA methods primarily rely on self-training frameworks based on output consistency and entropy-based loss functions, which often consider either the entire batch or reliable samples within the batch in isolation, failing to combine both of the advantages simultaneously. Inspired by fast and slow learning strategies in continual learning, we propose a fully source-free approach, dubbed CLAss-wise Shift-aware Hierarchical Continual Test-Time Adaptation (CLASH-CTTA). The adaptation process of CLASH-CTTA is carried out with a hierarchical updating strategy, where the test data stream promotes the slow learning of general representations, while the representative samples facilitate fast learning of domain-specific knowledge. The observation reveals that different classes exhibit varying sensitivities to different domains, leading to diverse discriminability across classes under domain shifts. Thus, a Class-wise Shift-aware Representative Set is maintained to provide representatives and mitigate the discrepancy. In addition, using Spearman's rank correlation as a novel perspective to examine the correctness of samples, we further filter representatives and align them to the class prototypes. Extensive experiments on three corruption domain shift datasets and one natural domain shift dataset demonstrate the superiority of our method compared with state-of-the-art methods, including in continual tasks, gradual tasks, and scenarios with diverse batch sizes.