Online spatio-temporal forecasting with retrieval-augmented orthogonal landmark transformer and masked teacher-student learning.

Abstract

Online spatio-temporal forecasting is widely applied across multiple domains, where streaming data under limited computational resources pose practical challenges. Models must efficiently capture long-context spatio-temporal dependencies while balancing fast adaptation to new patterns with stable retention of essential historical knowledge under distribution shifts. To address these challenges, we propose ROLT-MTSOL (Retrieval-Augmented Orthogonal Landmark Transformer with Masked Teacher-Student Online Learning), a scalable framework for online spatio-temporal forecasting. At the modeling level, ROLT uses retrieval augmentation to extend the key-value context with relevant historical segments, forming a long-context representation that enhances dynamic spatio-temporal modeling. Learnable Orthogonal Landmark Attention (LOLA) then compresses it into shared landmarks using Newton-Schulz orthogonalization, enabling efficient and stable attention. At the learning level, MTSOL provides a Masked Teacher-Student Online Learning framework that promotes fast adaptation to new patterns while retaining essential historical knowledge and mitigating forgetting. Experiments on diverse spatio-temporal benchmarks demonstrate that ROLT-MTSOL outperforms state-of-the-art baselines in accuracy while enabling highly efficient online inference and learning. Code is available at this repository: https://github.com/zhenzhen618/ROLT-MTSOL-main.