Any-step dynamics model improves future predictions for offline reinforcement learning via hierarchical roll-out.

Abstract

Offline reinforcement learning constrains policy optimization within a given dataset, avoiding the high costs and unpredictable risks associated with online sampling in the real world. Model-based approaches provide a promising direction for offline reinforcement learning, as extensive policy explorations and evaluations can happen within the data-driven dynamics model, transcending the limitations imposed by the dataset boundaries. However, accurately predicting long-horizon sequential steps in the learned dynamics model remains a significant challenge. Model roll-out in previous methods relies on bootstrapping prediction, which attributes the next state to the prediction of the current state, leading to error accumulation over time steps. In this paper, we introduce the Any-step Dynamics Model (ADM) that allows for the use of variable-length plans as inputs for predicting future states. Based on ADM, we propose the Hierarchical Roll-out (HiRo) to mitigate the compounding error by reducing bootstrapping prediction to direct prediction. Theoretical and empirical evidence demonstrate that HiRo has stronger long-horizon prediction capabilities than the widely used step-by-step roll-outs based on the single-step dynamics model. Furthermore, leveraging the inherent diverse predictions of ADM, HiRo can offer a better estimation of model uncertainty compared to the ensemble. Ultimately, offline reinforcement learning with HiRo demonstrates superior performance compared to recent state-of-the-art offline algorithms.