Result constraint behavior cloning for offline reinforcement learning.

Abstract

Offline reinforcement learning trains policies using pre-collected data without interacting with the environment, leading to extrapolation errors and the out-of-distribution (OOD) problem. While existing methods focus on avoiding OOD actions through policy constraints or value pessimism, they overlook the impact of OOD states on performance. To address both OOD actions and OOD states, we propose the Result Constraint Behavior Cloning (RCBC), which can constrain the result states within the dataset. We use the dynamics model and nearest neighbor search to realize the result constraint. In the implementation, we incorporate the result constraint into the optimization process for both actor and critic, to motivate actor to select actions that yield states within or near the dataset, and to motivate critic to rely more on these actions. The analysis experiments verify that RCBC can achieve adaptive adjustment of behavior cloning and more accurate value update. The evaluation experiments on the D4RL benchmark show that our algorithm achieves state-of-the-art performance.