Dynamic-based representation inconsistency and implicit constraints for offline reinforcement learning.

Abstract

Offline reinforcement learning (RL) faces significant challenges due to distributional shift and extrapolation errors when learning from static datasets. While existing methods employ pessimism to avoid out-of-distribution (OOD) actions, overly strict penalties often hinder policy improvement by neglecting potentially high-reward OOD explorations. To address this, we propose Dynamic-based Representation Inconsistency and Implicit Policy Constraints Reinforcement Learning (DRIPC), an algorithm that leverages a novel uncertainty quantification mechanism to balance OOD exploration and exploitation. First, we learn dynamic representations via ensemble models, utilizing their inconsistency as an uncertainty quantifier to perform pessimistic value iteration. Second, we reformulate policy constraints in Q-function space, enabling reward-aware optimization while aligning state-action distributions. The resulting framework integrates ensemble models with implicit constraints, achieving state-of-the-art performance on the D4RL benchmark with 67.4% higher average returns than Conservative Q Learning (CQL) on Antmaze tasks. Notably, DRIPC reduces computational overhead by 30.8% compared to prior ensemble-based methods while maintaining robust uncertainty characterization. Our code is available at https://github.com/dksen/DRIPC.