Cautious implicit value learning.

Abstract

Implicit Q-learning, a prominent approach within the implicit value learning (IVL) paradigm, mitigates distribution shift in offline reinforcement learning (offline RL) by avoiding out-of-distribution action queries. However, current IVL methods lack stepwise training analysis in value learning and commonly assume the behavior policy to be optimal, both of which overlook more realistic suboptimal scenarios. This work presents cautious implicit value learning (CIVL), a novel offline RL method that redefines the advantage weight to identify a previously unrecognized constraint in IVL training. By establishing a formal connection between the identified constraint and the target V-network, CIVL provides theoretical motivation for introducing the target V-network. In policy learning, CIVL introduces a negative-action penalty (NAP) that penalizes actions with negative advantage values, thereby better handling suboptimal actions. Since the NAP can be shown to stochastically induce an approximate gradient penalty on the Actor, CIVL introduces a floating term that synergistically increases the probability of the gradient penalty occurring. Finally, CIVL further incorporates trajectory-position weighting to provide temporally fine-grained control over the V-network's bias. Benchmark experiments demonstrate that CIVL outperforms existing IVL methods in terms of normalized score and training stability.