Bellman error centering.

Abstract

This paper re-examines recent reward centering methods in reinforcement learning (RL), focusing on Simple Reward Centering (SRC) and Value-based Reward Centering (VRC). Our analysis distinguishes SRC as direct reward centering, while proving VRC's mathematical equivalence to Bellman Error Centering (BEC)-a novel reinterpretation that clarifies its underlying mechanism. Leveraging BEC, we derive two theoretical advances: (1) a centered fixed-point solution for tabular value functions, and (2) a centered TD fixed-point solution under linear function approximation. We propose two algorithms to realize these: on-policy Centered Temporal Difference (CTD) and off-policy Centered Temporal Difference with Correction (CTDC), with convergence proofs under standard assumptions. Experiments on benchmark environments show our methods outperform baselines in stability and performance. The BEC paradigm facilitates seamless integration with existing RL architectures, enabling broad applicability.