The Most Overestimated Q Value Regularization in High-Dimensional Discrete Action Spaces for Offline Reinforcement Learning.

Abstract

Deep reinforcement learning (DRL) excels at learning control policies in high-dimensional action spaces, making it crucial for robotic manipulation. However, its real-world application is limited by costly and risky data collection. Offline reinforcement learning (offline RL) addresses this issue by training on precollected datasets but struggles with $Q$ value overestimation in high-dimensional discrete action spaces, where the number of out-of-distribution (OOD) actions rapidly increases, negatively impacting training stability. In this work, we propose the most overestimated $Q$ value regularization (MQR), a novel offline RL algorithm that penalizes the action with the most overestimated $Q$ value, effectively mitigating overestimation in high-dimensional discrete action spaces. By regulating the action most affected by $Q$ value overestimation, rather than applying uniform penalties across the entire action space as in existing methods, MQR further prevents the policy from converging incorrectly. We evaluate MQR on a robotic pushing and grasping task, a challenging high-dimensional discrete action space problem, in both simulated and real-world environments with random, dense, and unknown object arrangements. The results demonstrate that MQR significantly outperforms baseline algorithms, achieving a clearance rate of 96.94% in simulations and 99.04% in real-world dense configurations, while maintaining high action efficiency and stability. These findings highlight MQR's robustness, scalability, and adaptability for robotic manipulation, showcasing its potential for real-world deployment in industrial robotics. The code used in our research is publicly available at https://github.com/Hanyang-Robot/MQR.