Learning Stable Subgoal Representation in Hierarchical Reinforcement Learning With Causal Factorization.

Abstract

Goal-conditioned hierarchical reinforcement learning (GCHRL) has proven effective for complex control tasks, particularly in long-horizon and sparse-reward settings. By employing a multilevel policy hierarchy, the high-level policy generates feasible subgoals for the low-level policy, which in turn learns to achieve them. This structure enhances exploration and substantially improves learning efficiency. However, the subgoal space has a profound influence on policy training. A well-constructed subgoal space is essential for efficient learning, while a poorly learned one hinders it. Causal reinforcement learning (CRL), which integrates causal learning into reinforcement learning, enhances decision-making through discovery and utilization of causal relations. In this article, inspired by current CRL methods, we propose a novel causal approach to learning subgoal spaces. Leveraging causal relations among different parts of the state space, we extract components containing more high-level and decision-relevant information. Besides, to enhance the quality of representation learning, we propose a regularization method to alleviate representation collapse and improve stability. We evaluate our approach on a series of long-horizon and sparse-reward tasks. The experimental results show that our method learns higher quality subgoal representations and outperforms current methods.