Hit the spot: Reachability guided subgoal generation for hierarchical reinforcement learning in stochastic environments.

Abstract

Hierarchical reinforcement learning (HRL) with subgoals offers an effective approach to tackling tasks characterized by sparse rewards. HRL enables agents to plan across multiple levels where high levels generate subgoals and low levels pursue to reach them. However, execution ability of low level is unstable during training, particularly in stochastic environments, resulting in inaccurate expectations by high level regarding the reachability of subgoals. To address this problem, we propose a reachability guided subgoal generation method based on the representation of the low level's execution ability. Firstly, we theoretically prove that the shortest transition distance between the arrival state and the subgoal follows a normal distribution in both deterministic and stochastic environments, and derive its mean and variance. The high level utilizes the distance model to represent the abilities of the low level, which enhances its capacity to generate subgoals. Therefore, the distance model is incorporated into the value prediction network architecture of the high level, ultimately guiding the policy network to generate well-fitting subgoals for the low level in stochastic environments. Experimental results indicate that our approach achieves higher completion and faster convergence rate in stochastic environments.