Dilated memory in hierarchical reinforcement learning for long-horizontal task.

Abstract

Tasks in the real world are almost always partially observable, which does not satisfy the Markov property required by the reinforcement learning paradigm. Recent approaches attempt to approximate full state representations using temporal mechanisms such as simple concatenation, RNNs, or Transformers. However, these methods are limited to retaining only adjacent and short historical sequences. For long-horizon tasks, relying solely on adjacent memory is insufficient for optimal decision-making. At the same time, processing an entire episode at each step is computationally impractical. To address the memory challenges in partially observable long-horizon tasks, we propose a novel approach that treats subtask switching points as key time steps capable of effectively representing long-horizon tasks. We introduce Dilated Attentive Memory based Hierarchical Reinforcement Learning (DAM-HRL), which employs a Transformer at the higher level to capture and memorize these sparse key time steps, while an RNN at the lower level focuses on subtask representation, retaining adjacent historical sequences. This design establishes a hierarchical memory system where sparse long-term memory coexists with detailed short-term memory, enabling effective and scalable memory handling for long-horizon tasks. Furthermore, to mitigate the discrepancy between the behavior policy and the update policy during training, we propose a novel off-policy correction algorithm tailored for subtasks. This method integrates subtask switching probabilities into importance sampling, improving training stability and efficiency. To evaluate our approach, we design three long-horizon benchmark environments specifically to test long-term memory capabilities. Experimental results demonstrate that DAM-HRL significantly extends memory capacity from 50 steps to over 3000 steps under partially observable conditions, while remaining robust to the number of steps within individual subtasks. The source code is available at https://github.com/BlueFisher/Advanced-Soft-Actor-Critic.