Framework for hierarchical deep reinforcement learning with conceptual embedding.

Abstract

Deep reinforcement learning (DRL) faces challenges when the combinatorial state-action space becomes excessively large. Hierarchical DRL (HDRL) provides a potential method to address scalability; however, designing an efficient hierarchical structure remains challenging. To address this, we propose a general HDRL framework with conceptual embedding to restrict the exploration space. To the best of our knowledge, this is the first reported framework that explicitly formalizes recognition-decision decoupling through conceptual embedding within a hierarchical policy structure. It further clarifies the intrinsic relationship between the abstract state space and the goal space. This results in a transparent inference pipeline. It enables structured reasoning and integration of prior knowledge. Compared with unrestricted trial-and-error strategies, high-level abstract concepts are expected to guide the policy learning process and promote exploration efficiency. We define and analyze the complexity of the exploration space under this framework and experimentally validate its effectiveness.