Representation-driven sampling and adaptive policy resetting for improving multi-Agent reinforcement learning.

Abstract

Multi-agent reinforcement learning (MARL) plays a pivotal role in solving complex decision-making problems wherein multiple agents interact in a shared environment. However, mainstream MARL algorithms still suffer the following challenges: 1) the policies of agents tend to converge and stabilise during learning, which leads to insufficient explorations and sub-optimal strategies, particularly in environments with extremely large state, observation and action spaces and 2) the sampling inefficiency of MARL results in inadequate learning from the experience replay buffer, requiring a massive number of environmental interactions. To address these issues, we propose a novel MARL approach for various multi-agent decision-making tasks, namely efficient eXploration Joint with Training Unbiased for MARL (eXJTU-MARL), to fully enhance exploration efficiency during environmental interactions and the trajectory learning efficiency from the experience replay buffer. To achieve this, we introduce two core modules in eXJTU-MARL: adaptive policy resetting and state representation based balanced experience sampling. Specifically, for the first time, we introduce a state representation based sampling strategy that enhances data efficiency by improving the quality of experience replay samples in MARL. Accordingly, eXJTU-MARL effectively enhances sample efficiency, prevents agents from prematurely converging into sub-optimal policies and facilitates sufficient exploration of the state-action space. Extensive experiments in the StarCraft Multi-Agent Challenge environment demonstrate that our eXJTU-MARL consistently outperforms mainstream MARL baselines, highlighting the effectiveness of adaptive policy resetting and balanced experience sampling in enhancing the overall exploration capabilities and learning efficiency of MARL models in complex multi-agent environments. The code is available at GitHub: https://github.com/albert-jin/eXJTU-MARL.