CODE: Cross-Context Identity Extraction for Value Decomposition Based Multi-Agent Reinforcement Learning.

Abstract

The inherent challenge of adapting to varying task configurations in contextual reinforcement learning (CRL)is further exacerbated within multi-agent systems (MAS) due to complex inter-agent correlations. In MAS, agents struggle to discern their own identities within a multi-agent team and structural credit assignment consequently becomes non-trivial. As a result, multi-agent reinforcement learning (MARL) models often fail to demonstrate comparable task performance in cross-context transfer settings. This degradation substantially constrains the adaptability as well as applicability of MARL methods in dynamic open-world environments characterized by infinite diversity of task configurations. To tackle this challenge, in this research we propose Cross-Context Identity Extraction (CODE), a generic identity-aware framework designed to enhance the cross-context generalization capability of value decomposition (VD) based MARL algorithms such as VDN, QMIX and QPLEX. By extracting expressive identity representations via Vector Quantized-Variational AutoEncoder (VQ-VAE) and performing cross-context verifications during the contextual learning dynamics, CODE effectively captures agent-specific identities within a multi-agent team and transfers them across a series of similar yet configurationally distinct task scenarios, strengthening the contextual adaptability of VD MARL algorithms. Extensive experiments across several popular MARL benchmarks demonstrate that our method outperforms state-of-the-art competitors, showcasing superior data efficiency and task performance.