Efficient Exploration for Multi-Agent Diversity With Agent Identity.

Abstract

Multi-Agent Reinforcement Learning (MARL) has proven to be effective in learning cooperative policies, where agents learn decentralized policies, sharing the same network parameters, through centralized training. However, this parameter sharing can lead to similar behaviors among agents, hindering effective exploration. Existing multi-agent diversity methods that rely on the variational inference methods to differentiate agents may suffer from significant overfitting, which in turn hinders the exploration of new trajectories. To encourage multi-agent diversity and efficient exploration, we propose Active Exploration with Agent-Identity (AEAI), a novel exploration method, which maximizes the entropy over trajectories of different agents to promote sufficient exploration. Moreover, we derive a novel lower bound for the mutual information objective based on the successor features to align the directions of trajectories and agent identities in order to learn agent identity-conditioned policies. We combine these two items and integrate our method with existing MARL methods. We evaluate our proposed AEAI on challenging multi-agent tasks across various MARL benchmarks. Experimental results show that our method consistently outperforms existing state-of-the-art methods, highlighting its effectiveness in fostering diversity and improving exploration.