A Decentralized Actor-Critic Algorithm With Entropy Regularization and Its Finite-Time Analysis.

Abstract

Decentralized actor-critic (AC) is one of the most dominant algorithms for dealing with multiagent reinforcement learning (MARL) problems. However, exploration-efficient, sample-efficient, and communication-efficient are difficult to achieve simultaneously by existing decentralized AC methods. For this reason, this article develops a decentralized multiagent AC algorithm by incorporating entropy regularization to improve exploration with theoretical guarantees, referred to as multi-agent AC algorithm with entropy regularization (MACE). Moreover, we rigorously prove that MACE can achieve sample complexity $\mathcal {O}(\epsilon ^{-2}\ln \epsilon ^{-1})$ and communication complexity of $\mathcal {O}(\epsilon ^{-1}\ln \epsilon ^{-1})$ , which match the best complexities at present. Finally, the performance of MACE is also evaluated on reinforcement learning (RL) tasks. The experimental results show that the proposed algorithm achieves better exploration efficiency than state-of-the-art decentralized AC-type algorithms.