Collective reflection-based multi-agent reinforcement learning framework for task-oriented dialogue policy learning.

Abstract

Multi-agent reinforcement learning cooperatively models dialogue policies, aiming to select dialogue actions for accurate task completion. Existing multi-agent dialogue policy learning methods tackle collaborative relationships through credit assignment to maximize reward or utility. However, those methods suffer from the risk of error propagation and lack of adaptive collaboration capability, which hinders the performance of dialogue policies. The primary challenges involve appropriately assigning credit to each agent and maintaining a balanced cooperative relationship among them. In this paper, we propose a centralized Multi-Agent dialogue Policy Learning (MAPL) approach that incrementally enhances the credit assignment of agents, enabling the central agent to determine the collaboration priority. Specifically, our approach constructs multiple auxiliary agents and a main agent. The auxiliary agents interact with the environment, updating their Q-values and assigning credits at the user intent-level within dialogue scenarios. The main agent updates the behavior policy by distributing user intent credits from the auxiliary agents. To balance the user intent credibility of the auxiliary agent with the behavior policy of the main agent, we propose a balancing parameter that automatically identifies when to prioritize intent and when to prioritize policy, supported by theoretical analysis. The proposed MAPL is easy to implement and incorporates various reinforcement methods. Conducting experiments on three datasets, we conclude that MAPL achieves a more efficient policy learning capability and a higher dialogue success rate, verifying it performs specific tasks efficiently. Ablation studies have verified that the number of agents and the combination fashion of agents improve the performance of dialogue policy learning.