Learning to communicate in order to share state information is an active problem in the area of multi-agent reinforcement learning (MARL). The credit assignment problem, the non-stationarity of the communication environment and the creation of influenceable agents are major challenges within this research field which need to be overcome in order to learn a valid communication protocol. This paper introduces the novel multi-agent counterfactual communication learning (MACC) method which adapts counterfactual reasoning in order to overcome the credit assignment problem for communicating agents. Secondly, the non-stationarity of the communication environment while learning the communication Q-function is overcome by creating the communication Q-function using the action policy of the other agents and the Q-function of the action environment. Additionally, a social loss function is introduced in order to create influenceable agents which is required to learn a valid communication protocol. Our experiments show that MACC is able to outperform the state-of-the-art baselines in four different scenarios in the Particle environment.

翻译：学习交流以分享国家信息是多试剂强化学习(MARL)领域的一个积极问题。 信用分配问题、通信环境的不固定性以及建立具有影响力的代理人是这一研究领域的重大挑战,需要加以克服,以便学习有效的通信协议。本文件介绍了新的多试剂反事实交流学习方法,该方法适应反事实推理,以克服通信代理的信用分配问题。第二,在学习通信Q功能的同时,通信环境的不固定性通过利用其他代理人的行动政策和行动环境的功能来创建通信Q功能而得以克服。此外,还引入了社会损失功能,以创造具有影响力的代理人,而这是学习有效的通信协议所必需的。我们的实验表明,在粒子环境中,通信理事会能够在四种不同情景中超越最先进的基线。