We study a long-run mean-variance team stochastic game (MV-TSG), where each agent shares a common mean-variance objective for the system and takes actions independently to maximize it. MV-TSG has two main challenges. First, the variance metric is neither additive nor Markovian in a dynamic setting. Second, simultaneous policy updates of all agents lead to a non-stationary environment for each individual agent. Both challenges make dynamic programming inapplicable. In this paper, we study MV-TSGs from the perspective of sensitivity-based optimization. The performance difference and performance derivative formulas for joint policies are derived, which provide optimization information for MV-TSGs. We prove the existence of a deterministic Nash policy for this problem. Subsequently, we propose a Mean-Variance Multi-Agent Policy Iteration (MV-MAPI) algorithm with a sequential update scheme, where individual agent policies are updated one by one in a given order. We prove that the MV-MAPI algorithm converges to a first-order stationary point of the objective function. By analyzing the local geometry of stationary points, we derive specific conditions for stationary points to be (local) Nash equilibria, and further, strict local optima. To solve large-scale MV-TSGs in scenarios with unknown environmental parameters, we extend the idea of trust region methods to MV-MAPI and develop a multi-agent reinforcement learning algorithm named Mean-Variance Multi-Agent Trust Region Policy Optimization (MV-MATRPO). We derive a performance lower bound for each update of joint policies. Finally, numerical experiments on energy management in multiple microgrid systems are conducted.

翻译：本文研究长期均值-方差团队随机博弈（MV-TSG），其中每个智能体共享系统的共同均值-方差目标，并通过独立行动实现其最大化。MV-TSG面临两大挑战：其一，方差度量在动态环境中既非可加性亦非马尔可夫性；其二，所有智能体的同步策略更新导致个体面临非平稳环境。这两项挑战使得动态规划方法无法适用。本文从基于灵敏度的优化角度研究MV-TSG，推导出联合策略的性能差异与性能导数公式，为MV-TSG提供优化信息。我们证明了该问题存在确定性纳什策略，进而提出具有顺序更新机制的均值-方差多智能体策略迭代（MV-MAPI）算法，其中智能体策略按给定顺序逐一更新。理论证明MV-MAPI算法能收敛至目标函数的一阶驻点。通过分析驻点的局部几何特性，我们推导出驻点成为（局部）纳什均衡及严格局部最优解的具体条件。针对环境参数未知的大规模MV-TSG场景，我们将置信域方法思想拓展至MV-MAPI，开发了名为均值-方差多智能体置信域策略优化（MV-MATRPO）的多智能体强化学习算法，并推导了联合策略每次更新的性能下界。最后，通过在多微电网系统能源管理场景中的数值实验验证了所提方法的有效性。