Scalarisation functions are widely employed in MORL algorithms to enable intelligent decision-making. However, these functions often struggle to approximate the Pareto front accurately, rendering them unideal in complex, uncertain environments. This study examines selected Multi-Objective Reinforcement Learning (MORL) algorithms across MORL environments with discrete action and observation spaces. We aim to investigate further the limitations associated with scalarisation approaches for decision-making in multi-objective settings. Specifically, we use an outer-loop multi-policy methodology to assess the performance of a seminal single-policy MORL algorithm, MO Q-Learning implemented with linear scalarisation and Chebyshev scalarisation functions. In addition, we explore a pioneering inner-loop multi-policy algorithm, Pareto Q-Learning, which offers a more robust alternative. Our findings reveal that the performance of the scalarisation functions is highly dependent on the environment and the shape of the Pareto front. These functions often fail to retain the solutions uncovered during learning and favour finding solutions in certain regions of the solution space. Moreover, finding the appropriate weight configurations to sample the entire Pareto front is complex, limiting their applicability in uncertain settings. In contrast, inner-loop multi-policy algorithms may provide a more sustainable and generalizable approach and potentially facilitate intelligent decision-making in dynamic and uncertain environments.

翻译：标量化函数在多目标强化学习（MORL）算法中被广泛采用以实现智能决策。然而，这些函数往往难以精确逼近帕累托前沿，使其在复杂、不确定的环境中表现欠佳。本研究在具有离散动作和观测空间的MORL环境中，对选定的多目标强化学习算法进行了检验。我们旨在进一步探究多目标场景下基于标量化方法的决策局限性。具体而言，我们采用外层多策略方法评估了经典的单策略MORL算法——结合线性标量化与切比雪夫标量化函数实现的MO Q-Learning的性能。此外，我们探索了一种开创性的内层多策略算法——帕累托Q-Learning，其提供了更稳健的替代方案。研究结果表明，标量化函数的性能高度依赖于环境及帕累托前沿的形态。这些函数通常无法保留学习过程中发现的解，并倾向于在解空间的特定区域寻找解。此外，为采样整个帕累托前沿而寻找合适的权重配置十分复杂，限制了其在不确定环境中的适用性。相比之下，内层多策略算法可能提供更可持续、可泛化的方法，并有望在动态与不确定环境中促进智能决策。