Optimizing large language models (LLMs) for multi-turn conversational outcomes remains a significant challenge, especially in goal-oriented settings like AI marketing or sales agents who facilitate transactions via messaging platforms. The difficulty stems from sparse, long-horizon rewards and the discrepancy between response-level planning and token-level generation. In this technical note, we propose a formal reduction of the multi-turn RL problem into a sequence of single-turn RLHF-style problems. This is achieved by setting a learned multi-turn Q-function as the reward model for the single-turn problem. We demonstrate and prove a key insight: solving this single-turn RL problem with standard token-level PPO is equivalent to a policy improvement step within the multi-turn problem. This insight naturally leads to Iterative PPO, a batch online policy iteration algorithm that alternates between fitting Q-functions from logged conversation trajectories and improving the policy. A major practical advantage is that Iterative PPO directly leverages stable, off-the-shelf single-turn RLHF tools, making it straightforward to implement. Our method occupies a middle ground between fully online and fully offline approaches, retaining the adaptability of online updates while gaining the stability benefits of offline training.

翻译：针对多轮对话结果优化大语言模型（LLMs）仍是一项重大挑战，尤其在目标导向场景中，例如通过消息平台促进交易的AI营销或销售代理。该困难源于稀疏、长周期的奖励以及响应级规划与令牌级生成之间的不匹配。在本技术报告中，我们提出将多轮强化学习问题形式化约简为一系列单轮RLHF式问题。这是通过将学习到的多轮Q函数设置为单轮问题的奖励模型来实现的。我们展示并证明了一个关键见解：使用标准的令牌级PPO解决此单轮强化学习问题等价于在多轮问题中执行一次策略改进步骤。这一见解自然引出了迭代PPO——一种批量在线策略迭代算法，该算法交替进行从记录对话轨迹拟合Q函数和改进策略。一个主要的实际优势在于，迭代PPO直接利用了稳定、现成的单轮RLHF工具，使其易于实现。我们的方法介于完全在线和完全离线方法之间，既保留了在线更新的适应性，又获得了离线训练的稳定性优势。