Differentiable reinforcement learning (RL) frameworks like DiffRO offer a powerful approach for controllable text-to-speech (TTS), but are vulnerable to reward hacking, particularly for nuanced tasks like emotion control. The policy model can exploit a vanilla Reward Model (RM) by generating acoustic artifacts to achieve spurious rewards, but at the cost of degrading perceptual quality. To address this, we propose Robust Reward Policy Optimization (RRPO), a novel framework that employs a hybrid regularization scheme. This scheme develops a robust RM whose reward signal is more reliably aligned with human perception, compelling the policy to abandon detrimental shortcuts and instead learn the complex features of genuine emotions. Our ablation study confirms the enhanced robustness of our RM, as evidenced by its strong cross-lingual generalization. The subjective evaluation demonstrates that this robust RM effectively mitigates reward hacking, leading to significant improvements in both emotional expressiveness and naturalness over all baselines. Demo page: https://lrwinr.github.io/RRPO-CosyVoice.

翻译：DiffRO等可微分强化学习框架为可控文本转语音提供了强大方法，但在情感控制等精细任务中易受奖励攻击影响。策略模型可通过生成声学伪影来利用普通奖励模型获取虚假奖励，但会牺牲感知质量。为此，我们提出鲁棒奖励策略优化框架，采用混合正则化方案构建鲁棒奖励模型，使其奖励信号更可靠地符合人类感知，促使策略放弃有害捷径并学习真实情感的复杂特征。消融实验证实了奖励模型的增强鲁棒性，其强大的跨语言泛化能力即为明证。主观评估表明，该鲁棒奖励模型能有效缓解奖励攻击，在情感表现力与自然度上均显著超越所有基线方法。演示页面：https://lrwinr.github.io/RRPO-CosyVoice。