Direct Preference Optimization (DPO) is a widely used reinforcement learning from human feedback (RLHF) method across various domains. Recent research has increasingly focused on the role of token importance in improving DPO effectiveness. It is observed that identical or semantically similar content (defined as ambiguous content) frequently appears within the preference pairs. We hypothesize that the presence of ambiguous content during DPO training may introduce ambiguity, thereby limiting further improvements in alignment. Through mathematical analysis and proof-of-concept experiments, we reveal that ambiguous content may potentially introduce ambiguities, thereby degrading performance. To address this issue, we introduce Ambiguity Awareness Optimization (AAO), a simple yet effective approach that automatically re-weights ambiguous content to reduce ambiguities by calculating semantic similarity from preference pairs. Through extensive experiments, we demonstrate that AAO consistently and significantly surpasses state-of-the-art approaches in performance, without markedly increasing response length, across multiple model scales and widely adopted benchmark datasets, including AlpacaEval 2, MT-Bench, and Arena-Hard. Specifically, AAO outperforms DPO by up to 8.9 points on AlpacaEval 2 and achieves an improvement of by up to 15.0 points on Arena-Hard.

翻译：直接偏好优化（DPO）是一种广泛应用于各领域的基于人类反馈的强化学习（RLHF）方法。近期研究日益关注词元重要性在提升DPO效能中的作用。研究发现，相同或语义相似的内容（定义为歧义性内容）频繁出现在偏好对中。我们假设，DPO训练过程中歧义性内容的存在可能引入歧义，从而限制对齐效果的进一步提升。通过数学分析与概念验证实验，我们揭示了歧义性内容可能引入模糊性，进而导致性能下降。为解决此问题，我们提出了歧义感知优化（AAO），这是一种简单而有效的方法，通过计算偏好对间的语义相似度，自动对歧义性内容进行重加权以降低歧义。通过大量实验，我们证明AAO在多个模型规模和广泛采用的基准数据集（包括AlpacaEval 2、MT-Bench和Arena-Hard）上，均能持续且显著超越现有最优方法，且未明显增加响应长度。具体而言，AAO在AlpacaEval 2上较DPO提升高达8.9分，在Arena-Hard上提升幅度达15.0分。