Reliable reward models (RMs) are critical for ensuring the safe alignment of large language models (LLMs). However, current evaluation methods focus solely on preference perception accuracies in given specific scenarios, obscuring the critical vulnerabilities of RMs in real-world scenarios. We identify the true challenge lies in assessing a novel dimension: Suitability, defined as conditional reliability under specific real-world perturbations. To this end, we introduce Reward Auditor, a hypothesis-testing framework specifically designed for RM suitability inference. Rather than answering "How accurate is the RM's preference perception for given samples?", it employs scientific auditing to answer: "Can we infer RMs exhibit systematic vulnerabilities in specific real-world scenarios?". Under real-world perturbed scenarios, Reward Auditor quantifies statistical significance and effect size by auditing distribution degradation of RM preference perception confidence. This enables inference of both the certainty and severity of RM vulnerabilities across diverse real-world scenarios. This lays a solid foundation for building next-generation LLM alignment systems that are verifiably safe, more robust, and trustworthy.

翻译：可靠的奖励模型对于确保大型语言模型的安全对齐至关重要。然而，当前评估方法仅关注给定特定场景下的偏好感知准确度，掩盖了奖励模型在真实世界场景中的关键脆弱性。我们指出真正的挑战在于评估一个新维度：适用性，即特定真实世界扰动条件下的可靠性。为此，我们提出奖励审计员，一个专门为奖励模型适用性推断设计的假设检验框架。它不回答“奖励模型对给定样本的偏好感知准确度如何？”，而是通过科学审计回答：“我们能否推断奖励模型在特定真实世界场景中表现出系统性脆弱性？”。在真实世界扰动场景下，奖励审计员通过审计奖励模型偏好感知置信度的分布退化，量化统计显著性与效应大小。这使得能够推断不同真实世界场景中奖励模型脆弱性的确定性与严重程度。这为构建可验证安全、更鲁棒且可信赖的下一代大型语言模型对齐系统奠定了坚实基础。