Recent advances in large language models (LLMs) have shown strong reasoning capabilities through large-scale pretraining and post-training reinforcement learning, demonstrated by DeepSeek-R1. However, current post-training methods, such as Grouped Relative Policy Optimization (GRPO), mainly reward correctness, which is not aligned with the multi-dimensional objectives required in high-stakes fields such as medicine, where reasoning must also be faithful and comprehensive. We introduce Clinical-Objective Relative Policy Optimization (CRPO), a scalable, multi-objective, verifiable reinforcement learning method designed to align LLM post-training with clinical reasoning principles. CRPO integrates rule-based and verifiable reward signals that jointly optimize accuracy, faithfulness, and comprehensiveness without relying on human annotation. To demonstrate its effectiveness, we train Clinical-R1-3B, a 3B-parameter model for clinical reasoning. The experiments on three benchmarks demonstrate that our CRPO substantially improves reasoning on truthfulness and completeness over standard GRPO while maintaining comfortable accuracy enhancements. This framework provides a scalable pathway to align LLM reasoning with clinical objectives, enabling safer and more collaborative AI systems for healthcare while also highlighting the potential of multi-objective, verifiable RL methods in post-training scaling of LLMs for medical domains.

翻译：大语言模型（LLMs）的最新进展通过大规模预训练和后训练强化学习展现出强大的推理能力，DeepSeek-R1已证明了这一点。然而，当前的后训练方法（如分组相对策略优化GRPO）主要奖励正确性，这与医学等高风险领域所需的多维目标并不一致——在这些领域中，推理还必须具备忠实性和全面性。我们提出了临床目标相对策略优化（CRPO），这是一种可扩展、多目标、可验证的强化学习方法，旨在将LLM后训练与临床推理原则对齐。CRPO整合了基于规则和可验证的奖励信号，共同优化准确性、忠实性和全面性，且无需依赖人工标注。为验证其有效性，我们训练了Clinical-R1-3B，这是一个用于临床推理的30亿参数模型。在三个基准测试上的实验表明，我们的CRPO在保持显著准确性提升的同时，在真实性和完整性方面的推理能力大幅优于标准GRPO。该框架为将LLM推理与临床目标对齐提供了可扩展的路径，能够为医疗保健领域开发更安全、更具协作性的人工智能系统，同时也凸显了多目标、可验证的强化学习方法在医学领域LLM后训练扩展中的潜力。