Large reasoning models (LRMs) extend large language models by generating explicit chain-of-thought (CoT) reasoning, significantly improving mathematical and logical problem solving. However, this explicit reasoning process also introduces new safety risks, as unsafe behaviors often emerge within intermediate reasoning trajectories, even when final answers appear harmless. Existing safety alignment approaches primarily rely on supervised fine-tuning (SFT) over safety-oriented long CoT datasets. While intuitive, we find that SFT produces inconsistent safety improvements, degrades reasoning ability, and generalizes poorly across model families. These limitations suggest that purely supervised approaches are insufficient for robust safety alignment in LRMs. To address this, we investigate reinforcement learning (RL) as a complementary optimization framework for LRM safety training. Unlike SFT, RL directly optimizes model policies with reward feedback, enabling more adaptive and stable alignment. Extensive experiments across multiple model families and benchmarks show that RL achieves stronger and more consistent safety gains while maintaining reasoning competence. Further analysis of reflection dynamics and token-level entropy reveals that RL suppresses unsafe exploratory reasoning while preserving reflective depth, leading to safer and more reliable reasoning processes.

翻译：大型推理模型通过生成显式的思维链推理过程，扩展了大型语言模型的能力，显著提升了数学与逻辑问题的解决水平。然而，这种显式推理过程也引入了新的安全风险——不安全行为常出现在中间推理轨迹中，即使最终答案看似无害。现有的安全对齐方法主要依赖于基于安全导向的长思维链数据集进行监督微调。尽管该方法直观，但我们发现监督微调带来的安全性改进不一致，会降低推理能力，且在不同模型家族间泛化能力较差。这些局限表明，纯监督方法不足以实现大型推理模型的鲁棒安全对齐。为此，我们探索将强化学习作为大型推理模型安全训练的补充优化框架。与监督微调不同，强化学习通过奖励反馈直接优化模型策略，实现更具适应性和稳定性的对齐。跨多个模型家族与基准测试的广泛实验表明，强化学习在保持推理能力的同时，实现了更强且更一致的安全性提升。对反思动态和词元级熵的进一步分析揭示，强化学习能抑制不安全的探索性推理，同时保持反思深度，从而形成更安全可靠的推理过程。