The recent DeepSeek-R1 release has demonstrated the immense potential of reinforcement learning (RL) in enhancing the general reasoning capabilities of large language models (LLMs). While DeepSeek-R1 and other follow-up work primarily focus on applying RL to competitive coding and math problems, this paper introduces SWE-RL, the first approach to scale RL-based LLM reasoning for real-world software engineering. Leveraging a lightweight rule-based reward (e.g., the similarity score between ground-truth and LLM-generated solutions), SWE-RL enables LLMs to autonomously recover a developer's reasoning processes and solutions by learning from extensive open-source software evolution data -- the record of a software's entire lifecycle, including its code snapshots, code changes, and events such as issues and pull requests. Trained on top of Llama 3, our resulting reasoning model, Llama3-SWE-RL-70B, achieves a 41.0% solve rate on SWE-bench Verified -- a human-verified collection of real-world GitHub issues. To our knowledge, this is the best performance reported for medium-sized (<100B) LLMs to date, even comparable to leading proprietary LLMs like GPT-4o. Surprisingly, despite performing RL solely on software evolution data, Llama3-SWE-RL has even emerged with generalized reasoning skills. For example, it shows improved results on five out-of-domain tasks, namely, function coding, library use, code reasoning, mathematics, and general language understanding, whereas a supervised-finetuning baseline even leads to performance degradation on average. Overall, SWE-RL opens up a new direction to improve the reasoning capabilities of LLMs through reinforcement learning on massive software engineering data.

翻译：近期发布的DeepSeek-R1展示了强化学习在提升大语言模型通用推理能力方面的巨大潜力。尽管DeepSeek-R1及其他后续工作主要聚焦于将强化学习应用于竞技编程和数学问题，本文提出了SWE-RL——首个将基于强化学习的大语言模型推理方法扩展至真实世界软件工程的方案。通过利用轻量级基于规则的奖励机制（例如，真实解决方案与LLM生成方案之间的相似度评分），SWE-RL使大语言模型能够从海量开源软件演化数据（即记录软件全生命周期的数据，包括代码快照、代码变更以及问题报告和拉取请求等事件）中自主学习，从而自主还原开发者的推理过程与解决方案。基于Llama 3训练得到的推理模型Llama3-SWE-RL-70B，在SWE-bench Verified（一个经人工验证的真实GitHub问题数据集）上实现了41.0%的解决率。据我们所知，这是目前中等规模（<100B参数）大语言模型中报告的最佳性能，甚至可与GPT-4o等领先的专有大语言模型相媲美。令人惊讶的是，尽管仅在软件演化数据上进行强化学习训练，Llama3-SWE-RL还展现出泛化的推理能力。例如，它在五个领域外任务（即函数编码、库使用、代码推理、数学和通用语言理解）上均表现出改进效果，而监督微调基线模型在这些任务上的平均性能甚至有所下降。总体而言，SWE-RL为通过海量软件工程数据的强化学习来提升大语言模型的推理能力开辟了新的研究方向。