Large language models (LLMs) are gaining increasing popularity in software engineering (SE) due to their unprecedented performance across various applications. These models are increasingly being utilized for a range of SE tasks, including requirements engineering and design, code analysis and generation, software maintenance, and quality assurance. As LLMs become more integral to SE, evaluating their effectiveness is crucial for understanding their potential in this field. In recent years, substantial efforts have been made to assess LLM performance in various SE tasks, resulting in the creation of several benchmarks tailored to this purpose. This paper offers a thorough review of 291 benchmarks, addressing three main aspects: what benchmarks are available, how benchmarks are constructed, and the future outlook for these benchmarks. We begin by examining SE tasks such as requirements engineering and design, coding assistant, software testing, AIOPs, software maintenance, and quality management. We then analyze the benchmarks and their development processes, highlighting the limitations of existing benchmarks. Additionally, we discuss the successes and failures of LLMs in different software tasks and explore future opportunities and challenges for SE-related benchmarks. We aim to provide a comprehensive overview of benchmark research in SE and offer insights to support the creation of more effective evaluation tools.

翻译：大语言模型（LLMs）因其在各种应用中前所未有的性能表现，在软件工程（SE）领域日益受到关注。这些模型正被广泛应用于一系列软件工程任务，包括需求工程与设计、代码分析与生成、软件维护以及质量保证。随着LLMs在软件工程中扮演越来越重要的角色，评估其有效性对于理解其在该领域的潜力至关重要。近年来，为评估LLMs在不同软件工程任务中的性能，已投入大量努力，并为此专门构建了多个基准测试。本文对291个基准测试进行了全面综述，涵盖三个主要方面：现有基准测试的类型、基准测试的构建方法以及这些基准测试的未来展望。我们首先审视了需求工程与设计、编码助手、软件测试、AIOps、软件维护和质量管理等软件工程任务。随后，我们分析了这些基准测试及其开发过程，并指出了现有基准测试的局限性。此外，我们讨论了LLMs在不同软件任务中的成功与失败案例，并探讨了软件工程相关基准测试的未来机遇与挑战。本文旨在为软件工程领域的基准测试研究提供一个全面的概览，并为支持开发更有效的评估工具提供见解。