Large Language Models have gained remarkable interest in industry and academia. The increasing interest in LLMs in academia is also reflected in the number of publications on this topic over the last years. For instance, alone 78 of the around 425 publications at ICSE 2024 performed experiments with LLMs. Conducting empirical studies with LLMs remains challenging and raises questions on how to achieve reproducible results, for both researchers and practitioners. One important step towards excelling in empirical research on LLM and their application is to first understand to what extent current research results are eventually reproducible and what factors may impede reproducibility. This investigation is within the scope of our work. We contribute an analysis of the reproducibility of LLM-centric studies, provide insights into the factors impeding reproducibility, and discuss suggestions on how to improve the current state. In particular, we studied the 85 articles describing LLM-centric studies, published at ICSE 2024 and ASE 2024. Of the 85 articles, 18 provided research artefacts and used OpenAI models. We attempted to replicate those 18 studies. Of the 18 studies, only five were sufficiently complete and executable. For none of the five studies, we were able to fully reproduce the results. Two studies seemed to be partially reproducible, and three studies did not seem to be reproducible. Our results highlight not only the need for stricter research artefact evaluations but also for more robust study designs to ensure the reproducible value of future publications.

翻译：大语言模型在工业界和学术界引起了显著关注。学术界对LLMs日益增长的兴趣也反映在近年来相关主题的发表数量上。例如，仅ICSE 2024约425篇论文中就有78篇开展了LLM实验。开展基于LLMs的实证研究仍然具有挑战性，并向研究者和实践者提出了如何获得可复现结果的问题。要在LLM及其应用的实证研究中取得突破，首要步骤是理解当前研究成果在多大程度上最终可复现，以及哪些因素可能阻碍复现性。这正是我们工作的研究范畴。我们贡献了对以LLM为核心研究的可复现性分析，揭示了阻碍复现性的因素，并就如何改善现状提出了建议。具体而言，我们研究了发表于ICSE 2024和ASE 2024的85篇以LLM为核心研究的论文。其中18篇提供了研究构件并使用了OpenAI模型。我们尝试复现了这18项研究。在这18项研究中，仅五项具备足够的完整性和可执行性。而在这五项中，没有一项研究的结果能够被完全复现：两项研究似乎部分可复现，三项研究似乎不可复现。我们的结果不仅凸显了更严格的研究构件评估的必要性，也表明需要更稳健的研究设计来确保未来发表成果的可复现价值。