How many mistakes do published AI papers contain? Peer-reviewed publications form the foundation upon which new research and knowledge are built. Errors that persist in the literature can propagate unnoticed, creating confusion in follow-up studies and complicating reproducibility. The accelerating pace of research and the increasing demands on the peer-review system make such mistakes harder to detect and avoid. To address this, we developed a Paper Correctness Checker based on GPT-5 to systematically identify mistakes in papers previously published at top AI conferences and journals. Our analysis focuses on objective mistakes-e.g., errors in formulas, derivations, calculations, figures, and tables-that have a clearly verifiable ground truth. We intentionally exclude subjective considerations such as novelty, importance, or writing quality. We find that published papers contain a non-negligible number of objective mistakes and that the average number of mistakes per paper has increased over time-from 3.8 in NeurIPS 2021 to 5.9 in NeurIPS 2025 (55.3% increase); from 4.1 in ICLR 2018 to 5.2 in ICLR 2025; and from 5.0 in TMLR 2022/23 to 5.5 in TMLR 2025. Human experts reviewed 316 potential mistakes identified by the AI Checker and confirmed that 263 were actual mistakes, corresponding to a precision of 83.2%. While most identified issues are relatively minor, correcting them would reduce confusion in the literature and strengthen reproducibility. The AI Checker also surfaced potentially more substantive mistakes that could affect the interpretation of results. Moreover, we show that the AI Checker can propose correct fixes for 75.8% of the identified mistakes. Overall, this study highlights the potential of frontier LLMs to detect and correct objective mistakes in published papers, helping to establish a firmer foundation of knowledge.

翻译：已发表的AI论文中包含多少错误？同行评议的出版物构成了新研究和知识积累的基础。文献中持续存在的错误可能在未被察觉的情况下传播，导致后续研究产生混淆，并使可复现性复杂化。研究步伐的加速以及对同行评议系统日益增长的需求使得此类错误更难被检测和避免。为解决这一问题，我们开发了基于GPT-5的论文正确性检查器，用于系统性地识别在顶级AI会议和期刊上已发表论文中的错误。我们的分析聚焦于客观错误——例如公式、推导、计算、图表中的错误——这些错误具有明确可验证的事实依据。我们有意排除了主观考量因素，如新颖性、重要性或写作质量。我们发现已发表论文中包含不可忽视数量的客观错误，且每篇论文的平均错误数量随时间呈上升趋势——从NeurIPS 2021的3.8个增至NeurIPS 2025的5.9个（增长55.3%）；从ICLR 2018的4.1个增至ICLR 2025的5.2个；从TMLR 2022/23的5.0个增至TMLR 2025的5.5个。人类专家审查了AI检查器识别的316个潜在错误，确认其中263个为实际错误，对应精确率为83.2%。虽然大多数识别出的问题相对较小，但修正它们将减少文献中的混淆并增强可复现性。AI检查器还揭示了可能影响结果解读的更具实质性的潜在错误。此外，我们证明AI检查器能为75.8%的识别错误提出正确修正方案。总体而言，本研究突显了前沿大语言模型在检测和修正已发表论文中客观错误方面的潜力，有助于建立更坚实的知识基础。