The Consolidated Standards of Reporting Trials statement is the global benchmark for transparent and high-quality reporting of randomized controlled trials. Manual verification of CONSORT adherence is a laborious, time-intensive process that constitutes a significant bottleneck in peer review and evidence synthesis. This study aimed to systematically evaluate the accuracy and reliability of contemporary LLMs in identifying the adherence of published RCTs to the CONSORT 2010 statement under a zero-shot setting. We constructed a golden standard dataset of 150 published RCTs spanning diverse medical specialties. The primary outcome was the macro-averaged F1-score for the three-class classification task, supplemented by item-wise performance metrics and qualitative error analysis. Overall model performance was modest. The top-performing models, Gemini-2.5-Flash and DeepSeek-R1, achieved nearly identical macro F1 scores of 0.634 and Cohen's Kappa coefficients of 0.280 and 0.282, respectively, indicating only fair agreement with expert consensus. A striking performance disparity was observed across classes: while most models could identify compliant items with high accuracy (F1 score > 0.850), they struggled profoundly with identifying non-compliant and not applicable items, where F1 scores rarely exceeded 0.400. Notably, some high-profile models like GPT-4o underperformed, achieving a macro F1-score of only 0.521. LLMs show potential as preliminary screening assistants for CONSORT checks, capably identifying well-reported items. However, their current inability to reliably detect reporting omissions or methodological flaws makes them unsuitable for replacing human expertise in the critical appraisal of trial quality.

翻译：CONSORT声明是随机对照试验透明、高质量报告的全球基准。人工核查CONSORT遵循情况是一项耗时费力的过程，已成为同行评审和证据合成的重要瓶颈。本研究旨在系统评估当代大型语言模型在零样本设置下识别已发表随机对照试验对CONSORT 2010声明遵循情况的准确性与可靠性。我们构建了一个包含150篇跨医学各专科已发表随机对照试验的黄金标准数据集。主要结局指标为三分类任务的宏平均F1分数，辅以条目级性能指标和定性错误分析。总体模型性能表现一般。表现最佳的模型Gemini-2.5-Flash和DeepSeek-R1分别获得近乎相同的宏F1分数0.634，Cohen's Kappa系数分别为0.280和0.282，表明与专家共识仅达到中等一致性。观察到显著的类别间性能差异：虽然大多数模型能高精度识别合规条目（F1分数>0.850），但在识别不合规与不适用条目方面存在严重困难，其F1分数鲜有超过0.400。值得注意的是，部分知名模型如GPT-4o表现欠佳，宏F1分数仅为0.521。大型语言模型展现出作为CONSORT核查初步筛查工具的潜力，能够有效识别报告良好的条目。然而，其当前无法可靠检测报告遗漏或方法学缺陷的能力，使其尚不足以在试验质量的关键评估中替代人类专家。