As large language models become components of larger agentic systems, evaluation reliability becomes critical: unreliable sub-agents introduce brittleness into downstream system behavior. Yet current evaluation practice, reporting a single accuracy number from a single run, obscures the variance underlying these results, making it impossible to distinguish genuine capability improvements from lucky sampling. We propose adopting Intraclass Correlation Coefficient (ICC), a metric from measurement science, to characterize this variance. ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise. We evaluated on GAIA (Levels 1-3, measuring agentic capabilities across varying reasoning complexity) and FRAMES (measuring retrieval and factuality across multiple documents). We found that ICC varies dramatically with task structure, with reasoning and retrieval tasks (FRAMES) exhibit ICC=0.4955-0.7118 across models, and agentic tasks (GAIA) exhibiting ICC=0.304-0.774 across models. For sub-agent replacement decisions in agentic systems, accuracy improvements are only trustworthy if ICC also improves. We demonstrate that ICC converges by n=8-16 trials for structured tasks and n>=32 for complex reasoning, enabling practitioners to set evidence-based resampling budgets. We recommend reporting accuracy alongside ICC and within-query variance as standard practice, and propose updated Evaluation Cards capturing these metrics. By making evaluation stability visible, we aim to transform agentic benchmarking from opaque leaderboard competition to trustworthy experimental science. Our code is open-sourced at https://github.com/youdotcom-oss/stochastic-agent-evals.

翻译：随着大语言模型成为大型智能体系统的组成部分，评估的可靠性变得至关重要：不可靠的子智能体会导致下游系统行为脆弱。然而，当前的评估实践仅报告单次运行的单一准确率数值，掩盖了结果背后的方差，使得无法区分真实能力提升与幸运采样。我们建议采用测量科学中的组内相关系数（ICC）来表征这种方差。ICC将观测方差分解为查询间方差（任务难度）和查询内方差（智能体不一致性），从而揭示报告结果反映的是真实能力还是测量噪声。我们在GAIA（第1-3级，测量不同推理复杂度下的智能体能力）和FRAMES（测量跨多文档的检索与事实性）上进行了评估。我们发现ICC随任务结构变化显著：推理与检索任务（FRAMES）在不同模型间ICC为0.4955-0.7118，而智能体任务（GAIA）在不同模型间ICC为0.304-0.774。对于智能体系统中的子智能体替换决策，仅当ICC同时提升时，准确率改进才可信。我们证明ICC在结构化任务中经n=8-16次试验即可收敛，复杂推理任务需n≥32次，这使实践者能够设定基于证据的重采样预算。我们建议将准确率与ICC及查询内方差作为标准实践一并报告，并提出包含这些指标的更新版评估卡片。通过使评估稳定性可见，我们旨在将智能体基准测试从模糊的排行榜竞争转变为可信的实验科学。代码已开源：https://github.com/youdotcom-oss/stochastic-agent-evals。