The rapid proliferation of benchmarks has created significant challenges in reproducibility, transparency, and informed decision-making. However, unlike datasets and models -- which benefit from structured documentation frameworks like Datasheets and Model Cards -- evaluation methodologies lack systematic documentation standards. We introduce Eval Factsheets, a structured, descriptive framework for documenting AI system evaluations through a comprehensive taxonomy and questionnaire-based approach. Our framework organizes evaluation characteristics across five fundamental dimensions: Context (Who made the evaluation and when?), Scope (What does it evaluate?), Structure (With what the evaluation is built?), Method (How does it work?) and Alignment (In what ways is it reliable/valid/robust?). We implement this taxonomy as a practical questionnaire spanning five sections with mandatory and recommended documentation elements. Through case studies on multiple benchmarks, we demonstrate that Eval Factsheets effectively captures diverse evaluation paradigms -- from traditional benchmarks to LLM-as-judge methodologies -- while maintaining consistency and comparability. We hope Eval Factsheets are incorporated into both existing and newly released evaluation frameworks and lead to more transparency and reproducibility.

翻译：基准测试的快速扩散在可复现性、透明度和知情决策方面带来了显著挑战。然而，与数据集和模型不同——它们受益于诸如数据说明书和模型卡片等结构化文档框架——评估方法缺乏系统化的文档标准。我们提出了评估事实说明书，这是一种通过综合性分类法和基于问卷的方法来记录人工智能系统评估的结构化描述框架。我们的框架将评估特征组织为五个基本维度：背景（评估由谁在何时进行？）、范围（评估对象是什么？）、结构（评估基于什么构建？）、方法（评估如何运作？）以及对齐性（在哪些方面具有可靠性/有效性/鲁棒性？）。我们将此分类法实现为一个实用的问卷，涵盖五个部分，包含必需和推荐的文档要素。通过对多个基准测试的案例研究，我们证明评估事实说明书能够有效捕捉多样化的评估范式——从传统基准测试到LLM作为评判者的方法——同时保持一致性和可比性。我们希望评估事实说明书能够被纳入现有及新发布的评估框架中，并促进更高的透明度和可复现性。