As model context lengths continue to grow, concerns about whether models effectively use the full context length have persisted. While several carefully designed long-context evaluations have recently been released, these evaluations tend to rely on retrieval from one or more sections of the context, which allows nearly all of the context tokens to be disregarded as noise. This represents only one type of task that might be performed with long context. We introduce Oolong, a benchmark of long-context reasoning tasks that require analyzing individual chunks of text on an atomic level, and then aggregating these analyses to answer distributional questions. Oolong is separated into two task sets: Oolong-synth, a set of naturalistic synthetic tasks, where we can easily ablate components of the reasoning problem; and Oolong-real, a downstream setting which requires reasoning over real-world conversational data. Oolong requires models to reason over large quantities of examples, to perform both classification and counting in-context, and to reason over temporal and user relations. Even frontier models struggle on Oolong, with GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro all achieving less than 50% accuracy on both splits at 128K. We release the data and evaluation harness for Oolong to enable further development of models that can reason over large quantities of text.

翻译：随着模型上下文长度持续增长，关于模型是否能有效利用完整上下文长度的担忧始终存在。尽管近期已发布若干精心设计的长上下文评估方案，但这些评估通常依赖于从上下文的一个或多个部分进行检索，这使得几乎所有上下文标记均可被视为噪声而忽略。这仅代表了长上下文可能执行的任务类型之一。我们推出Oolong，一个长上下文推理任务基准，要求对文本片段进行原子级分析，并聚合这些分析以回答分布性问题。Oolong分为两个任务集：Oolong-synth是一组自然主义合成任务，可轻松消融推理问题的组成部分；Oolong-real则是一个需要基于真实世界对话数据进行推理的下游场景。Oolong要求模型对大量示例进行推理，在上下文中执行分类与计数，并处理时序和用户关系。即使是前沿模型在Oolong上也表现不佳，GPT-5、Claude-Sonnet-4和Gemini-2.5-Pro在128K上下文长度下对两个任务集的准确率均低于50%。我们公开Oolong的数据集与评估框架，以促进能够处理大规模文本推理的模型进一步发展。