We tested over 20 Transformer models for ranking long documents (including recent LongP models trained with FlashAttention and RankGPT models "powered" by OpenAI and Anthropic cloud APIs). We compared them with the simple FirstP baseline, which applied the same model to truncated input (up to 512 tokens). On MS MARCO, TREC DL, and Robust04 no long-document model outperformed FirstP by more than 5% (on average). We hypothesized that this lack of improvement is not due to inherent model limitations, but due to benchmark positional bias (most relevant passages tend to occur early in documents), which is known to exist in MS MARCO. To confirm this, we analyzed positional relevance distributions across four long-document corpora (with six query sets) and observed the same early-position bias. Surprisingly, we also found bias in six BEIR collections, which are typically categorized as short-document datasets. We then introduced a new diagnostic dataset, MS MARCO FarRelevant, where relevant spans were deliberately placed beyond the first 512 tokens. On this dataset, many long-context models (including RankGPT) performed at random-baseline level, suggesting overfitting to positional bias. We also experimented with debiasing training data, but with limited success. Our findings (1) highlight the need for careful benchmark design in evaluating long-context models for document ranking, (2) identify model types that are more robust to positional bias, and (3) motivate further work on approaches to debias training data. We release our code and data to support further research.

翻译：我们测试了超过20个用于长文档排序的Transformer模型（包括近期使用FlashAttention训练的LongP模型以及基于OpenAI和Anthropic云API的RankGPT模型）。我们将这些模型与简单的FirstP基线进行比较，后者将相同模型应用于截断输入（最多512个词元）。在MS MARCO、TREC DL和Robust04数据集上，没有任何长文档模型的平均性能超过FirstP基线5%以上。我们假设这种改进的缺失并非源于模型固有局限，而是由于基准数据集存在位置偏差（相关段落往往出现在文档前部），这在MS MARCO中已被证实。为验证此假设，我们分析了四个长文档语料库（包含六组查询）中的位置相关性分布，均观察到相同的前置偏差。令人惊讶的是，在通常归类为短文档数据集的六个BEIR集合中也发现了此类偏差。随后我们提出了新的诊断数据集MS MARCO FarRelevant，其中相关文本段被刻意置于前512个词元之后。在该数据集上，包括RankGPT在内的多数长上下文模型表现仅达到随机基线水平，表明其存在对位置偏差的过拟合。我们尝试对训练数据进行去偏差处理，但效果有限。本研究贡献在于：（1）揭示评估长上下文文档排序模型时需要谨慎设计基准数据集，（2）识别对位置偏差更具鲁棒性的模型类型，（3）推动训练数据去偏差方法的后续研究。我们将公开代码与数据以支持进一步探索。