Multimodal document retrieval aims to identify and retrieve various forms of multimodal content, such as figures, tables, charts, and layout information from extensive documents. Despite its increasing popularity, there is a notable lack of a comprehensive and robust benchmark to effectively evaluate the performance of systems in such tasks. To address this gap, this work introduces a new benchmark, named MMDocIR, that encompasses two distinct tasks: page-level and layout-level retrieval. The former evaluates the performance of identifying the most relevant pages within a long document, while the later assesses the ability of detecting specific layouts, providing a more fine-grained measure than whole-page analysis. A layout refers to a variety of elements, including textual paragraphs, equations, figures, tables, or charts. The MMDocIR benchmark comprises a rich dataset featuring 1,685 questions annotated by experts and 173,843 questions with bootstrapped labels, making it a valuable resource in multimodal document retrieval for both training and evaluation. Through rigorous experiments, we demonstrate that (i) visual retrievers significantly outperform their text counterparts, (ii) MMDocIR training set effectively enhances the performance of multimodal document retrieval and (iii) text retrievers leveraging VLM-text significantly outperforms retrievers relying on OCR-text. Our dataset is available at https://mmdocrag.github.io/MMDocIR/.

翻译：多模态文档检索旨在从海量文档中识别并检索多种形式的多模态内容，如图表、表格、图表及版面信息。尽管该领域日益受到关注，但目前仍缺乏一个全面且稳健的基准来有效评估系统在此类任务中的性能。为填补这一空白，本研究提出了名为MMDocIR的新基准，涵盖两个独立任务：页面级检索与版面级检索。前者评估在长文档中识别最相关页面的性能，后者则评测检测特定版面的能力，提供了比整页分析更细粒度的度量标准。版面包含多种元素，如文本段落、公式、插图、表格或图表。MMDocIR基准集成了一个丰富的数据集，包含专家标注的1,685个问题及通过自举标注生成的173,843个问题，为多模态文档检索的训练与评估提供了宝贵资源。通过严谨的实验，我们证明：（i）视觉检索模型显著优于纯文本检索模型；（ii）MMDocIR训练集能有效提升多模态文档检索的性能；（iii）利用VLM-text的文本检索模型显著优于依赖OCR-text的检索模型。我们的数据集发布于https://mmdocrag.github.io/MMDocIR/。