Contemporary large language model (LLM) training pipelines require the assembly of internet-scale databases full of text data from a variety of sources (e.g., web, academic, and publishers). Preprocessing these datasets via deduplication -- detecting and eliminating additional instances of the same content -- is a major focus for assembling and curating training datasets for LLMs. Unrestrained, duplicates in the training dataset increase training costs and lead to undesirable properties such as memorization in trained models or cheating on evaluation. Unfortunately, contemporary approaches to document-level deduplication are either unreliable at accurately identifying duplicate documents or extremely expensive in terms of both runtime and memory. We propose LSHBloom, an extension to MinhashLSH, which replaces the expensive LSHIndex with lightweight Bloom filters. LSHBloom demonstrates the same state-of-the-art deduplication performance as MinhashLSH, with only a marginal increase in false positives (near zero in our experiments), while boasting competitive runtime (12$\times$ faster than MinhashLSH on peS2o) and, crucially, using 18$\times$ less disk space than MinhashLSH (as measured on peS2o). Based on extrapolation, we show that this advantage in space and runtime remains even at the extreme scale of several billion documents. LSHBloom allows practitioners to access the deduplication quality of MinHashLSH at scales that are normally only tractable for less sophisticated, heuristic solutions. As a result, LSHBloom promises to enable scaling high-quality document deduplication to internet-scale text datasets.

翻译：当代大规模语言模型（LLM）训练流程需要整合来自多种来源（如网络、学术文献和出版机构）的互联网规模文本数据库。通过去重（即检测并消除重复内容）对这些数据集进行预处理，是构建和优化LLM训练数据集的核心环节。若不加控制，训练数据中的重复项会增加训练成本，并导致模型出现记忆效应或在评估中作弊等不良特性。然而，当前文档级去重方法要么在准确识别重复文档方面不可靠，要么在运行时间和内存消耗上极为昂贵。本文提出LSHBloom，作为MinhashLSH的扩展方法，它用轻量级布隆过滤器替代了昂贵的LSH索引。LSHBloom在保持与MinhashLSH相同的先进去重性能的同时，仅略微增加误报率（实验中接近零），并在运行效率上具有竞争力（在peS2o数据集上比MinhashLSH快12倍），关键的是，其磁盘空间占用比MinhashLSH减少18倍（基于peS2o数据集测量）。通过外推分析，我们证明这种空间和运行时间优势在数十亿文档的超大规模场景下依然存在。LSHBloom使实践者能够在通常仅适用于简单启发式解决方案的规模上，实现MinhashLSH级别的去重质量。因此，LSHBloom有望将高质量文档去重技术扩展到互联网规模的文本数据集。