Data filtering has become a powerful tool for improving model performance while reducing computational cost. However, as large language model compute budgets continue to grow, the limited data volume provided by heavily filtered and deduplicated datasets will become a practical constraint. In efforts to better understand how to proceed, we study model performance at various compute budgets and across multiple pre-training datasets created through data filtering and deduplication. We find that, given appropriate modifications to the training recipe, repeating existing aggressively filtered datasets for up to ten epochs can outperform training on the ten times larger superset for a single epoch across multiple compute budget orders of magnitude. While this finding relies on repeating the dataset for many epochs, we also investigate repeats within these datasets at the document level. We find that not all documents within a dataset are equal, and we can create better datasets relative to a token budget by explicitly manipulating the counts of individual documents. We conclude by arguing that even as large language models scale, data filtering remains an important direction of research.

翻译：数据过滤已成为提升模型性能并降低计算成本的有力工具。然而，随着大型语言模型计算预算的持续增长，经过严格过滤和去重处理的数据集所提供的有限数据量将成为实际制约因素。为深入探索后续发展方向，我们研究了不同计算预算下模型在多个通过数据过滤和去重构建的预训练数据集上的表现。研究发现，在适当调整训练方案的前提下，对现有经过激进过滤的数据集重复训练多达十个周期，其性能可优于在十倍规模超集上仅训练单个周期，这一结论在多个数量级的计算预算下均成立。尽管该发现依赖于对数据集的多周期重复训练，我们也探究了这些数据集内部在文档层面的重复问题。研究发现，数据集中并非所有文档都具有同等价值，通过显式调控单个文档的重复次数，我们能够在给定令牌预算下构建更优质的数据集。最后我们指出，即使大型语言模型持续扩展规模，数据过滤仍是值得深入研究的重要方向。