Humans and intelligent animals can internalize new information and accurately internalize their implications to perform downstream tasks. While large language models (LLMs) can achieve this through in-context learning (ICL) when the information (news) is explicitly given as context, adequately integrating the information into model weights via fine-tuning remains challenging. In this paper, we introduce New News, a dataset composed of hypothetical yet plausible news spanning multiple domains (mathematics, coding, discoveries, leaderboards, events), accompanied by downstream evaluation questions whose correct answers critically depend on understanding and internalizing the news. First, we demonstrate a substantial gap between naive fine-tuning and in-context learning (FT-ICL gap) on our dataset. To address this gap, we explore a suite of self-play data generation protocols -- paraphrases, implications, and Self-QA -- designed to distill the knowledge processed by the model with context into the weights of the model, which we term System-2 Fine-tuning (Sys2-FT). We systematically evaluate ICL and Sys2-FT performance across data domains and model scales with the Qwen 2.5 family of models. Our results demonstrate that the Self-QA protocol of Sys2-FT significantly improves models' in-weight learning of the news while preserving general capabilities. Furthermore, we discover the contextual shadowing effect, where training with the news in context followed by its rephrases or QAs catastrophically degrades learning of the news. Finally, we show preliminary evidence of an emerging scaling law of Sys2-FT.

翻译：人类与智能动物能够内化新信息并准确理解其含义，以执行下游任务。尽管大型语言模型（LLM）可通过上下文学习（ICL）在信息（消息）被显式提供为上下文时实现这一目标，但通过微调将信息充分整合至模型权重中仍具挑战性。本文提出New News数据集，该数据集包含跨多个领域（数学、编程、科学发现、排行榜、事件）的假设性但合理的新消息，并附有下游评估问题，其正确答案关键依赖于对消息的理解与内化。首先，我们在该数据集上证明了朴素微调与上下文学习之间存在显著差距（FT-ICL差距）。为弥合此差距，我们探索了一套自博弈数据生成协议——包括释义、推理与自问自答（Self-QA）——旨在将由模型在上下文中处理的知识提炼至模型权重中，我们将此方法称为系统2微调（Sys2-FT）。我们基于Qwen 2.5系列模型，系统评估了ICL与Sys2-FT在不同数据领域和模型规模下的性能。结果表明，Sys2-FT的自问自答协议显著提升了模型对消息的权重内学习能力，同时保持了通用能力。此外，我们发现了上下文遮蔽效应：当模型在上下文中学习消息后，再使用其重述或问答进行训练，会灾难性地削弱对消息的学习效果。最后，我们初步揭示了Sys2-FT中呈现的扩展规律迹象。