Large language models (LLMs) suffer from forgetting of upstream knowledge when fine-tuned. Despite efforts on mitigating forgetting, few have investigated how forgotten upstream examples are dependent on newly learned tasks. Insights on such dependencies enable efficient and targeted mitigation of forgetting. In this paper, we empirically analyze forgetting that occurs in $N$ upstream examples of language modeling or instruction-tuning after fine-tuning LLMs on one of $M$ new tasks, visualized in $M\times N$ matrices. We show that the matrices are often well-approximated with low-rank matrices, indicating the dominance of simple associations between the learned tasks and forgotten upstream examples. Leveraging the analysis, we predict forgetting of upstream examples when fine-tuning LLMs on unseen tasks with matrix completion over the empirical associations. This enables fast identification of most forgotten examples without expensive inference on the entire upstream data. Despite simplicity, the approach outperforms prior approaches that learn semantic relationships of learned tasks and upstream examples with LMs. We demonstrate the practical utility of our analysis by showing statistically significantly reduced forgetting as we upweight predicted examples for replay during fine-tuning. Code, data, and statistics collected: https://github.com/AuCson/low-rank-forgetting

翻译：大型语言模型（LLMs）在微调过程中常出现上游知识遗忘现象。尽管已有研究致力于缓解遗忘，但鲜有工作深入探究被遗忘的上游示例与新学习任务之间的依赖关系。理解此类依赖关系有助于实现高效且有针对性的遗忘缓解。本文通过实证分析，在将LLMs在$M$个新任务之一上微调后，对$N$个语言建模或指令调优上游示例的遗忘情况进行可视化，构建$M\\times N$矩阵。我们发现这些矩阵常可被低秩矩阵良好近似，表明学习任务与被遗忘上游示例之间存在占主导地位的简单关联。基于此分析，我们利用矩阵补全技术，通过经验关联预测LLMs在未见任务上微调时对上游示例的遗忘情况。该方法无需对整个上游数据进行昂贵推理，即可快速识别最易被遗忘的示例。尽管方法简洁，其性能优于以往基于语言模型学习任务与上游示例语义关系的方法。我们通过实验证明了本分析的实际效用：在微调过程中对预测示例进行加权回放，可显著降低遗忘程度，且统计意义显著。代码、数据及统计信息见：https://github.com/AuCson/low-rank-forgetting