Continual learning, one's ability to adapt to a sequence of tasks without forgetting previously acquired knowledge, remains a major challenge in machine learning and a key gap between artificial and human intelligence. While regularisation and replay perform well in vision, they lag behind multi-task learning for large language models (LLMs), especially at scale with many tasks. We revisit replay and argue that two failure modes drive this gap: selection (what to rehearse) and integration (how to consolidate new knowledge). To address selection, we propose Surprise-prioritised Replay (SuRe), a simple, architecture-agnostic rule that ranks and stores the most surprising (high Negative Log-Likelihood) sequences. SuRe achieves state-of-the-art performance in the Large Number of Tasks (LNT) setting and delivers the best overall average across both Standard CL and LNT benchmarks. To address integration, we add a dual-learner design with fast and slow LoRA adapters merged via an exponential moving average (EMA), enabling rapid adaptation while stabilising long-term knowledge. Combining SuRe with the dual learner yields further gains, including improvements of up to +5 accuracy points on LNT over prior SOTA. Ablation studies confirm that our proposed method remains robust under reduced replay frequency and small buffer size, demonstrating both effectiveness and sample efficiency. Taken together, our results establish replay as a strong baseline for continual LLM fine-tuning and demonstrate that surprise-based selection and slow-weight consolidation are complementary components for mitigating catastrophic forgetting.

翻译：持续学习，即在不遗忘先前已获得知识的前提下适应一系列任务的能力，仍然是机器学习领域的一项重大挑战，也是人工智能与人类智能之间的关键差距。尽管正则化与回放方法在视觉领域表现良好，但在大规模语言模型（LLMs）中，尤其是在涉及大量任务的大规模场景下，这些方法仍落后于多任务学习。我们重新审视回放机制，认为两种失效模式导致了这一差距：选择（应复习哪些内容）与整合（如何巩固新知识）。针对选择问题，我们提出了意外优先级回放（SuRe），这是一种简单且与架构无关的规则，用于对最意外（高负对数似然）的序列进行排序与存储。SuRe在“大量任务”（LNT）设定中实现了最先进的性能，并在标准持续学习与LNT基准测试中取得了最佳整体平均表现。针对整合问题，我们引入了一种双学习器设计，通过指数移动平均（EMA）合并快速与慢速LoRA适配器，从而实现快速适应并稳定长期知识。将SuRe与双学习器结合进一步提升了性能，包括在LNT上比先前最优方法提高多达+5个准确率百分点。消融研究证实，我们提出的方法在降低回放频率和减小缓冲区规模的情况下仍保持鲁棒性，证明了其有效性与样本效率。综上所述，我们的研究确立了回放作为持续LLM微调的强基线，并表明基于意外的选择与慢权重整合是缓解灾难性遗忘的互补组件。