Large Language Models increasingly possess capabilities that carry dual-use risks. While data filtering has emerged as a pretraining-time mitigation, it faces significant challenges: labeling whether data is harmful is expensive at scale, and given improving sample efficiency with larger models, even small amounts of mislabeled content could give rise to dangerous capabilities. To address risks associated with mislabeled harmful content, prior work proposed Gradient Routing (Cloud et al., 2024) -- a technique that localizes target knowledge into a dedicated subset of model parameters so they can later be removed. We explore an improved variant of Gradient Routing, which we call Selective GradienT Masking (SGTM), with particular focus on evaluating its robustness to label noise. SGTM zero-masks selected gradients such that target domain examples only update their dedicated parameters. We test SGTM's effectiveness in two applications: removing knowledge of one language from a model trained on a bilingual synthetic dataset, and removing biology knowledge from a model trained on English Wikipedia. In both cases SGTM provides better retain/forget trade-off in the presence of labeling errors compared to both data filtering and a previously proposed instantiation of Gradient Routing. Unlike shallow unlearning approaches that can be quickly undone through fine-tuning, SGTM exhibits strong robustness to adversarial fine-tuning, requiring seven times more fine-tuning steps to reach baseline performance on the forget set compared to a finetuning-based unlearning method (RMU). Our results suggest SGTM provides a promising pretraining-time complement to existing safety mitigations, particularly in settings where label noise is unavoidable.

翻译：大语言模型日益具备可能带来双重用途风险的能力。尽管数据过滤已成为预训练阶段的缓解手段，但其面临显著挑战：大规模标注数据是否具有危害性成本高昂，且随着模型规模增大带来的样本效率提升，即使少量错误标注的内容也可能催生危险能力。为应对错误标注有害内容相关的风险，先前研究提出了梯度路由技术（Cloud等人，2024）——该方法将目标知识定位至模型参数的专用子集中，以便后续移除。我们探索了一种改进的梯度路由变体，称为选择性梯度掩码，重点评估其对标签噪声的鲁棒性。该方法通过零掩码选定梯度，使得目标领域样本仅更新其专用参数。我们在两个应用中测试SGTM的有效性：从双语合成数据集训练的模型中移除特定语言知识，以及从英文维基百科训练的模型中移除生物学知识。两种场景下，相较于数据过滤和先前提出的梯度路由实现，SGTM在存在标注错误时均展现出更优的知识保留/遗忘权衡特性。与可通过微调快速逆转的浅层遗忘方法不同，SGTM对对抗性微调表现出强鲁棒性——相较于基于微调的遗忘方法，其在遗忘集上达到基线性能所需微调步数增加七倍。研究结果表明，SGTM为现有安全缓解措施提供了有前景的预训练阶段补充方案，尤其在标签噪声不可避免的场景中具有应用价值。