We characterize how memorization is represented in transformer models and show that it can be disentangled in the weights of both language models (LMs) and vision transformers (ViTs) using a decomposition based on the loss landscape curvature. This insight is based on prior theoretical and empirical work showing that the curvature for memorized training points is much sharper than non memorized, meaning ordering weight components from high to low curvature can reveal a distinction without explicit labels. This motivates a weight editing procedure that suppresses far more recitation of untargeted memorized data more effectively than a recent unlearning method (BalancedSubnet), while maintaining lower perplexity. Since the basis of curvature has a natural interpretation for shared structure in model weights, we analyze the editing procedure extensively on its effect on downstream tasks in LMs, and find that fact retrieval and arithmetic are specifically and consistently negatively affected, even though open book fact retrieval and general logical reasoning is conserved. We posit these tasks rely heavily on specialized directions in weight space rather than general purpose mechanisms, regardless of whether those individual datapoints are memorized. We support this by showing a correspondence between task data's activation strength with low curvature components that we edit out, and the drop in task performance after the edit. Our work enhances the understanding of memorization in neural networks with practical applications towards removing it, and provides evidence for idiosyncratic, narrowly-used structures involved in solving tasks like math and fact retrieval.

翻译：我们刻画了Transformer模型中记忆的表示方式，并表明基于损失景观曲率的分解方法可在语言模型（LMs）和视觉Transformer（ViTs）的权重中解耦记忆成分。这一洞见基于先前的理论与实证研究，表明被记忆训练点的曲率远高于非记忆点，这意味着按曲率从高到低排序权重分量可在无需显式标签的情况下揭示其区别。这启发了一种权重编辑方法，相比近期的一种遗忘方法（BalancedSubnet），该方法能更有效地抑制非目标记忆数据的复现，同时保持较低的困惑度。由于曲率基对模型权重中的共享结构具有自然解释力，我们在语言模型上广泛分析了编辑过程对下游任务的影响，发现事实检索和算术任务受到特定且一致的负面影响，尽管开放域事实检索和一般逻辑推理能力得以保留。我们认为这些任务高度依赖于权重空间中的专用方向，而非通用机制，无论这些独立数据点是否被记忆。我们通过展示任务数据在编辑掉的低曲率分量上的激活强度与编辑后任务性能下降之间的对应关系，为此提供了支持。我们的研究增强了对神经网络中记忆机制的理解，并为实际应用中移除记忆提供了方法，同时为数学和事实检索等任务所依赖的特异性、窄用途结构提供了证据。