Forgetting is often seen as an unwanted characteristic in both human and machine learning. However, we propose that forgetting can in fact be favorable to learning. We introduce "forget-and-relearn" as a powerful paradigm for shaping the learning trajectories of artificial neural networks. In this process, the forgetting step selectively removes undesirable information from the model, and the relearning step reinforces features that are consistently useful under different conditions. The forget-and-relearn framework unifies many existing iterative training algorithms in the image classification and language emergence literature, and allows us to understand the success of these algorithms in terms of the disproportionate forgetting of undesirable information. We leverage this understanding to improve upon existing algorithms by designing more targeted forgetting operations. Insights from our analysis provide a coherent view on the dynamics of iterative training in neural networks and offer a clear path towards performance improvements.

翻译：在人类和机器学习中,人们往往认为忘却是一种不必要的特征。然而,我们提议,忘却实际上会有利于学习。我们引入了“忘记和重复”作为塑造人工神经网络学习轨迹的强大范例。在这个过程中,忘却的步骤有选择地从模型中删除了不受欢迎的信息,再学习的步骤强化了在不同条件下始终有用的特征。忘却和再学习的框架将许多现有的迭代培训算法统一在图像分类和语言生成文学中,并使我们能够理解这些算法的成功之处,即过度忘记不良信息。我们利用这一理解来改进现有的算法,设计更有针对性的忘却操作。我们的分析从中得出关于神经网络互动培训动态的一致观点,并为改进业绩提供了一条明确的道路。