Humans learn continually throughout their lifespan by accumulating diverse knowledge and fine-tuning it for future tasks. When presented with a similar goal, neural networks suffer from catastrophic forgetting if data distributions across sequential tasks are not stationary over the course of learning. An effective approach to address such continual learning (CL) problems is to use hypernetworks which generate task dependent weights for a target network. However, the continual learning performance of existing hypernetwork based approaches are affected by the assumption of independence of the weights across the layers in order to maintain parameter efficiency. To address this limitation, we propose a novel approach that uses a dependency preserving hypernetwork to generate weights for the target network while also maintaining the parameter efficiency. We propose to use recurrent neural network (RNN) based hypernetwork that can generate layer weights efficiently while allowing for dependencies across them. In addition, we propose novel regularisation and network growth techniques for the RNN based hypernetwork to further improve the continual learning performance. To demonstrate the effectiveness of the proposed methods, we conducted experiments on several image classification continual learning tasks and settings. We found that the proposed methods based on the RNN hypernetworks outperformed the baselines in all these CL settings and tasks.

翻译：人类在整个生命周期中不断学习,积累各种知识,并微调这些知识,以适应未来的任务。当提出类似的目标时,神经网络将遭受灾难性的忘记,如果相继任务的数据分配在学习过程中不是固定不变的。解决这种持续学习(CL)问题的有效办法是使用超网络,为目标网络产生任务依赖加权数。然而,现有超网络方法的持续学习表现受到跨层加权数独立假设的影响,以保持参数效率。为了应对这一限制,我们提出了一种新颖的办法,即利用依赖保护超网络来生成目标网络的加权数,同时保持参数效率。我们提议使用基于经常的神经网络(RNN)的超网络(RNN)来高效生成层加权数,同时允许它们之间的依赖性。此外,我们提议为基于RNN网络的超网络提供新的正规化和网络增长技术,以进一步提高持续学习业绩。为了证明拟议方法的有效性,我们进行了几项图像持续学习任务和设置环境的实验。我们发现,基于RNNT超网络的拟议方法超越了所有这些C设置的基线。