Memory-augmented neural networks (MANNs) can perform algorithmic tasks such as sorting. However, they often fail to generalise to input sequence lengths not encountered during training. We introduce two approaches that constrain the state space of the MANN's controller network: state compression and state regularisation. We empirically demonstrated that both approaches can improve generalisation to input sequences of out-of-distribution lengths for a specific type of MANN: the differentiable neural computer (DNC). The constrained DNC could process input sequences that were up to 2.3 times longer than those processed by an unconstrained baseline controller network. Notably, the applied constraints enabled the extension of the DNC's memory matrix without the need for retraining and thus allowed the processing of input sequences that were 10.4 times longer. However, the improvements were not consistent across all tested algorithmic tasks. Interestingly, solutions that performed better often had a highly structured state space, characterised by state trajectories exhibiting increased curvature and loop-like patterns. Our experimental work demonstrates that state-space constraints can enable the training of a DNC using shorter input sequences, thereby saving computational resources and facilitating training when acquiring long sequences is costly.

翻译：记忆增强神经网络（MANNs）能够执行排序等算法任务，但通常无法泛化至训练中未出现过的输入序列长度。我们提出了两种约束MANN控制器网络状态空间的方法：状态压缩与状态正则化。实验表明，这两种方法均能提升特定类型MANN——可微分神经计算机（DNC）——对分布外长度输入序列的泛化能力。经约束的DNC可处理的输入序列长度最高达到无约束基线控制器网络的2.3倍。值得注意的是，所施加的约束使得DNC无需重新训练即可扩展记忆矩阵，从而能够处理长度达10.4倍的输入序列。然而，这些改进并非在所有测试算法任务中均保持一致。有趣的是，表现更优的解决方案往往具有高度结构化的状态空间，其特征表现为状态轨迹呈现更高的曲率与环状模式。本实验工作表明，状态空间约束能够通过更短的输入序列训练DNC，从而节省计算资源，并在获取长序列成本较高时降低训练难度。