A standard assumption in deep learning is that the inductive bias introduced by a neural network architecture must persist from training through inference. The architecture you train with is the architecture you deploy. This assumption constrains the community from selecting architectures that may have desirable efficiency or design properties due to difficulties with optimization. We challenge this assumption with Network of Theseus (NoT), a method for progressively converting a trained, or even untrained, guide network architecture part-by-part into an entirely different target network architecture while preserving the performance of the guide network. At each stage, components in the guide network architecture are incrementally replaced with target architecture modules and aligned via representational similarity metrics. This procedure largely preserves the functionality of the guide network even under substantial architectural changes-for example, converting a convolutional network into a multilayer perceptron, or GPT-2 into a recurrent neural network. By decoupling optimization from deployment, NoT expands the space of viable inference-time architectures, opening opportunities for better accuracy-efficiency tradeoffs and enabling more directed exploration of the architectural design space.

翻译：深度学习中的一个标准假设是：神经网络架构引入的归纳偏置必须从训练到推理持续保持不变。训练时使用的架构即是部署时使用的架构。这一假设限制了研究社区选择那些可能因优化困难而具有理想效率或设计特性的架构。我们通过忒修斯网络（Network of Theseus, NoT）挑战这一假设，该方法能够逐步将已训练（甚至未训练）的引导网络架构逐部分转换为完全不同的目标网络架构，同时保持引导网络的性能。在每个阶段，引导网络架构中的组件被逐步替换为目标架构模块，并通过表示相似性度量进行对齐。这一过程在很大程度上保留了引导网络的功能性，即使在进行重大架构变更时——例如将卷积网络转换为多层感知机，或将GPT-2转换为循环神经网络。通过将优化与部署解耦，NoT扩展了可行的推理时架构空间，为更优的精度-效率权衡提供了机会，并支持对架构设计空间进行更定向的探索。