In recent years, increasingly large models have achieved outstanding performance across CV tasks. However, these models demand substantial computational resources and storage, and their growing complexity limits our understanding of how they make decisions. Most of these architectures rely on the attention mechanism within Transformer-based designs. Building upon the connection between residual neural networks and ordinary differential equations (ODEs), we introduce ODE-ViT, a Vision Transformer reformulated as an ODE system that satisfies the conditions for well-posed and stable dynamics. Experiments on CIFAR-10 and CIFAR-100 demonstrate that ODE-ViT achieves stable, interpretable, and competitive performance with up to one order of magnitude fewer parameters, surpassing prior ODE-based Transformer approaches in classification tasks. We further propose a plug-and-play teacher-student framework in which a discrete ViT guides the continuous trajectory of ODE-ViT by treating the intermediate representations of the teacher as solutions of the ODE. This strategy improves performance by more than 10% compared to training a free ODE-ViT from scratch.

翻译：近年来，日益增大的模型在计算机视觉任务中取得了卓越的性能。然而，这些模型需要大量的计算资源和存储空间，且其不断增长的复杂性限制了我们理解其决策机制。大多数此类架构依赖于基于Transformer设计中的注意力机制。基于残差神经网络与常微分方程（ODEs）之间的联系，我们提出了ODE-ViT——一种被重构为满足适定性和稳定性动力学条件的ODE系统的视觉Transformer。在CIFAR-10和CIFAR-100数据集上的实验表明，ODE-ViT以最多减少一个数量级的参数量，实现了稳定、可解释且具有竞争力的性能，在分类任务中超越了先前基于ODE的Transformer方法。我们进一步提出了一种可插拔的师生框架，其中离散ViT通过将教师的中间表示视为ODE的解来指导ODE-ViT的连续轨迹。与从头训练自由ODE-ViT相比，该策略将性能提升了超过10%。