Transformers recently are adapted from the community of natural language processing as a promising substitute of convolution-based neural networks for visual learning tasks. However, its supremacy degenerates given an insufficient amount of training data (e.g., ImageNet). To make it into practical utility, we propose a novel distillation-based method to train vision transformers. Unlike previous works, where merely heavy convolution-based teachers are provided, we introduce lightweight teachers with different architectural inductive biases (e.g., convolution and involution) to co-advise the student transformer. The key is that teachers with different inductive biases attain different knowledge despite that they are trained on the same dataset, and such different knowledge compounds and boosts the student's performance during distillation. Equipped with this cross inductive bias distillation method, our vision transformers (termed as CivT) outperform all previous transformers of the same architecture on ImageNet.

翻译：最近从自然语言处理社区改编的变异器最近被改编为以自然语言处理社区为视觉学习任务的有希望的替代以革命为基础的神经网络。然而,由于培训数据(如图像网络)数量不足,其至高无上的地位就退化了。为了使它成为实用的实用性,我们提议了一种基于蒸馏的新颖方法来培训视觉变异器。与以前只提供重革命教师的工程不同,我们引入了具有不同建筑感官偏见(如混凝土和进化)的轻质教师来共同咨询学生变异器。关键是,尽管有不同感官偏见的教师接受过相同的数据集培训,但获得不同的知识,以及这种不同的知识化合物和提升学生在蒸馏过程中的成绩。用这种交叉感带偏见蒸馏法将我们的视觉变异器(称为CivT)超越了图像网络上所有先前的变异器。