Vision transformers (ViTs) have been an alternative design paradigm to convolutional neural networks (CNNs). However, the training of ViTs is much harder than CNNs, as it is sensitive to the training parameters, such as learning rate, optimizer and warmup epoch. The reasons for training difficulty are empirically analysed in ~\cite{xiao2021early}, and the authors conjecture that the issue lies with the \textit{patchify-stem} of ViT models and propose that early convolutions help transformers see better. In this paper, we further investigate this problem and extend the above conclusion: only early convolutions do not help for stable training, but the scaled ReLU operation in the \textit{convolutional stem} (\textit{conv-stem}) matters. We verify, both theoretically and empirically, that scaled ReLU in \textit{conv-stem} not only improves training stabilization, but also increases the diversity of patch tokens, thus boosting peak performance with a large margin via adding few parameters and flops. In addition, extensive experiments are conducted to demonstrate that previous ViTs are far from being well trained, further showing that ViTs have great potential to be a better substitute of CNNs.

翻译：视觉变压器(ViTs)是革命神经网络(CNNs)的替代设计范式。然而,ViTs的训练比CNN公司要难得多,因为它对学习率、优化率和暖化度等培训参数敏感。培训困难的原因在“cite{xiao2021early}”中进行了经验分析,作者们推测,这个问题不仅在于ViT模型的Textit{patchfatchify-stem},并且建议早期的革命有助于变压器看得更好。在本文中,我们进一步调查了这一问题,并扩大了上述结论:只有早期的革命无助于稳定培训,而对于在\textit{conv-stem} (\textit{conv-stem})中扩大的ReLU操作也很重要。我们从理论上和实验角度都证实,在ViT模型中扩大的ReLU规模不仅改善了培训稳定性,而且还增加了补装品的多样性,因此,通过经过培训的参数和软质变的远的模型来提升了巨大的优势。