Masked image modeling (MIM) learns representations with remarkably good fine-tuning performances, overshadowing previous prevalent pre-training approaches such as image classification, instance contrastive learning, and image-text alignment. In this paper, we show that the inferior fine-tuning performance of these pre-training approaches can be significantly improved by a simple post-processing in the form of feature distillation (FD). The feature distillation converts the old representations to new representations that have a few desirable properties just like those representations produced by MIM. These properties, which we aggregately refer to as optimization friendliness, are identified and analyzed by a set of attention- and optimization-related diagnosis tools. With these properties, the new representations show strong fine-tuning performance. Specifically, the contrastive self-supervised learning methods are made as competitive in fine-tuning as the state-of-the-art masked image modeling (MIM) algorithms. The CLIP models' fine-tuning performance is also significantly improved, with a CLIP ViT-L model reaching 89.0% top-1 accuracy on ImageNet-1K classification. On the 3-billion-parameter SwinV2-G model, the fine-tuning accuracy is improved by +1.5 mIoU / +1.1 mAP to 61.4 mIoU / 64.2 mAP on ADE20K semantic segmentation and COCO object detection, respectively, creating new records on both benchmarks. More importantly, our work provides a way for the future research to focus more effort on the generality and scalability of the learnt representations without being pre-occupied with optimization friendliness since it can be enhanced rather easily. The code will be available at https://github.com/SwinTransformer/Feature-Distillation.

翻译：蒙面图像建模( MIM) 学习以非常优美的微调性能来显示旧的表示式, 并学习非常优美的性能。 这些属性, 我们统称为优化友好度, 被一组关注和优化相关诊断工具所识别和分析。 有了这些属性, 新的表示式显示了这些培训前方法的微调性能。 具体地说, 以特性蒸馏( FD) 的形式进行简单的后处理, 可以大大改进这些微调性能。 特性蒸馏将旧的表示式转换为新的表示式, 与 MIM 生成的表示式相似。 这些属性, 我们统称为优化友好度, 由一组关注和优化相关诊断工具来识别和分析。 有了这些属性, 新表示式的微调性性能表现式, 具体地说, 对比式自我超强的学习方法, 与状态的蒙面图像建模模型( MIM) 相比, 还可以大大改进 CLIP ViT-L 调整性能性能性能, 在图像网络1- k 的精确度上达到89. 0; 在310- 的检测记录上, 改进S1.5- AS- 的Smaryal- adrodustryalalalalal 工作上, am- am- labalalalalal labal labal labal labal ladal mado mm) labal 。