While self-supervised representation learning (SSL) has proved to be effective in the large model, there is still a huge gap between the SSL and supervised method in the lightweight model when following the same solution. We delve into this problem and find that the lightweight model is prone to collapse in semantic space when simply performing instance-wise contrast. To address this issue, we propose a relation-wise contrastive paradigm with Relation Knowledge Distillation (ReKD). We introduce a heterogeneous teacher to explicitly mine the semantic information and transferring a novel relation knowledge to the student (lightweight model). The theoretical analysis supports our main concern about instance-wise contrast and verify the effectiveness of our relation-wise contrastive learning. Extensive experimental results also demonstrate that our method achieves significant improvements on multiple lightweight models. Particularly, the linear evaluation on AlexNet obviously improves the current state-of-art from 44.7% to 50.1%, which is the first work to get close to the supervised 50.5%. Code will be made available.

翻译：虽然自我监督的代表学习(SSL)在大型模型中证明是有效的,但在采用同一解决方案时,SSL与轻量级模型中受监督的方法之间仍然存在着巨大的差距。 我们深入研究这一问题,发现轻量级模型在仅仅进行实例对比时容易在语义空间中崩溃。 为解决这一问题,我们提出了一个与 " 关系知识蒸馏 " (ReKD)相对的对比模式。我们引入了一位异质教师,以明确清除语义信息,向学生传授新的关系知识(轻量级模型)。理论分析支持了我们对实例对比的主要关切,并验证了我们从关系角度进行对比学习的有效性。广泛的实验结果还表明,我们的方法在多个轻量级模型上也取得了显著的改进。特别是,对亚历克斯网的线性评估显然将当前的艺术状态从44.7%提高到50.1%,这是接近50.5%的首项工作。将提供代码。