Understanding self-supervised learning is important but challenging. Previous theoretical works study the role of pretraining losses, and view neural networks as general black boxes. However, the recent work of Saunshi et al. argues that the model architecture -- a component largely ignored by previous works -- also has significant influences on the downstream performance of self-supervised learning. In this work, we provide the first theoretical analysis of self-supervised learning that incorporates the effect of inductive biases originating from the model class. In particular, we focus on contrastive learning -- a popular self-supervised learning method that is widely used in the vision domain. We show that when the model has limited capacity, contrastive representations would recover certain special clustering structures that are compatible with the model architecture, but ignore many other clustering structures in the data distribution. As a result, our theory can capture the more realistic setting where contrastive representations have much lower dimensionality than the number of clusters in the data distribution. We instantiate our theory on several synthetic data distributions, and provide empirical evidence to support the theory.

翻译：自我监督学习很重要,但具有挑战性。 以前的理论工作研究培训前损失的作用, 并将神经网络视为普通黑盒。 但是, Saunshi等人最近的工作认为, 模型结构 -- -- 先前的工作基本上忽视了这一组成部分 -- -- 也对自监督学习的下游表现产生了重大影响。 在这项工作中, 我们对自我监督学习的首次理论分析, 其中包括来自模型类的感应偏差的影响。 特别是, 我们侧重于对比学习 -- -- 一种在视觉领域广泛使用的流行的自我监督学习方法。 我们表明,当模型能力有限时, 对比性表述将恢复某些与模型结构相容的特殊集群结构, 但却忽视数据分配中许多其他集群结构。 因此, 我们的理论可以捕捉更现实的场景, 即对比性表达方式比数据分布中的组数要低得多。 我们将我们的理论放在几个合成数据分布上, 并提供经验证据来支持理论。