Contrastive learning has recently seen tremendous success in self-supervised learning. So far, however, it is largely unclear why the learned representations generalize so effectively to a large variety of downstream tasks. We here prove that feedforward models trained with objectives belonging to the commonly used InfoNCE family learn to implicitly invert the underlying generative model of the observed data. While the proofs make certain statistical assumptions about the generative model, we observe empirically that our findings hold even if these assumptions are severely violated. Our theory highlights a fundamental connection between contrastive learning, generative modeling, and nonlinear independent component analysis, thereby furthering our understanding of the learned representations as well as providing a theoretical foundation to derive more effective contrastive losses.

翻译：最近,在自我监督的学习中,反向学习取得了巨大的成功。然而,迄今为止,人们所了解的表述在很大程度上还不清楚为什么如此有效地概括了大量的下游任务。我们在这里证明,经过培训的具有通用的InfoNCE家庭目标的进食模式学会隐含地颠倒了所观察到的数据的基本基因模型。虽然这些证据对基因模型作了某些统计假设,但我们从经验上观察,即使这些假设遭到严重违反,我们的调查结果仍然有说服力。我们的理论强调了对比学习、基因模型和非线性独立组成部分分析之间的根本联系,从而加深了我们对所了解的进食模型的理解,并为产生更有效的对比损失提供了理论基础。