A prominent technique for self-supervised representation learning has been to contrast semantically similar and dissimilar pairs of samples. Without access to labels, dissimilar (negative) points are typically taken to be randomly sampled datapoints, implicitly accepting that these points may, in reality, actually have the same label. Perhaps unsurprisingly, we observe that sampling negative examples from truly different labels improves performance, in a synthetic setting where labels are available. Motivated by this observation, we develop a debiased contrastive objective that corrects for the sampling of same-label datapoints, even without knowledge of the true labels. Empirically, the proposed objective consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks. Theoretically, we establish generalization bounds for the downstream classification task.

翻译：自我监督代表制学习的一个突出技术是对比结构相似和不同的样本。 由于没有标签,不同(消极)点通常被认为是随机抽样的数据点,暗示这些点实际上可能具有相同的标签。 也许不奇怪的是,我们观察到,从真正不同标签中抽样的负面例子在有标签的合成环境中提高了性能。受这一观察的驱动,我们制定了一种扭曲的对比性目标,纠正同一标签数据点的抽样,即使不知道真实的标签。 典型的是,拟议目标始终超越了在视觉、语言和强化学习基准方面进行代表性学习的最先进的标准。理论上,我们为下游分类任务建立了通用的界限。