In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities. Subsequently, we propose the Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) by combining contrastive learning and masked data modeling, two major self-supervised learning frameworks, to learn a joint and coordinated audio-visual representation. Our experiments show that the contrastive audio-visual correspondence learning objective not only enables the model to perform audio-visual retrieval tasks, but also helps the model learn a better joint representation. As a result, our fully self-supervised pretrained CAV-MAE achieves a new SOTA accuracy of 65.9% on VGGSound, and is comparable with the previous best supervised pretrained model on AudioSet in the audio-visual event classification task. Code and pretrained models are at https://github.com/yuangongnd/cav-mae.

翻译：本文首先将最近的蒙特卡洛(MAE)模型从单一模态扩展到音视觉多模态。随后，我们将对比学习和数据蒙版建模两种重要的自监督学习框架相结合，提出了对比音视觉蒙特卡洛自编码器(CAV-MAE)，以学习一个联合和协调的音视频表示。我们的实验表明，对比音视频对应关系学习的目标不仅使模型能够执行音视频检索任务，而且有助于模型学习更好的联合表示。因此，我们完全自监督预训练的CAV-MAE在VGGSound上实现了新的SOTA准确性，达到了65.9％，在音视频事件分类任务上与以前最佳的监督预训练模型相比，具有可比性。代码和预训练模型位于 https://github.com/yuangongnd/cav-mae.