Large-scale vision-language pre-training has shown impressive advances in a wide range of downstream tasks. Existing methods mainly model the cross-modal alignment by the similarity of the global representations of images and texts, or advanced cross-modal attention upon image and text features. However, they fail to explicitly learn the fine-grained semantic alignment between visual regions and textual phrases, as only global image-text alignment information is available. In this paper, we introduce LOUPE, a fine-grained semantically aLigned visiOn-langUage PrE-training framework, which learns fine-grained semantic alignment from the novel perspective of game-theoretic interactions. To efficiently compute the game-theoretic interactions, we further propose an uncertainty-aware neural Shapley interaction learning module. Experiments show that LOUPE achieves state-of-the-art performance on a variety of vision-language tasks. Furthermore, without any object-level human annotations and fine-tuning, LOUPE achieves competitive performance on object detection and visual grounding. More importantly, LOUPE opens a new promising direction of learning fine-grained semantics from large-scale raw image-text pairs. The repository of this work is at https://github.com/YYJMJC/LOUPE.

翻译：大规模视觉语言培训前培训在一系列广泛的下游任务中取得了令人印象深刻的进展。现有方法主要以图像和文字全球表述的相似性或图像和文字特征的高级超时关注来模拟跨模式调整;然而,它们未能明确学习视觉区域和文字短语之间的细微语义调整,因为只有全球图像-文字校正信息才能得到。在本文中,我们引入了LOUPE,这是一个精美的语义比对On-langUage PrE培训框架,从游戏-理论互动的新视角中学习精细的语义调整。为高效地理解游戏-理论互动,我们进一步提议了一个具有不确定性的神经特征互动学习模块。实验显示,LOUPE在各种视觉语言任务中达到了最先进的表现。此外,没有目标级人类描述和微调,LOUPE在对象探测和视觉定位互动互动的新动作上取得了具有竞争力的成绩。更重要的是,LOUPUPE在大型空间/视觉地面图像中开启了一个新的方向。