Visual speech (i.e., lip motion) is highly related to auditory speech due to the co-occurrence and synchronization in speech production. This paper investigates this correlation and proposes a cross-modal speech co-learning paradigm. The primary motivation of our cross-modal co-learning method is modeling one modality aided by exploiting knowledge from another modality. Specifically, two cross-modal boosters are introduced based on an audio-visual pseudo-siamese structure to learn the modality-transformed correlation. Inside each booster, a max-feature-map embedded Transformer variant is proposed for modality alignment and enhanced feature generation. The network is co-learned both from scratch and with pretrained models. Experimental results on the LRSLip3, GridLip, LomGridLip, and VoxLip datasets demonstrate that our proposed method achieves 60% and 20% average relative performance improvement over independently trained audio-only/visual-only and baseline fusion systems, respectively.

翻译：视觉演讲( 即, 唇动) 与听觉演讲高度相关, 原因是语音制作中同时出现和同步。 本文调查了这一相关性, 并提出了一个跨模式的语音共同学习模式。 我们的跨模式共同学习方法的主要动力是利用另一种模式的知识来模拟一种模式。 具体地说, 采用两种跨模式的助推器, 学习模式转换的关联。 在每个助推器中, 都为模式调整和增强功能生成提议了一种最大性能嵌入式变异器。 网络是从零到预培训模型共同学习的。 LRSLip3、 GridLip、 LombGridLip、 LombGridLip和VoxLip 数据集的实验结果显示, 我们的拟议方法分别实现了60%和20%的平均相对性改进, 而不是独立训练的只用视听和基线融合系统。