Protein language models (pLMs) pre-trained on vast protein sequence databases excel at various downstream tasks but often lack the structural knowledge essential for some biological applications. To address this, we introduce a method to enrich pLMs with structural knowledge by leveraging pre-trained protein graph neural networks (pGNNs). First, a latent-level contrastive learning task aligns residue representations from pLMs with those from pGNNs across multiple proteins, injecting inter-protein structural information. Additionally, a physical-level task integrates intra-protein information by training pLMs to predict structure tokens. Together, the proposed dual-task framework effectively incorporates both inter- and intra-protein structural knowledge into pLMs. Given the variability in the quality of protein structures in PDB, we further introduce a residue loss selection module that uses a small model trained on high-quality structures to select reliable yet challenging residue losses for the pLM to learn. Applying our structure alignment method as a simple, lightweight post-training step to the state-of-the-art ESM2 and AMPLIFY yields notable performance gains. These improvements are consistent across a wide range of tasks, including substantial gains in deep mutational scanning (DMS) fitness prediction and a 59% increase in P@L for ESM2 650M contact prediction on CASP16. Furthermore, we demonstrate that these performance gains are robust, scaling with model sizes from 8M to 650M and extending to different downstream tasks.

翻译：蛋白质语言模型（pLMs）在大量蛋白质序列数据库上预训练后，在下游任务中表现出色，但往往缺乏某些生物学应用所必需的结构知识。为解决这一问题，我们提出了一种利用预训练的蛋白质图神经网络（pGNNs）来增强pLMs结构知识的方法。首先，通过潜在层面的对比学习任务，将pLMs与pGNNs在多个蛋白质中的残基表示对齐，从而注入蛋白质间的结构信息。此外，通过物理层面的任务，训练pLMs预测结构标记，以整合蛋白质内的信息。所提出的双任务框架有效结合了蛋白质间和蛋白质内的结构知识。鉴于PDB中蛋白质结构质量的变异性，我们进一步引入了一个残基损失选择模块，该模块使用在高质量结构上训练的小型模型，为pLM选择可靠且具有挑战性的残基损失进行学习。将我们的结构对齐方法作为简单、轻量级的后训练步骤应用于最先进的ESM2和AMPLIFY模型，取得了显著的性能提升。这些改进在广泛的任务中保持一致，包括在深度突变扫描（DMS）适应性预测中的显著增益，以及在CASP16上ESM2 650M接触预测的P@L提高了59%。此外，我们证明这些性能提升具有鲁棒性，可随模型规模从8M到650M扩展，并适用于不同的下游任务。