Contextual synonym knowledge is crucial for those similarity-oriented tasks whose core challenge lies in capturing semantic similarity between entities in their contexts, such as entity linking and entity matching. However, most Pre-trained Language Models (PLMs) lack synonym knowledge due to inherent limitations of their pre-training objectives such as masked language modeling (MLM). Existing works which inject synonym knowledge into PLMs often suffer from two severe problems: (i) Neglecting the ambiguity of synonyms, and (ii) Undermining semantic understanding of original PLMs, which is caused by inconsistency between the exact semantic similarity of the synonyms and the broad conceptual relevance learned from the original corpus. To address these issues, we propose PICSO, a flexible framework that supports the injection of contextual synonym knowledge from multiple domains into PLMs via a novel entity-aware Adapter which focuses on the semantics of the entities (synonyms) in the contexts. Meanwhile, PICSO stores the synonym knowledge in additional parameters of the Adapter structure, which prevents it from corrupting the semantic understanding of the original PLM. Extensive experiments demonstrate that PICSO can dramatically outperform the original PLMs and the other knowledge and synonym injection models on four different similarity-oriented tasks. In addition, experiments on GLUE prove that PICSO also benefits general natural language understanding tasks. Codes and data will be public.

翻译：将同义词知识注入PLM(MLM)等类似的任务,其核心挑战在于获取实体联系和实体匹配等实体背景的语义相似性;然而,由于培训前语言模型(PLM)在培训前的目标(如隐形语言建模(MLM)等)方面的固有局限性,大多数先行语言模型(PLM)缺乏同义性知识。 将同义词知识注入PLMS的现有工作往往有两个严重问题:(一) 忽视同义词的模糊性,以及(二) 挖掘原始PLMs对语义的语义理解,这是由于这些同义的确切相似性与从原始材料中学到的广泛概念相关性不一致造成的。为解决这些问题,我们建议PICSO(PICSO)是一个灵活的框架,支持将多种领域的相通性同义词知识注入PLMSM(同义性语言),重点是实体(同义语言)的语义学,同时,PICSO(同义性)将同义知识储存在适应性模型的更多参数上,使PLMSLMSOFM的原始实验能展示其他原始的原始数据。