Entity disambiguation (ED) is the last step of entity linking (EL), when candidate entities are reranked according to the context they appear in. All datasets for training and evaluating models for EL consist of convenience samples, such as news articles and tweets, that propagate the prior probability bias of the entity distribution towards more frequently occurring entities. It was previously shown that the performance of the EL systems on such datasets is overestimated since it is possible to obtain higher accuracy scores by merely learning the prior. To provide a more adequate evaluation benchmark, we introduce the ShadowLink dataset, which includes 16K short text snippets annotated with entity mentions. We evaluate and report the performance of popular EL systems on the ShadowLink benchmark. The results show a considerable difference in accuracy between more and less common entities for all of the EL systems under evaluation, demonstrating the effects of prior probability bias and entity overshadowing.

翻译：实体偏差(ED)是实体联系的最后一步,当候选实体根据它们所处的背景重新排序时,实体的偏差(EL)是实体联系的最后一步。所有用于培训和评价EL模型的数据集都包含方便样本,例如新闻文章和推特,这些样本传播了实体先前向更频繁发生实体分布的概率偏差;以前曾显示,由于仅通过学习之前的学习就可以获得更高的准确度分数,EL系统在这类数据集上的性能被高估过高。为了提供更充分的评估基准,我们引入了“阴影链接”数据集,其中包括16K短文本,与实体一起附加注释的16K条文字片段。我们评估和报告在阴影链接基准上流行的EL系统的性能。结果显示,评价中的所有EL系统的通用实体在准确性上存在相当大的差异,表明先前概率偏差和实体蒙上阴影的影响。