Advances in NLP have yielded impressive results for the task of machine reading comprehension (MRC), with approaches having been reported to achieve performance comparable to that of humans. In this paper, we investigate whether state-of-the-art MRC models are able to correctly process Semantics Altering Modifications (SAM): linguistically-motivated phenomena that alter the semantics of a sentence while preserving most of its lexical surface form. We present a method to automatically generate and align challenge sets featuring original and altered examples. We further propose a novel evaluation methodology to correctly assess the capability of MRC systems to process these examples independent of the data they were optimised on, by discounting for effects introduced by domain shift. In a large-scale empirical study, we apply the methodology in order to evaluate extractive MRC models with regard to their capability to correctly process SAM-enriched data. We comprehensively cover 12 different state-of-the-art neural architecture configurations and four training datasets and find that -- despite their well-known remarkable performance -- optimised models consistently struggle to correctly process semantically altered data.

翻译：机器阅读理解(MRC)任务的进展令人印象深刻,据报告,在机器阅读理解(MRC)任务方面,我们取得了令人印象深刻的成果,并采取了与人类相似的绩效。在本文件中,我们调查了最先进的MRC模型是否能够正确处理语义变换改变(SAM):改变句子语义的由语言驱动的现象,同时保留其大部分地表法形式。我们提出了一个自动生成和调整以原始和变换实例为特点的成套挑战的方法。我们进一步提出一种新的评估方法,以正确评估MRC系统处理这些例子的能力,而这种能力独立于它们所选取的数据。在一项大规模的经验研究中,我们应用这一方法来评价采掘MRC模型正确处理SAM丰富数据的能力。我们全面覆盖了12个不同的状态的神经结构配置和4个培训数据集。我们发现,尽管它们有众所周知的显著性能,但最优化的模型始终在努力争取正确处理过程语义变数据。