Several studies have been carried out on revealing linguistic features captured by BERT. This is usually achieved by training a diagnostic classifier on the representations obtained from different layers of BERT. The subsequent classification accuracy is then interpreted as the ability of the model in encoding the corresponding linguistic property. Despite providing insights, these studies have left out the potential role of token representations. In this paper, we provide a more in-depth analysis on the representation space of BERT in search for distinct and meaningful subspaces that can explain the reasons behind these probing results. Based on a set of probing tasks and with the help of attribution methods we show that BERT tends to encode meaningful knowledge in specific token representations (which are often ignored in standard classification setups), allowing the model to detect syntactic and semantic abnormalities, and to distinctively separate grammatical number and tense subspaces.

翻译：对BERT所捕捉的语言特征进行了一些研究,通常通过培训诊断分类员了解从BERT不同层次获得的表述方式来实现。随后的分类准确性被解释为该模型在编码相应的语言属性方面的能力。这些研究尽管提供了深刻的见解,但遗漏了象征性表述方式的潜在作用。在本文件中,我们更深入分析了BERT在寻找独特和有意义的子空间以解释这些检验结果背后的原因方面的代表空间。基于一套调查任务,并在归属方法的帮助下,我们表明,BERT倾向于将有意义的知识纳入具体的象征性表述方式(在标准的分类设置中常常被忽略),从而使得模型能够检测合成和语义异常,并允许有区别地区分的语法数字和紧张的子空间。