NLP has a rich history of representing our prior understanding of language in the form of graphs. Recent work on analyzing contextualized text representations has focused on hand-designed probe models to understand how and to what extent do these representations encode a particular linguistic phenomenon. However, due to the inter-dependence of various phenomena and randomness of training probe models, detecting how these representations encode the rich information in these linguistic graphs remains a challenging problem. In this paper, we propose a new information-theoretic probe, Bird's Eye, which is a fairly simple probe method for detecting if and how these representations encode the information in these linguistic graphs. Instead of using classifier performance, our probe takes an information-theoretic view of probing and estimates the mutual information between the linguistic graph embedded in a continuous space and the contextualized word representations. Furthermore, we also propose an approach to use our probe to investigate localized linguistic information in the linguistic graphs using perturbation analysis. We call this probing setup Worm's Eye. Using these probes, we analyze BERT models on their ability to encode a syntactic and a semantic graph structure, and find that these models encode to some degree both syntactic as well as semantic information; albeit syntactic information to a greater extent.

翻译：NLP在以图表形式代表我们先前对语言的理解方面有着丰富的历史。最近关于分析背景化文本表达方式的工作侧重于手动设计的探测模型,以了解这些表达方式如何和在何种程度上对特定语言现象进行编码。然而,由于各种现象的相互依存性以及培训探测模型的随机性,发现这些表达方式如何将语言图中的丰富信息编码成这些语言图中的问题仍是一个具有挑战性的问题。在本文件中,我们建议使用一个新的信息理论探测器,即Bird's Eye,这是一个相当简单的探测方法,用来检测这些表达方式是否和如何对这些语言图中的信息进行编码。我们的探测器不是使用分类仪的性能,而是对连续空间内嵌入的语言图与背景化的表达方式之间的相互信息进行研究和估计。此外,我们还建议采用一种方法,利用我们的探测方法来调查语言图中本地化的语言信息。我们称之为这个探测设置线眼。我们用这些探测器,用这些探测器来分析这些模型是否有能力将合成合成方法和精密度的图像结构结构,我们用这些模型来分析这些模型作为较深层次的图表结构。