By introducing a small set of additional parameters, a probe learns to solve specific linguistic tasks (e.g., dependency parsing) in a supervised manner using feature representations (e.g., contextualized embeddings). The effectiveness of such probing tasks is taken as evidence that the pre-trained model encodes linguistic knowledge. However, this approach of evaluating a language model is undermined by the uncertainty of the amount of knowledge that is learned by the probe itself. Complementary to those works, we propose a parameter-free probing technique for analyzing pre-trained language models (e.g., BERT). Our method does not require direct supervision from the probing tasks, nor do we introduce additional parameters to the probing process. Our experiments on BERT show that syntactic trees recovered from BERT using our method are significantly better than linguistically-uninformed baselines. We further feed the empirically induced dependency structures into a downstream sentiment classification task and find its improvement compatible with or even superior to a human-designed dependency schema.

翻译：通过引入一小套额外的参数,一个探测器学会以监督的方式使用特征表现(例如背景化嵌入)解决特定语言任务(例如依赖分析),这种测试任务的效力被作为经过预先培训的模型将语言知识编码的证据。然而,这种评价语言模式的方法由于探测器本身所学知识数量的不确定性而受到损害。对这些工作的补充,我们提出一种无参数的测试技术,用于分析预先训练的语言模型(例如BERT)。我们的方法不需要直接监督测试任务,也不需要我们为测试进程引入额外的参数。我们对BERT的实验表明,使用我们的方法从BERT中回收的合成树比语言上不知情的基线要好得多。我们进一步将经验引导的依赖结构纳入下游情感分类任务,发现其改进与人设计的依赖计划相容,甚至优于人设计的依赖计划。