Abstraction is a desirable capability for deep learning models, which means to induce abstract concepts from concrete instances and flexibly apply them beyond the learning context. At the same time, there is a lack of clear understanding about both the presence and further characteristics of this capability in deep learning models. In this paper, we introduce a systematic probing framework to explore the abstraction capability of deep learning models from a transferability perspective. A set of controlled experiments are conducted based on this framework, providing strong evidence that two probed pre-trained language models (PLMs), T5 and GPT2, have the abstraction capability. We also conduct in-depth analysis, thus shedding further light: (1) the whole training phase exhibits a "memorize-then-abstract" two-stage process; (2) the learned abstract concepts are gathered in a few middle-layer attention heads, rather than being evenly distributed throughout the model; (3) the probed abstraction capabilities exhibit robustness against concept mutations, and are more robust to low-level/source-side mutations than high-level/target-side ones; (4) generic pre-training is critical to the emergence of abstraction capability, and PLMs exhibit better abstraction with larger model sizes and data scales.

翻译：深层次学习模型的抽象能力是深层次学习模型的可取能力,这意味着从具体实例中引出抽象概念,并在学习背景之外灵活应用这些概念。与此同时,深层次学习模型对这一能力的存在和进一步特点缺乏明确了解。在本文中,我们引入了一个系统的探索框架,从可转移的角度探索深层次学习模型的抽象能力。基于这一框架,进行了一套受控制的实验,提供了强有力的证据,证明两个经过检测的预先培训的语言模型(PLM、T5和GPT2)具有抽象能力。我们还进行了深入分析,从而进一步深化:(1)整个培训阶段展示了“模范-当时-抽象”的两阶段进程;(2) 学到的抽象概念汇集在几个中层关注层,而不是在整个模型中均匀分布;(3) 探索的抽象能力显示对概念突变的强健性,比高层次/目标方更强;(4) 通用培训前对于抽象能力、规模更大和PLMS和图像的模型更大规模形成至关重要。