How related are the representations learned by neural language models, translation models, and language tagging tasks? We answer this question by adapting an encoder-decoder transfer learning method from computer vision to investigate the structure among 100 different feature spaces extracted from hidden representations of various networks trained on language tasks. This method reveals a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We call this low-dimensional structure a language representation embedding because it encodes the relationships between representations needed to process language for a variety of NLP tasks. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. Additionally, we find that the principal dimension of this structure can be used to create a metric which highlights the brain's natural language processing hierarchy. This suggests that the embedding captures some part of the brain's natural language representation structure.

翻译：神经语言模型、翻译模型以及语言标注任务所学习的表示之间具有怎样的关联性？为回答此问题，我们借鉴计算机视觉领域的编码器-解码器迁移学习方法，对从多种语言任务训练的网络隐藏表示中提取的100个不同特征空间进行结构分析。该方法揭示出一种低维结构：语言模型与翻译模型在词嵌入、句法与语义任务以及未来词嵌入之间实现平滑插值。我们将此低维结构称为语言表示嵌入，因其编码了处理各类自然语言处理任务所需的表示间关系。研究发现，该表示嵌入能够预测每个独立特征空间与通过功能磁共振成像记录的人类大脑对自然语言刺激响应的映射程度。此外，该结构的主成分维度可用于构建一种度量指标，该指标能够凸显大脑自然语言处理的层级结构。这表明该嵌入捕获了大脑自然语言表示结构的某些特征。