In this paper, we explore an important yet previously neglected question: Do context aggregation patterns across Language Models (LMs) share commonalities? While some works have investigated context aggregation or attention weights in LMs, they typically focus on individual models or attention heads, lacking a systematic analysis across multiple LMs to explore their commonalities. In contrast, we focus on the commonalities among LMs, which can deepen our understanding of LMs and even facilitate cross-model knowledge transfer. In this work, we introduce the Order-Level Attention (OLA) derived from the order-wise decomposition of Attention Rollout and reveal that the OLA at the same order across LMs exhibits significant similarities. Furthermore, we discover an implicit mapping between OLA and syntactic knowledge. Based on these two findings, we propose the Transferable OLA Adapter (TOA), a training-free cross-LM adapter transfer method. Specifically, we treat the OLA as a unified syntactic feature representation and train an adapter that takes OLA as input. Due to the similarities in OLA across LMs, the adapter generalizes to unseen LMs without requiring any parameter updates. Extensive experiments demonstrate that TOA's cross-LM generalization effectively enhances the performance of unseen LMs. Code is available at https://github.com/jinglin-liang/OLAS.

翻译：本文探讨了一个重要但先前被忽视的问题：语言模型（LMs）间的上下文聚合模式是否具有共性？尽管已有研究关注语言模型中的上下文聚合或注意力权重，但它们通常聚焦于单个模型或注意力头，缺乏跨多个语言模型的系统性分析以探索其共性。相比之下，我们关注语言模型间的共性，这能深化我们对语言模型的理解，甚至促进跨模型知识迁移。在本工作中，我们引入了从注意力展开的阶次分解中导出的阶次注意力（OLA），并揭示了不同语言模型间相同阶次的OLA表现出显著相似性。此外，我们发现OLA与句法知识之间存在隐式映射关系。基于这两项发现，我们提出了可迁移的OLA适配器（TOA），一种无需训练的跨语言模型适配器迁移方法。具体而言，我们将OLA视为统一的句法特征表示，并训练一个以OLA为输入的适配器。由于不同语言模型间OLA的相似性，该适配器能够泛化至未见过的语言模型，无需任何参数更新。大量实验表明，TOA的跨语言模型泛化能力有效提升了未见语言模型的性能。代码发布于https://github.com/jinglin-liang/OLAS。