We present GRAPE (Group RepresentAtional Position Encoding), a unified framework for positional encoding based on group actions. GRAPE brings together two families of mechanisms: (i) multiplicative rotations (Multiplicative GRAPE) in $\mathrm{SO}(d)$ and (ii) additive logit biases (Additive GRAPE) arising from unipotent actions in the general linear group $\mathrm{GL}$. In Multiplicative GRAPE, a position $n \in \mathbb{Z}$ (or $t \in \mathbb{R}$) acts as $\mathbf{G}(n)=\exp(n\,ω\,\mathbf{L})$ with a rank-2 skew generator $\mathbf{L} \in \mathbb{R}^{d \times d}$, yielding a relative, compositional, norm-preserving map with a closed-form matrix exponential. RoPE is recovered exactly when the $d/2$ planes are the canonical coordinate pairs with log-uniform spectrum. Learned commuting subspaces and compact non-commuting mixtures strictly extend this geometry to capture cross-subspace feature coupling at $O(d)$ and $O(r d)$ cost per head, respectively. In Additive GRAPE, additive logits arise as rank-1 (or low-rank) unipotent actions, recovering ALiBi and the Forgetting Transformer (FoX) as exact special cases while preserving an exact relative law and streaming cacheability. Altogether, GRAPE supplies a principled design space for positional geometry in long-context models, subsuming RoPE and ALiBi as special cases. Project Page: https://github.com/model-architectures/GRAPE.

翻译：我们提出了GRAPE（Group RepresentAtional Position Encoding，群表示位置编码），一个基于群作用的统一位置编码框架。GRAPE整合了两类机制：（i）在$\mathrm{SO}(d)$中的乘法旋转（乘法GRAPE），以及（ii）源于一般线性群$\mathrm{GL}$中幂幺作用的加法对数偏置（加法GRAPE）。在乘法GRAPE中，位置$n \in \mathbb{Z}$（或$t \in \mathbb{R}$）通过$\mathbf{G}(n)=\exp(n\,ω\,\mathbf{L})$作用，其中$\mathbf{L} \in \mathbb{R}^{d \times d}$为秩2斜对称生成元，产生一个相对、复合、保范的映射，并具有闭式矩阵指数。当$d/2$个平面为具有对数均匀谱的规范坐标对时，RoPE被精确恢复。学习的交换子空间和紧致非交换混合分别以每头$O(d)$和$O(r d)$的代价严格扩展了这一几何结构，以捕捉跨子空间的特征耦合。在加法GRAPE中，加法对数作为秩1（或低秩）幂幺作用出现，将ALiBi和遗忘Transformer（FoX）恢复为精确特例，同时保持精确的相对律和流式缓存能力。总体而言，GRAPE为长上下文模型中的位置几何提供了一个原则性的设计空间，将RoPE和ALiBi作为特例包含在内。项目页面：https://github.com/model-architectures/GRAPE。