Neural models learn representations of high-dimensional data on low-dimensional manifolds. Multiple factors, including stochasticities in the training process, model architectures, and additional inductive biases, may induce different representations, even when learning the same task on the same data. However, it has recently been shown that when a latent structure is shared between distinct latent spaces, relative distances between representations can be preserved, up to distortions. Building on this idea, we demonstrate that exploiting the differential-geometric structure of latent spaces of neural models, it is possible to capture precisely the transformations between representational spaces trained on similar data distributions. Specifically, we assume that distinct neural models parametrize approximately the same underlying manifold, and introduce a representation based on the pullback metric that captures the intrinsic structure of the latent space, while scaling efficiently to large models. We validate experimentally our method on model stitching and retrieval tasks, covering autoencoders and vision foundation discriminative models, across diverse architectures, datasets, and pretraining schemes.

翻译：神经模型在高维数据上学习低维流形表示。训练过程中的随机性、模型架构以及额外的归纳偏置等多种因素可能导致不同的表示，即使在同一任务和相同数据上进行学习。然而，近期研究表明，当不同潜在空间共享潜在结构时，表示之间的相对距离可以在失真范围内得以保持。基于这一思想，我们证明通过利用神经模型潜在空间的微分几何结构，可以精确捕捉在相似数据分布上训练的表示空间之间的变换关系。具体而言，我们假设不同的神经模型近似参数化同一底层流形，并引入一种基于拉回度量的表示方法，该方法能够捕捉潜在空间的内在结构，同时可高效扩展至大型模型。我们在模型拼接与检索任务上通过实验验证了该方法，涵盖自编码器与视觉基础判别模型，涉及多种架构、数据集与预训练方案。