Neighbor-based methods are a natural alternative to linear prediction for tabular data when relationships between inputs and targets exhibit complexity such as nonlinearity, periodicity, or heteroscedasticity. Yet in deep learning on unstructured data, nonparametric neighbor-based approaches are rarely implemented in lieu of simple linear heads. This is primarily due to the ability of systems equipped with linear regression heads to co-learn internal representations along with the linear head's parameters. To unlock the full potential of neighbor-based methods in neural networks we introduce SoftStep, a parametric module that learns sparse instance-wise similarity measures directly from data. When integrated with existing neighbor-based methods, SoftStep enables regression models that consistently outperform linear heads across diverse architectures, domains, and training scenarios. We focus on regression tasks, where we show theoretically that neighbor-based prediction with a mean squared error objective constitutes a metric learning algorithm that induces well-structured embedding spaces. We then demonstrate analytically and empirically that this representational structure translates into superior performance when combined with the sparse, instance-wise similarity measures introduced by SoftStep. Beyond regression, SoftStep is a general method for learning instance-wise similarity in deep neural networks, with broad applicability to attention mechanisms, metric learning, representational alignment, and related paradigms.

翻译：对于表格数据，当输入与目标之间的关系呈现非线性、周期性或异方差性等复杂性时，基于近邻的方法自然成为线性预测的有效替代方案。然而，在非结构化数据的深度学习中，非参数的基于近邻的方法很少被采用，通常仅使用简单的线性输出层。这主要是因为配备线性回归输出层的系统能够共同学习内部表示与线性层参数。为充分释放基于近邻方法在神经网络中的潜力，我们提出了SoftStep——一种能够直接从数据中学习稀疏实例级相似性度量的参数化模块。当与现有基于近邻的方法结合时，SoftStep构建的回归模型在多样化架构、领域及训练场景中均持续优于线性输出层。我们聚焦于回归任务，从理论上证明：基于均方误差目标的近邻预测构成了一种度量学习算法，能够诱导出结构良好的嵌入空间。随后，我们通过分析和实验验证，这种表示结构与SoftStep引入的稀疏实例级相似性度量相结合时，可转化为更优越的性能表现。除回归任务外，SoftStep作为一种在深度神经网络中学习实例级相似性的通用方法，可广泛适用于注意力机制、度量学习、表示对齐及相关范式。