The high dimensional parameter space of modern deep neural networks -- the neuromanifold -- is endowed with a unique metric tensor defined by the Fisher information, estimating which is crucial for both theory and practical methods in deep learning. To analyze this tensor for classification networks, we return to a low dimensional space of probability distributions -- the core space -- and carefully analyze the spectrum of its Riemannian metric. We extend our discoveries there into deterministic bounds of the metric tensor on the neuromanifold. We introduce an unbiased random estimate of the metric tensor and its bounds based on Hutchinson's trace estimator. It can be evaluated efficiently through a single backward pass, with a standard deviation bounded by the true value up to scaling.

翻译：现代深度神经网络的高维参数空间——神经流形——被赋予了一个由费舍尔信息定义的独特度量张量，其估计对于深度学习的理论和实践方法都至关重要。为了分析分类网络的这一张量，我们回到概率分布的低维空间——核心空间——并仔细分析其黎曼度量的谱。我们将此处的发现推广至神经流形上度量张量的确定性界。我们基于哈钦森迹估计器，引入了该度量张量及其界的无偏随机估计。该估计可通过单次反向传播高效计算，其标准差在缩放范围内以真实值为界。