We propose new limiting dynamics for stochastic gradient descent in the small learning rate regime called stochastic modified flows. These SDEs are driven by a cylindrical Brownian motion and improve the so-called stochastic modified equations by having regular diffusion coefficients and by matching the multi-point statistics. As a second contribution, we introduce distribution dependent stochastic modified flows which we prove to describe the fluctuating limiting dynamics of stochastic gradient descent in the small learning rate - infinite width scaling regime.

翻译：我们提议在称为随机调整流动的小型学习率制度中,对随机梯度梯度下降进行新的限制。这些SDE是由圆柱形布朗运动驱动的,通过定期的传播系数和与多点统计相匹配来改进所谓的随机梯度修改方程。作为第二点贡献,我们引入了经分配依赖的随机梯度修改流程,我们证明它描述了在小型学习率----无限宽度比例制度中,随机梯度梯度下降变化不定的限制性动态。