We explore unique considerations involved in fitting ML models to data with very high precision, as is often required for science applications. We empirically compare various function approximation methods and study how they scale with increasing parameters and data. We find that neural networks can often outperform classical approximation methods on high-dimensional examples, by auto-discovering and exploiting modular structures therein. However, neural networks trained with common optimizers are less powerful for low-dimensional cases, which motivates us to study the unique properties of neural network loss landscapes and the corresponding optimization challenges that arise in the high precision regime. To address the optimization issue in low dimensions, we develop training tricks which enable us to train neural networks to extremely low loss, close to the limits allowed by numerical precision.

翻译：我们探索了将ML模型与非常精确的数据相匹配的独特考虑,这是科学应用经常需要的。我们从经验上比较了各种功能近似方法,并研究其规模如何与不断增加的参数和数据相比较。我们发现,神经网络通过自动发现和利用其中的模块结构,往往能够超过高维实例的典型近似方法。然而,由通用优化器培训的神经网络对于低维案例的影响力较小,这促使我们研究神经网络损失景观的独特性质以及高精度系统中产生的相应的优化挑战。为了解决低维度优化问题,我们开发了培训技巧,使我们能够训练神经网络,使其损失极低,接近数字精确允许的限度。