Communication compression is essential for scalable distributed training of modern machine learning models, but it often degrades convergence due to the noise it introduces. Error Feedback (EF) mechanisms are widely adopted to mitigate this issue of distributed compression algorithms. Despite their popularity and training efficiency, existing distributed EF algorithms often require prior knowledge of problem parameters (e.g., smoothness constants) to fine-tune stepsizes. This limits their practical applicability especially in large-scale neural network training. In this paper, we study normalized error feedback algorithms that combine EF with normalized updates, various momentum variants, and parameter-agnostic, time-varying stepsizes, thus eliminating the need for problem-dependent tuning. We analyze the convergence of these algorithms for minimizing smooth functions, and establish parameter-agnostic complexity bounds that are close to the best-known bounds with carefully-tuned problem-dependent stepsizes. Specifically, we show that normalized EF21 achieve the convergence rate of near ${O}(1/T^{1/4})$ for Polyak's heavy-ball momentum, ${O}(1/T^{2/7})$ for Iterative Gradient Transport (IGT), and ${O}(1/T^{1/3})$ for STORM and Hessian-corrected momentum. Our results hold with decreasing stepsizes and small mini-batches. Finally, our empirical experiments confirm our theoretical insights.

翻译：通信压缩对于现代机器学习模型的可扩展分布式训练至关重要，但它引入的噪声往往会降低收敛性能。误差反馈（EF）机制被广泛采用以缓解分布式压缩算法的这一问题。尽管现有分布式EF算法具有普及性和训练效率，但它们通常需要预先了解问题参数（如光滑常数）以微调步长，这限制了其实际适用性，尤其在大规模神经网络训练中。本文研究了归一化误差反馈算法，该算法将EF与归一化更新、多种动量变体以及参数无关的时变步长相结合，从而消除了对问题依赖性调参的需求。我们分析了这些算法在最小化光滑函数时的收敛性，并建立了参数无关的复杂度界，该界接近于采用精心调优的问题依赖步长时的最佳已知界。具体而言，我们证明归一化EF21算法在Polyak重球动量下实现了接近${O}(1/T^{1/4})$的收敛速率，在迭代梯度传输（IGT）下为${O}(1/T^{2/7})$，在STORM和Hessian修正动量下为${O}(1/T^{1/3})$。我们的结果在递减步长和小批量设置下成立。最后，实证实验验证了我们的理论见解。