Protecting privacy in learning while maintaining the model performance has become increasingly critical in many applications that involve sensitive data. Private Gradient Descent (PGD) is a commonly used private learning framework, which noises gradients based on the Differential Privacy protocol. Recent studies show that \emph{dynamic privacy schedules} of decreasing noise magnitudes can improve loss at the final iteration, and yet theoretical understandings of the effectiveness of such schedules and their connections to optimization algorithms remain limited. In this paper, we provide comprehensive analysis of noise influence in dynamic privacy schedules to answer these critical questions. We first present a dynamic noise schedule minimizing the utility upper bound of PGD, and show how the noise influence from each optimization step collectively impacts utility of the final model. Our study also reveals how impacts from dynamic noise influence change when momentum is used. We empirically show the connection exists for general non-convex losses, and the influence is greatly impacted by the loss curvature.

翻译：在涉及敏感数据的许多应用中,在维护模型性能的同时保护学习隐私已变得日益重要。私人渐变源(PGD)是一个常用的私人学习框架,它根据不同隐私协议发出渐变的声音。最近的研究显示,降低噪声规模的下降可以改善最后迭代时的损失,但从理论上理解这种时间表的有效性及其与优化算法的联系仍然有限。在本文中,我们对动态隐私时间表中的噪音影响进行了全面分析,以回答这些关键问题。我们首先提出了一个动态噪音时间表,最大限度地减少PGD的效用上限,并展示了每个优化步骤的噪音影响如何共同影响最终模型的效用。我们的研究还揭示了在使用动力时动态噪音影响变化的影响。我们从经验上表明,一般的非电流损失存在联系,其影响受到损失曲线的极大影响。