Preserving privacy in contemporary NLP models allows us to work with sensitive data, but unfortunately comes at a price. We know that stricter privacy guarantees in differentially-private stochastic gradient descent (DP-SGD) generally degrade model performance. However, previous research on the efficiency of DP-SGD in NLP is inconclusive or even counter-intuitive. In this short paper, we provide an extensive analysis of different privacy preserving strategies on seven downstream datasets in five different `typical' NLP tasks with varying complexity using modern neural models based on BERT and XtremeDistil architectures. We show that unlike standard non-private approaches to solving NLP tasks, where bigger is usually better, privacy-preserving strategies do not exhibit a winning pattern, and each task and privacy regime requires a special treatment to achieve adequate performance.

翻译：在当代国家劳工政策模式中保护隐私使我们得以与敏感数据合作,但不幸的是,其代价是一定的。我们知道,在差异性私人随机梯度下降(DP-SGD)方面,更严格的隐私保障通常会降低模型性能。然而,以前关于国家劳工政策中DP-SGD效率的研究没有结论,甚至反直觉。在本短文中,我们对使用基于BERT和XtremeDistil结构的现代神经模型的五种不同“典型”国家劳工政策下游数据集的七种不同“典型”的维护隐私战略进行了广泛分析。我们表明,与标准性非私人方法不同的是,解决国家劳工政策任务(其中规模通常更好),保护隐私的战略并不呈现出一种获胜的模式,而每一项任务和隐私制度都需要特殊待遇才能达到适当的业绩。