Bayesian neural network (BNN) allows for uncertainty quantification in prediction, offering an advantage over regular neural networks that has not been explored in the differential privacy (DP) framework. We fill this important gap by leveraging recent development in Bayesian deep learning and privacy accounting to offer a more precise analysis of the trade-off between privacy and accuracy in BNN. We propose three DP-BNNs that characterize the weight uncertainty for the same network architecture in distinct ways, namely DP-SGLD (via the noisy gradient method), DP-BBP (via changing the parameters of interest) and DP-MC Dropout (via the model architecture). Interestingly, we show a new equivalence between DP-SGD and DP-SGLD, implying that some non-Bayesian DP training naturally allows for uncertainty quantification. However, the hyperparameters such as learning rate and batch size, can have different or even opposite effects in DP-SGD and DP-SGLD. Extensive experiments are conducted to compare DP-BNNs, in terms of privacy guarantee, prediction accuracy, uncertainty quantification, calibration, computation speed, and generalizability to network architecture. As a result, we observe a new tradeoff between the privacy and the reliability. When compared to non-DP and non-Bayesian approaches, DP-SGLD is remarkably accurate under strong privacy guarantee, demonstrating the great potential of DP-BNN in real-world tasks.

翻译：Bayesian神经网络(BNN)在预测中允许对不确定性进行量化,为不同隐私(DP)框架中尚未探索的常规神经网络提供了优势。我们利用Bayesian深层学习和隐私会计的最新发展,填补了这一重要差距,对Bayesian隐私和准确性之间的权衡进行了更精确的分析。我们提议了三个DP-BNNN,这些DP-BNNN,以不同的方式,即DP-SGLD(通过骚动梯度方法)、DP-BBNP(通过改变兴趣参数)和DP-MC辍学(通过模型结构),对正常神经神经网络网络网络网络网络网络网络进行新的等同,意味着某些非Bayesian DP培训自然允许对不确定性进行量化。然而,学习率和批量大小等超常参数,在DP-SGD和DP-SGLD中可能会产生不同或甚至相反的影响。我们进行了广泛的实验,以比较DP-BNNP(通过隐私保证、预测准确性量化、校准、计算速度和通用的DP-D-R)与网络架构的精确性任务之间,我们观察到了在可靠度和不可靠度下进行新的贸易。