In this work, we investigate the existence and effect of percolation in training deep Neural Networks (NNs) with dropout. Dropout methods are regularisation techniques for training NNs, first introduced by G. Hinton et al. (2012). These methods temporarily remove connections in the NN, randomly at each stage of training, and update the remaining subnetwork with Stochastic Gradient Descent (SGD). The process of removing connections from a network at random is similar to percolation, a paradigm model of statistical physics. If dropout were to remove enough connections such that there is no path between the input and output of the NN, then the NN could not make predictions informed by the data. We study new percolation models that mimic dropout in NNs and characterise the relationship between network topology and this path problem. The theory shows the existence of a percolative effect in dropout. We also show that this percolative effect can cause a breakdown when training NNs without biases with dropout; and we argue heuristically that this breakdown extends to NNs with biases.

翻译：本研究探讨了在采用Dropout方法训练深度神经网络（NNs）时渗流现象的存在及其影响。Dropout方法是一种由G. Hinton等人（2012）首次提出的神经网络正则化技术。该方法在训练的每个阶段随机移除神经网络中的连接，并利用随机梯度下降（SGD）更新剩余的子网络。这种随机移除网络连接的过程类似于渗流——统计物理学中的一个范式模型。如果Dropout移除了足够多的连接，导致神经网络输入与输出之间不存在路径，那么神经网络将无法基于数据做出预测。我们研究了模拟神经网络中Dropout行为的新渗流模型，并刻画了网络拓扑结构与这一路径问题之间的关系。理论分析表明，Dropout中存在渗流效应。我们还证明，这种渗流效应可能导致在训练不带偏置的神经网络时出现崩溃；并通过启发式论证指出，这种崩溃现象也适用于带偏置的神经网络。