Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks, where now excessively large models are used. However, such models face several problems during the learning process, mainly due to the redundancy of the individual neurons, which results in sub-optimal accuracy or the need for additional training steps. Here, we explore the diversity of the neurons within the hidden layer during the learning process, and analyze how the diversity of the neurons affects predictions of the model. As following, we introduce several techniques to dynamically reinforce diversity between neurons during the training. These decorrelation techniques improve learning at early stages and occasionally help to overcome local minima faster. Additionally, we describe novel weight initialization method to obtain decorrelated, yet stochastic weight initialization for a fast and efficient neural network training. Decorrelated weight initialization in our case shows about 40% relative increase in test accuracy during the first 5 epochs.

翻译：具有有限的可培训参数的小型神经网络可以成为许多简单任务的适当资源高效候选人,目前使用的模型过于庞大。然而,这些模型在学习过程中面临若干问题,主要原因是个体神经元的冗余,导致次优化精度或需要额外的培训步骤。在这里,我们探索学习过程中隐藏层内神经元的多样性,分析神经元的多样性如何影响模型的预测。正如下文所述,我们引入了几种技术,在培训期间动态地加强神经元之间的多样性。这些装饰技术改善了早期的学习,有时有助于更快地克服当地迷你形。此外,我们描述了新颖的权重初始化方法,以获得与装饰相关,但为快速有效的神经网络培训进行随机重初始化。我们案例的与神经元相关的重量初始化显示,在前5个世纪,测试精度增加约40%。