Deep Learning has a hierarchical network architecture to represent the complicated feature of input patterns. We have developed the adaptive structure learning method of Deep Belief Network (DBN) that can discover an optimal number of hidden neurons for given input data in a Restricted Boltzmann Machine (RBM) by neuron generation-annihilation algorithm, and can obtain the appropriate number of hidden layers in DBN. In this paper, our model is applied to a facial expression image data set, AffectNet. The system has higher classification capability than the traditional CNN. However, our model was not able to classify some test cases correctly because human emotions contain many ambiguous features or patterns leading wrong answer by two or more annotators who have different subjective judgment for a facial image. In order to represent such cases, this paper investigated a distillation learning model of Adaptive DBN. The original trained model can be seen as a parent model and some child models are trained for some mis-classified cases. For the difference between the parent model and the child one, KL divergence is monitored and then some appropriate new neurons at the parent model are generated according to KL divergence to improve classification accuracy. In this paper, the classification accuracy was improved from 78.4% to 91.3% by the proposed method.

翻译：深层学习有一个分级网络结构架构, 以代表输入模式的复杂特征。 我们开发了深信仰网络(DBN)的适应性结构学习方法, 通过神经神经元的代代破坏算法, 可以发现在受限制的Boltzmann机器(RBM)中, 特定输入数据的最佳隐藏神经元数量, 并且可以在 DBN 中获取适当数量的隐藏层。 在本文中, 我们的模型应用到面部表达图像数据集 AffectNet 。 这个系统比传统的CNN 高分级能力。 但是, 我们的模式无法正确分类一些测试案例, 因为人类情感包含许多模糊的特征或模式, 导致两个或更多对面部图像有不同主观判断的注解者做出错误的回答。 为了代表这类案例, 本文调查了适应性 DBN 的蒸馏学习模型。 原始的训练模型可以被视为父系模型, 一些儿童模型被培训用于某些错误分类的案例。 关于父型和子型的区别, 监测 KL 差异, 然后根据KL 差异产生一些适当的父型模型, 新的神经元, 根据 KL 差异来根据 KL 差异来提高分类。