While classic studies proved that wide networks allow universal approximation, recent research and successes of deep learning demonstrate the power of deep networks. Based on a symmetric consideration, we investigate if the design of artificial neural networks should have a directional preference, and what the mechanism of interaction is between the width and depth of a network. Inspired by the De Morgan law, we address this fundamental question by establishing a quasi-equivalence between the width and depth of ReLU networks in two aspects. First, we formulate two transforms for mapping an arbitrary ReLU network to a wide network and a deep network respectively for either regression or classification so that the essentially same capability of the original network can be implemented. Then, we replace the mainstream artificial neuron type with a quadratic counterpart, and utilize the factorization and continued fraction representations of the same polynomial function to construct a wide network and a deep network, respectively. Based on our findings, a deep network has a wide equivalent, and vice versa, subject to an arbitrarily small error.

翻译：传统的研究证明,广泛的网络可以实现普遍近似,最近的研究和深层学习的成功证明了深层网络的力量。 基于对称考虑,我们调查人工神经网络的设计是否应该具有方向偏好,以及网络宽度和深度之间的互动机制是什么。受德摩根法律的启发,我们通过在ReLU网络宽度和深度之间建立准等值的两个方面来解决这一根本问题。首先,我们制定两种转变,将任意的ReLU网络绘制成宽广的网络和深度的网络,以便分别进行倒退或分类,从而可以实现原始网络的基本上相同的能力。然后,我们用一个二次对称取代主流人造神经类型,并分别利用同一复合功能的因子化和连续的片段表示来构建一个宽广的网络和深度网络。根据我们的调查结果,一个深层的网络具有广泛的等同,反之则有任意的小错误。