This paper proposes a new neural network architecture by introducing an additional dimension called height beyond width and depth. Neural network architectures with height, width, and depth as hyper-parameters are called three-dimensional architectures. It is shown that neural networks with three-dimensional architectures are significantly more expressive than the ones with two-dimensional architectures (those with only width and depth as hyper-parameters), e.g., standard fully connected networks. The new network architecture is constructed recursively via a nested structure, and hence we call a network with the new architecture nested network (NestNet). A NestNet of height $s$ is built with each hidden neuron activated by a NestNet of height $\le s-1$. When $s=1$, a NestNet degenerates to a standard network with a two-dimensional architecture. It is proved by construction that height-$s$ ReLU NestNets with $\mathcal{O}(n)$ parameters can approximate $1$-Lipschitz continuous functions on $[0,1]^d$ with an error $\mathcal{O}(n^{-(s+1)/d})$, while the optimal approximation error of standard ReLU networks with $\mathcal{O}(n)$ parameters is $\mathcal{O}(n^{-2/d})$. Furthermore, such a result is extended to generic continuous functions on $[0,1]^d$ with the approximation error characterized by the modulus of continuity. Finally, we use numerical experimentation to show the advantages of the super-approximation power of ReLU NestNets.

翻译：本文提出一个新的神经网络架构, 其方法是引入一个额外的维度, 称为宽度和深度以外的高度。 高度、 宽度和深度的神经网络架构, 高度、 宽度和深度的超参数结构, 被称为三维结构。 显示三维结构的神经网络比二维结构( 仅以宽度和深度为超参数) 的神经网络更能表达, 例如标准的完全连接网络。 新的网络架构是通过一个巢状结构循环构建的, 因此我们用新架构嵌入网络( NestNet Net 网络) 建立一个网络。 高度网络以高度、 宽度和深度为高度的超度结构。 当美元=1时, NAstNet 网络会退化为双维结构的标准网络。 以 $\ mathcal_ cality ority $0, 1, 1, 1美元 美元 美元 美元 美元 美元 美元 美元 美元 和 美元 美元 美元 美元 美元 美元 美元 的内值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 值 ======== ===== = ==== = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =