Deep Neural Networks (DNNs) have obtained impressive performance across tasks, however they still remain as black boxes, e.g., hard to theoretically analyze. At the same time, Polynomial Networks (PNs) have emerged as an alternative method with a promising performance and improved interpretability but have yet to reach the performance of the powerful DNN baselines. In this work, we aim to close this performance gap. We introduce a class of PNs, which are able to reach the performance of ResNet across a range of six benchmarks. We demonstrate that strong regularization is critical and conduct an extensive study of the exact regularization schemes required to match performance. To further motivate the regularization schemes, we introduce D-PolyNets that achieve a higher-degree of expansion than previously proposed polynomial networks. D-PolyNets are more parameter-efficient while achieving a similar performance as other polynomial networks. We expect that our new models can lead to an understanding of the role of elementwise activation functions (which are no longer required for training PNs). The source code is available at https://github.com/grigorisg9gr/regularized_polynomials.

翻译：深度神经网络（DNN）在各种任务中取得了卓越的性能，但它们仍然是黑箱，即难以理论分析。同时，多项式网络(PN)已经成为一种具有较好性能和更好可解释性的替代方法，但仍未达到强大的DNN基线的表现。在本文中，我们旨在缩小这一性能差距。我们引入了一类PN，能够在六个基准测试中达到与ResNet相同的性能。我们展示了强大的正则化是至关重要的，并对需要达到匹配性能的确切正则化方案进行了广泛的研究。为了进一步推动正则化方案，我们引入了D-PolyNet，它们实现了比以前建议的多项式网络更高次的扩展。D-PolyNet更具参数效率，同时实现了与其他多项式网络类似的性能。我们希望我们的新模型能够引导理解元素级激活函数的作用(对于PN的训练不再需要)。源代码可在https://github.com/grigorisg9gr/regularized_polynomials上获得。