Normalization layers and activation functions are critical components in deep neural networks that frequently co-locate with each other. Instead of designing them separately, we unify them into a single computation graph, and evolve its structure starting from low-level primitives. Our layer search algorithm leads to the discovery of EvoNorms, a set of new normalization-activation layers that go beyond existing design patterns. Several of these layers enjoy the property of being independent from the batch statistics. Our experiments show that EvoNorms not only work well on a variety of image classification models including ResNets, MobileNets and EfficientNets but also transfer well to Mask R-CNN, SpineNet for instance segmentation and BigGAN for image synthesis, significantly outperforming BatchNorm and GroupNorm based layers in many cases.

翻译：正常化层和激活功能是深层神经网络的关键组成部分,这些网络经常相互连接。 我们不是单独设计它们,而是将它们合并成一个单一的计算图,并从低层次原始人开始演变其结构。 我们的层搜索算法导致发现一套超越现有设计模式的新的正常化活动层EvoNorms。 这些层中有几个层具有独立于批量统计的特性。 我们的实验显示, EvoNorms不仅在各种图像分类模型(包括ResNets、移动网络和高效网络)上运作良好,而且向Mask R-CNN、SpineNet(例如分解)和BigGAN(图像合成)转移良好,在许多情况下大大超过BatchNorm和GroupNorm(基于图像的层)。