The advent of sparsity inducing techniques in neural networks has been of a great help in the last few years. Indeed, those methods allowed to find lighter and faster networks, able to perform more efficiently in resource-constrained environment such as mobile devices or highly requested servers. Such a sparsity is generally imposed on the weights of neural networks, reducing the footprint of the architecture. In this work, we go one step further by imposing sparsity jointly on the weights and on the input data. This can be achieved following a three-step process: 1) impose a certain structured sparsity on the weights of the network; 2) track back input features corresponding to zeroed blocks of weight; 3) remove useless weights and input features and retrain the network. Performing pruning both on the network and on input data not only allows for extreme reduction in terms of parameters and operations but can also serve as an interpretation process. Indeed, with the help of data pruning, we now have information about which input feature is useful for the network to keep its performance. Experiments conducted on a variety of architectures and datasets: MLP validated on MNIST, CIFAR10/100 and ConvNets (VGG16 and ResNet18), validated on CIFAR10/100 and CALTECH101 respectively, show that it is possible to achieve additional gains in terms of total parameters and in FLOPs by performing pruning on input data, while also increasing accuracy.

翻译：近年来，稀疏诱导技术在神经网络中的应用已经极大地帮助了人们。这些技术不仅减少了架构的占用空间，同时使网络更轻、更快，更适合于资源有限的环境，如移动设备或高需求的服务器。通常，这种稀疏性是在神经网络的权重上施加的，减少了体系结构的占用空间。在本文中，我们通过同时施加权重和输入数据上的稀疏性来进一步推动这一进展。这可以通过以下三个步骤实现：1）在网络权重上施加一定的结构稀疏性；2）跟踪对应于零化权重块的输入特征；3）删除无用的权重和输入特征，并重新训练网络。在网络和输入数据上进行裁剪不仅可以极其减少参数和操作，而且还可以作为解释过程。事实上，通过数据裁剪，我们现在有关于哪些输入特征对于网络保持性能是有用的信息。在MLP（在MNIST上验证）、CIFAR10/100和ConvNets（VGG16和ResNet18）等各种体系结构和数据集上进行的实验验证了，在输入数据上进行裁剪可以实现在总参数和FLOP方面的额外收益，并增加准确性。