Local processing is an essential feature of CNNs and other neural network architectures - it is one of the reasons why they work so well on images where relevant information is, to a large extent, local. However, perspective effects stemming from the projection in a conventional camera vary for different global positions in the image. We introduce Perspective Crop Layers (PCLs) - a form of perspective crop of the region of interest based on the camera geometry - and show that accounting for the perspective consistently improves the accuracy of state-of-the-art 3D pose reconstruction methods. PCLs are modular neural network layers, which, when inserted into existing CNN and MLP architectures, deterministically remove the location-dependent perspective effects while leaving end-to-end training and the number of parameters of the underlying neural network unchanged. We demonstrate that PCL leads to improved 3D human pose reconstruction accuracy for CNN architectures that use cropping operations, such as spatial transformer networks (STN), and, somewhat surprisingly, MLPs used for 2D-to-3D keypoint lifting. Our conclusion is that it is important to utilize camera calibration information when available, for classical and deep-learning-based computer vision alike. PCL offers an easy way to improve the accuracy of existing 3D reconstruction networks by making them geometry aware. Our code is publicly available at github.com/yu-frank/PerspectiveCropLayers.

翻译：本地处理是CNN和其他神经网络结构的一个基本特征,这是它们之所以在相关信息在很大程度上是本地信息的图像上如此出色工作的原因之一,然而,常规相机投影所产生的观点效应因图像中不同的全球位置而异。我们引入了透视作物图层(PCL)——一种基于相机几何法的受关注区域视角作物作物(PCL)的一种形式,表明从这一角度的考虑,始终可以提高最先进的3D状态3D构成重建方法的准确性。PCL是模块型神经网络层,一旦插入现有的CNN和MLP结构中,就可决定性地消除视点影响,同时保持端到端的培训和基本神经网络参数数量不变。我们表明,PCL导致3D对使用空间变换网络等裁剪机操作的CNN型结构的重建精确性不断提高,以及令人惊讶的是,MLP用于2D至3D关键点的提升方法。我们的结论是,在可以方便地利用相机校准网络,以便让公众了解我们现有3L的模型。