While deep learning methods have achieved impressive success in many vision benchmarks, it remains difficult to understand and explain the representations and decisions of these models. Though vision models are typically trained on 2D inputs, they are often assumed to develop an implicit representation of the underlying 3D scene (for example, showing tolerance to partial occlusion, or the ability to reason about relative depth). Here, we introduce MRD (metamers rendered differentiably), an approach that uses physically based differentiable rendering to probe vision models' implicit understanding of generative 3D scene properties, by finding 3D scene parameters that are physically different but produce the same model activation (i.e. are model metamers). Unlike previous pixel-based methods for evaluating model representations, these reconstruction results are always grounded in physical scene descriptions. This means we can, for example, probe a model's sensitivity to object shape while holding material and lighting constant. As a proof-of-principle, we assess multiple models in their ability to recover scene parameters of geometry (shape) and bidirectional reflectance distribution function (material). The results show high similarity in model activation between target and optimized scenes, with varying visual results. Qualitatively, these reconstructions help investigate the physical scene attributes to which models are sensitive or invariant. MRD holds promise for advancing our understanding of both computer and human vision by enabling analysis of how physical scene parameters drive changes in model responses.

翻译：尽管深度学习方法在许多视觉基准测试中取得了显著成功，但理解和解释这些模型的表征与决策过程仍然具有挑战性。视觉模型通常在二维输入上进行训练，但通常被认为能够形成对底层三维场景的隐式表征（例如表现出对部分遮挡的容忍度，或具备相对深度推理能力）。本文提出MRD（可微分渲染的元刺激），该方法利用基于物理的可微分渲染技术，通过寻找物理属性不同但能产生相同模型激活（即成为模型元刺激）的三维场景参数，来探究视觉模型对生成式三维场景属性的隐式理解。与以往基于像素的模型表征评估方法不同，这种重建结果始终基于物理场景描述。这意味着我们可以，例如，在保持材质和光照恒定的情况下，探究模型对物体形状的敏感性。作为原理验证，我们评估了多个模型在恢复几何（形状）和双向反射分布函数（材质）等场景参数方面的能力。结果显示目标场景与优化场景之间的模型激活具有高度相似性，但视觉重建结果存在差异。从定性角度看，这些重建有助于探究模型对哪些物理场景属性敏感或不敏感。MRD通过分析物理场景参数如何驱动模型响应的变化，为深化对计算机视觉和人类视觉的理解提供了新的可能性。