Existing methods for isolating hard subpopulations and spurious correlations in datasets often require human intervention. This can make these methods labor-intensive and dataset-specific. To address these shortcomings, we present a scalable method for automatically distilling a model's failure modes. Specifically, we harness linear classifiers to identify consistent error patterns, and, in turn, induce a natural representation of these failure modes as directions within the feature space. We demonstrate that this framework allows us to discover and automatically caption challenging subpopulations within the training dataset, and intervene to improve the model's performance on these subpopulations. Code available at https://github.com/MadryLab/failure-directions

翻译：孤立硬亚集群和数据集中虚假关联的现有方法往往需要人工干预。这样可以使这些方法具有劳动密集型和数据集的特殊性。为了解决这些缺陷,我们提出了一个可扩展的方法,用于自动蒸馏模型的失败模式。具体地说,我们利用线性分类器来识别一致的错误模式,并反过来促使这些失败模式自然地在特性空间内被描述为方向。我们证明,这一框架使我们能够发现和自动说明培训数据集中具有挑战性的亚集群,并进行干预,以改进模型在这些亚群群中的性能。代码可在https://github.com/MadryLab/failure-directications上查阅。