We study the problem of (learning) algorithm comparison, where the goal is to find differences between models trained with two different learning algorithms. We begin by formalizing this goal as one of finding distinguishing feature transformations, i.e., input transformations that change the predictions of models trained with one learning algorithm but not the other. We then present ModelDiff, a method that leverages the datamodels framework (Ilyas et al., 2022) to compare learning algorithms based on how they use their training data. We demonstrate ModelDiff through three case studies, comparing models trained with/without data augmentation, with/without pre-training, and with different SGD hyperparameters. Our code is available at https://github.com/MadryLab/modeldiff .

翻译：我们研究(学习)算法比较问题,目的是找出以两种不同的学习算法所培训的模型之间的差异。我们首先将这一目标正规化,以找到不同的特征变换,即改变用一种学习算法培训的模型预测的输入转换,而不是另一种。然后我们提出模型Diff,这是利用数据模型框架(Ilyas等人,2022年)来比较学习算法的方法。我们通过三个案例研究展示模型Diff,将经过培训的模型与/没有数据增强、/没有培训前和不同的SGD双参数进行比较。我们的代码可以在https://github.com/MadryLab/modeldiff上查阅。