Experimental studies are a cornerstone of Machine Learning (ML) research. A common and often implicit assumption is that the study's results will generalize beyond the study itself, e.g., to new data. That is, repeating the same study under different conditions will likely yield similar results. Existing frameworks to measure generalizability, borrowed from the casual inference literature, cannot capture the complexity of the results and the goals of an ML study. The problem of measuring generalizability in the more general ML setting is thus still open, also due to the lack of a mathematical formalization of experimental studies. In this paper, we propose such a formalization, use it to develop a framework to quantify generalizability, and propose an instantiation based on rankings and the Maximum Mean Discrepancy. We show how our framework offers insights into the number of experiments necessary for a generalizable study, and how experimenters can benefit from it. Finally, we release the genexpy Python package, which allows for an effortless evaluation of the generalizability of other experimental studies.

翻译：实验研究是机器学习（ML）研究的基石。一个常见且往往隐含的假设是，研究结果将在研究本身之外（例如对新数据）具有泛化性。也就是说，在不同条件下重复同一研究很可能产生相似的结果。现有用于衡量泛化性的框架，借鉴自因果推断文献，无法捕捉ML研究结果的复杂性和研究目标。因此，在更一般的ML设置中衡量泛化性的问题仍然开放，部分原因也在于缺乏对实验研究的数学形式化。在本文中，我们提出了这样一种形式化，利用它开发了一个量化泛化性的框架，并提出了一个基于排序和最大均值差异的实例化方法。我们展示了我们的框架如何为进行泛化性研究所需的实验数量提供见解，以及实验者如何从中受益。最后，我们发布了genexpy Python包，该包允许对其他实验研究的泛化性进行便捷评估。