Recent advances in generative machine learning have opened new possibilities for the discovery and design of novel materials. However, as these models become more sophisticated, the need for rigorous and meaningful evaluation metrics has grown. Existing evaluation approaches often fail to capture both the quality and novelty of generated structures, limiting our ability to assess true generative performance. In this paper, we introduce the Transport Novelty Distance (TNovD) to judge generative models used for materials discovery jointly by the quality and novelty of the generated materials. Based on ideas from Optimal Transport theory, TNovD uses a coupling between the features of the training and generated sets, which is refined into a quality and memorization regime by a threshold. The features are generated from crystal structures using a graph neural network that is trained to distinguish between materials, their augmented counterparts, and differently sized supercells using contrastive learning. We evaluate our proposed metric on typical toy experiments relevant for crystal structure prediction, including memorization, noise injection and lattice deformations. Additionally, we validate the TNovD on the MP20 validation set and the WBM substitution dataset, demonstrating that it is capable of detecting both memorized and low-quality material data. We also benchmark the performance of several popular material generative models. While introduced for materials, our TNovD framework is domain-agnostic and can be adapted for other areas, such as images and molecules.

翻译：生成式机器学习的最新进展为新颖材料的发现与设计开辟了新的可能性。然而，随着这些模型日益复杂，对严谨且有意义评估指标的需求也随之增长。现有评估方法往往未能同时捕捉生成结构的质量与新颖性，限制了我们评估真实生成性能的能力。本文引入传输新颖性距离（TNovD），通过生成材料的质量和新颖性共同评判用于材料发现的生成模型。基于最优传输理论的思想，TNovD利用训练集与生成集特征之间的耦合关系，并通过阈值将其细化为质量和记忆化两个机制。这些特征通过图神经网络从晶体结构中生成，该网络经过对比学习训练，能够区分材料、其增强版本以及不同尺寸的超晶胞。我们在与晶体结构预测相关的典型玩具实验（包括记忆化、噪声注入和晶格形变）上评估了所提出的度量。此外，我们在MP20验证集和WBM替代数据集上验证了TNovD，证明其能够同时检测记忆化和低质量的材料数据。我们还对几种流行的材料生成模型进行了性能基准测试。尽管TNovD框架是针对材料领域提出的，但其具有领域无关性，可适配于其他领域，如图像和分子。