Explainable AI (XAI) is a rapidly evolving field that aims to improve transparency and trustworthiness of AI systems to humans. One of the unsolved challenges in XAI is estimating the performance of these explanation methods for neural networks, which has resulted in numerous competing metrics with little to no indication of which one is to be preferred. In this paper, to identify the most reliable evaluation method in a given explainability context, we propose MetaQuantus -- a simple yet powerful framework that meta-evaluates two complementary performance characteristics of an evaluation method: its resilience to noise and reactivity to randomness. We demonstrate the effectiveness of our framework through a series of experiments, targeting various open questions in XAI, such as the selection of explanation methods and optimisation of hyperparameters of a given metric. We release our work under an open-source license to serve as a development tool for XAI researchers and Machine Learning (ML) practitioners to verify and benchmark newly constructed metrics (i.e., ``estimators'' of explanation quality). With this work, we provide clear and theoretically-grounded guidance for building reliable evaluation methods, thus facilitating standardisation and reproducibility in the field of XAI.

翻译：解释性大赦国际(XAI)是一个迅速演变的领域,目的是提高AI系统对人类的透明度和可信赖性。XAI的未解决的挑战之一是评估神经网络解释方法的性能,这导致许多相互竞争的计量标准,很少或完全没有说明应选用哪种标准。在本文中,为了确定在特定可解释性背景下最可靠的评价方法,我们提议MetaQuantus -- -- 一个简单而有力的框架,对一种评价方法的两个互补性性能特征进行元评价:它对噪音的抗御力和随机反应的回弹性。我们通过一系列实验,针对XAI的各种开放问题,例如选择解释方法,优化某一指标的超参数,展示了我们框架的有效性。我们以开放源许可方式发布我们的工作,作为XAI研究人员和机械学习(ML)从业者核查和基准确定新构建的计量标准(即“估算者”解释质量的工具。我们通过这项工作为建立可靠的评价方法提供了明确和基于理论的指导,从而促进XAI的实地的标准化和再生化。