Conformal prediction constructs a set of labels instead of a single point prediction, while providing a probabilistic coverage guarantee. Beyond the coverage guarantee, adaptiveness to example difficulty is an important property. It means that the method should produce larger prediction sets for more difficult examples, and smaller ones for easier examples. Existing evaluation methods for adaptiveness typically analyze coverage rate violation or average set size across bins of examples grouped by difficulty. However, these approaches often suffer from imbalanced binning, which can lead to inaccurate estimates of coverage or set size. To address this issue, we propose a binning method that leverages input transformations to sort examples by difficulty, followed by uniform-mass binning. Building on this binning, we introduce two metrics to better evaluate adaptiveness. These metrics provide more reliable estimates of coverage rate violation and average set size due to balanced binning, leading to more accurate adaptivity assessment. Through experiments, we demonstrate that our proposed metric correlates more strongly with the desired adaptiveness property compared to existing ones. Furthermore, motivated by our findings, we propose a new adaptive prediction set algorithm that groups examples by estimated difficulty and applies group-conditional conformal prediction. This allows us to determine appropriate thresholds for each group. Experimental results on both (a) an Image Classification (ImageNet) (b) a medical task (visual acuity prediction) show that our method outperforms existing approaches according to the new metrics.

翻译：共形预测构建的是标签集合而非单一的点预测，同时提供概率覆盖保证。除了覆盖保证外，对样本难度的自适应性是一个重要特性。这意味着该方法应为更困难的样本生成更大的预测集，为更简单的样本生成更小的预测集。现有的自适应性评估方法通常通过按难度分组的样本箱来分析覆盖率违反情况或平均集合大小。然而，这些方法常因分箱不平衡而导致对覆盖率或集合大小的估计不准确。为解决此问题，我们提出一种分箱方法，该方法利用输入变换按难度对样本进行排序，随后进行均匀质量分箱。基于此分箱，我们引入了两个指标以更好地评估自适应性。由于平衡分箱，这些指标能更可靠地估计覆盖率违反情况和平均集合大小，从而实现更准确的自适应性评估。通过实验，我们证明与现有指标相比，我们提出的指标与期望的自适应性特性具有更强的相关性。此外，基于我们的发现，我们提出一种新的自适应预测集算法，该算法按估计难度对样本进行分组，并应用组条件共形预测。这使我们能够为每个组确定适当的阈值。在（a）图像分类（ImageNet）和（b）医疗任务（视力预测）上的实验结果表明，根据新指标，我们的方法优于现有方法。