Most estimators collapse all uncertainty modes into a single confidence score, preventing reliable reasoning about when to allocate more compute or adjust inference. We introduce Uncertainty-Guided Inference-Time Selection, a lightweight inference time framework that disentangles aleatoric (data-driven) and epistemic (model-driven) uncertainty directly in deep feature space. Aleatoric uncertainty is estimated using a regularized global density model, while epistemic uncertainty is formed from three complementary components that capture local support deficiency, manifold spectral collapse, and cross-layer feature inconsistency. These components are empirically orthogonal and require no sampling, no ensembling, and no additional forward passes. We integrate the decomposed uncertainty into a distribution free conformal calibration procedure that yields significantly tighter prediction intervals at matched coverage. Using these components for uncertainty guided adaptive model selection reduces compute by approximately 60 percent on MOT17 with negligible accuracy loss, enabling practical self regulating visual inference. Additionally, our ablation results show that the proposed orthogonal uncertainty decomposition consistently yields higher computational savings across all MOT17 sequences, improving margins by 13.6 percentage points over the total-uncertainty baseline.

翻译：多数估计器将所有不确定性模式压缩为单一置信度分数，阻碍了对何时应分配更多计算资源或调整推理过程的可靠判断。本文提出不确定性引导的推理时选择框架，这是一种轻量级的推理时方法，可在深度特征空间中直接解耦偶然不确定性（数据驱动）与认知不确定性（模型驱动）。偶然不确定性通过正则化全局密度模型进行估计，而认知不确定性由三个互补分量构成：局部支持缺失、流形谱塌缩和跨层特征不一致性。这些分量在经验上正交，无需采样、集成或额外前向传播计算。我们将分解后的不确定性融入无分布保形校准流程，在相同覆盖度下获得显著更紧凑的预测区间。利用这些分量进行不确定性引导的自适应模型选择，在MOT17数据集上减少约60%计算量的同时保持精度损失可忽略，实现了实用的自调节视觉推理。消融实验进一步表明，所提出的正交不确定性分解在所有MOT17序列中均能实现更高的计算效率提升，较全不确定性基线平均提升13.6个百分点。