Image segmentation foundation models (SFMs) like Segment Anything Model (SAM) have achieved impressive zero-shot and interactive segmentation across diverse domains. However, they struggle to segment objects with certain structures, particularly those with dense, tree-like morphology and low textural contrast from their surroundings. These failure modes are crucial for understanding the limitations of SFMs in real-world applications. To systematically study this issue, we introduce interpretable metrics quantifying object tree-likeness and textural separability. On carefully controlled synthetic experiments and real-world datasets, we show that SFM performance (\eg, SAM, SAM 2, HQ-SAM) noticeably correlates with these factors. We attribute these failures to SFMs misinterpreting local structure as global texture, resulting in over-segmentation or difficulty distinguishing objects from similar backgrounds. Notably, targeted fine-tuning fails to resolve this issue, indicating a fundamental limitation. Our study provides the first quantitative framework for modeling the behavior of SFMs on challenging structures, offering interpretable insights into their segmentation capabilities.

翻译：图像分割基础模型（SFMs），如Segment Anything Model（SAM），已在多种领域中实现了令人印象深刻的零样本和交互式分割。然而，它们在分割具有特定结构的物体时面临困难，尤其是那些具有密集、树状形态且与周围环境纹理对比度较低的物体。这些失败模式对于理解SFMs在实际应用中的局限性至关重要。为系统研究此问题，我们引入了可解释的度量指标，用于量化物体的树状特性和纹理可分离性。通过精心控制的合成实验和真实世界数据集，我们表明SFM性能（例如SAM、SAM 2、HQ-SAM）与这些因素显著相关。我们将这些失败归因于SFMs将局部结构误解为全局纹理，导致过度分割或难以从相似背景中区分物体。值得注意的是，针对性微调未能解决此问题，表明存在根本性限制。我们的研究首次提供了量化框架，用于建模SFMs在挑战性结构上的行为，为其分割能力提供了可解释的见解。