In medical image segmentation, heterogeneous privacy policies across institutions often make joint training on pooled datasets infeasible, motivating continual image segmentation-learning from data streams without catastrophic forgetting. While the Segment Anything Model (SAM) offers strong zero-shot priors and has been widely fine-tuned across downstream tasks, its large parameter count and computational overhead challenge practical deployment. This paper demonstrates that the SAM paradigm is highly promising once its computational efficiency and performance can be balanced. To this end, we introduce the Alignment Layer, a lightweight, plug-and-play module which aligns encoder-decoder feature distributions to efficiently adapt SAM to specific medical images, improving accuracy while reducing computation. Building on SAM and the Alignment Layer, we then propose Continual Alignment for SAM (CA-SAM), a continual learning strategy that automatically adapts the appropriate Alignment Layer to mitigate catastrophic forgetting, while leveraging SAM's zero-shot priors to preserve strong performance on unseen medical datasets. Experimented across nine medical segmentation datasets under continual-learning scenario, CA-SAM achieves state-of-the-art performance. Our code, models and datasets will be released on \mbox{https://github.com/azzzzyo/Continual-Alignment-for-SAM.}

翻译：在医学图像分割领域，机构间异构的隐私政策常使得基于合并数据集的联合训练难以实现，这促使了从数据流中进行持续图像分割学习以避免灾难性遗忘的需求。尽管Segment Anything Model（SAM）提供了强大的零样本先验，并已在众多下游任务中广泛微调，但其庞大的参数量与计算开销对实际部署构成了挑战。本文论证了SAM范式在计算效率与性能达到平衡后具有极高潜力。为此，我们提出了对齐层（Alignment Layer），一种轻量级即插即用模块，通过对齐编码器-解码器特征分布，高效地将SAM适配至特定医学图像，在提升精度的同时降低计算负担。基于SAM与对齐层，我们进一步提出了SAM的持续对齐（CA-SAM），这是一种持续学习策略，能自动适配合适的对齐层以缓解灾难性遗忘，同时利用SAM的零样本先验保持对未见医学数据集的强性能。在持续学习场景下对九个医学分割数据集的实验表明，CA-SAM实现了最先进的性能。我们的代码、模型及数据集将在https://github.com/azzzzyo/Continual-Alignment-for-SAM发布。