We present SAGA, a versatile and adaptive framework for visuomotor control that can generalize across various environments, task objectives, and user specifications. To efficiently learn such capability, our key idea is to disentangle high-level semantic intent from low-level visuomotor control by explicitly grounding task objectives in the observed environment. Using an affordance-based task representation, we express diverse and complex behaviors in a unified, structured form. By leveraging multimodal foundation models, SAGA grounds the proposed task representation to the robot's visual observation as 3D affordance heatmaps, highlighting task-relevant entities while abstracting away spurious appearance variations that would hinder generalization. These grounded affordances enable us to effectively train a conditional policy on multi-task demonstration data for whole-body control. In a unified framework, SAGA can solve tasks specified in different forms, including language instructions, selected points, and example demonstrations, enabling both zero-shot execution and few-shot adaptation. We instantiate SAGA on a quadrupedal manipulator and conduct extensive experiments across eleven real-world tasks. SAGA consistently outperforms end-to-end and modular baselines by substantial margins. Together, these results demonstrate that structured affordance grounding offers a scalable and effective pathway toward generalist mobile manipulation.

翻译：我们提出了SAGA，一种通用且自适应的视觉运动控制框架，能够泛化至多种环境、任务目标和用户需求。为实现这一能力的高效学习，我们的核心思想是通过将任务目标显式地接地于观测环境，将高层语义意图与低层视觉运动控制解耦。利用基于可供性的任务表示，我们将多样且复杂的行为表达为统一的结构化形式。通过利用多模态基础模型，SAGA将提出的任务表示接地至机器人的视觉观测，生成3D可供性热图，突出任务相关实体，同时抽象掉可能阻碍泛化的无关外观变化。这些接地后的可供性使我们能够基于多任务演示数据有效训练用于全身控制的策略。在统一框架下，SAGA能够处理以不同形式指定的任务，包括语言指令、选定点和示例演示，实现零样本执行和少样本适应。我们在四足机械臂上实例化了SAGA，并在十一项真实世界任务中进行了广泛实验。SAGA始终以显著优势超越端到端和模块化基线方法。这些结果表明，结构化可供性接地为实现通用移动操控提供了可扩展且有效的途径。