Rollout-training disaggregation is emerging as the standard architecture for Reinforcement Learning (RL) post-training, where memory-bound rollout and compute-bound training are physically disaggregated onto purpose-built clusters to maximize hardware efficiency. However, the strict synchronization required by on-policy algorithms introduces severe dependency bubbles, forcing one cluster to idle while the dependent phase is running on the other. We present RollMux, a cluster scheduling framework that reclaims these bubbles through cross-cluster orchestration. RollMux is built on the insight that the structural idleness of one job can be effectively utilized by the active phase of another. To realize this, we introduce the co-execution group abstraction, which partitions the cluster into isolated locality domains. This abstraction enables a two-tier scheduling architecture: an inter-group scheduler that optimizes job placement using conservative stochastic planning, and an intra-group scheduler that orchestrates a provably optimal round-robin schedule. The group abstraction also imposes a residency constraint, ensuring that massive model states remain cached in host memory to enable "warm-star" context switching. We evaluate RollMux on a production-scale testbed with 328 H20 and 328 H800 GPUs. RollMux improves cost efficiency by 1.84x over standard disaggregation and 1.38x over state-of-the-art co-located baselines, all while achieving 100% SLO attainment.

翻译：解耦式推演训练正成为强化学习后训练的标准架构，其中内存受限的推演阶段与计算受限的训练阶段被物理解耦至专用集群，以最大化硬件效率。然而，同策略算法所需的严格同步引入了严重的依赖气泡，导致一个集群在依赖相位于另一集群运行时被迫闲置。本文提出RollMux，一种通过跨集群编排回收这些气泡的集群调度框架。RollMux基于以下洞察构建：一个任务的结构性闲置时段可被另一任务的活跃相位有效利用。为实现此目标，我们引入协同执行组抽象，将集群划分为隔离的局部性域。该抽象支持双层调度架构：组间调度器通过保守随机规划优化任务放置，组内调度器编排可证明最优的轮转调度方案。组抽象同时施加驻留约束，确保海量模型状态持续缓存于主机内存中，以实现“热启动”上下文切换。我们在包含328块H20与328块H800 GPU的生产级测试平台上评估RollMux。实验表明，RollMux较标准解耦架构提升成本效益1.84倍，较先进共置基线提升1.38倍，同时保持100%的服务水平目标达成率。