Large-scale GPU clusters are widely-used to speed up both latency-critical (online) and best-effort (offline) deep learning (DL) workloads. However, most DL clusters either dedicate each GPU to one workload or share workloads in time, leading to very low GPU resource utilization. We present MuxFlow, the first production cluster system that supports efficient and safe space-sharing for DL workloads. NVIDIA MPS provides an opportunity to share multiple workloads in space on widely-deployed NVIDIA GPUs, but it cannot guarantee the performance and safety of online workloads. MuxFlow introduces a two-level protection mechanism for memory and computation to guarantee the performance of online workloads. Based on our practical error analysis, we design a mixed error-handling mechanism to guarantee the safety of online workloads. MuxFlow further proposes dynamic streaming multiprocessor (SM) allocation and matching-based scheduling to improve the efficiency of offline workloads. MuxFlow has been deployed at CompanyX's clusters with more than 20,000 GPUs. The deployment results indicate that MuxFlow substantially improves the GPU utilization from 26$\%$ to 76$\%$, SM activity from 16$\%$ to 33$\%$, and GPU memory from 42$\%$ to 48$\%$.

翻译：大规模 GPU 集群广泛用于加速深度学习工作负载的在线和离线处理，但大多数 DL 集群要么将每个 GPU 分配给一个工作负载，要么分享工作负载的时间，导致 GPU 资源利用率非常低。我们提出了 MuxFlow，它是第一个支持 DL 工作负载高效且安全的空间共享的生产集群系统。NVIDIA MPS 为在广泛部署的 NVIDIA GPU 上空间共享多个工作负载提供了机会，但不能保证在线工作负载的性能和安全性。MuxFlow 引入了一种二级内存和计算保护机制，以保证在线工作负载的性能。基于我们的实际误差分析，我们设计了一种混合错误处理机制，以保证在线工作负载的安全性。MuxFlow 还提出了动态流多处理器分配和基于匹配的调度，以改善离线工作负载的效率。MuxFlow 已在 CompanyX 的集群中部署了超过 20,000 个 GPU。部署结果显示，MuxFlow 将 GPU 利用率从 26％ 提高到 76％，SM 活动从 16% 提高到 33%，GPU 内存从 42％ 提高到 48％。