Large language models (LLMs) rely on Key-Value (KV) cache to reduce time- to-first-token (TTFT) latency, but existing disk-based KV cache systems using file-per-object layouts suffer from severe scalability bottlenecks due to file system metadata overhead, I/O inefficiency, and poor spatial locality. This paper presents SGLANG-LSM, a database-inspired system that leverages Log-Structured Merge- tree (LSM-tree) architectures for scalable KV cache management. SGLANG-LSM implements a layered system design with three coordinated components: (1) a prefix-preserving storage engine that maintains token sequence locality while efficiently storing large KV cache tensors through key-value separation, (2) an adaptive controller that dynamically optimizes LSM-tree configurations based on shifting workload characteristics, and (3) runtime services including batch opera- tions and automatic resource management for production deployment. Evaluation on large-scale dynamic workloads demonstrates that SGLANG-LSM significantly improves cache hits by up to 143% and reduces TTFT by up to 24% compared to state-of-the-art systems, representing the first systematic application of database storage architectures to large-scale LLM cache management.

翻译：大型语言模型（LLMs）依赖键值（KV）缓存来降低首词生成时间（TTFT）延迟，但现有基于磁盘的键值缓存系统采用每个对象单独文件的布局，由于文件系统元数据开销、I/O效率低下以及空间局部性差，存在严重的可扩展性瓶颈。本文提出了SGLANG-LSM，一个受数据库启发的系统，利用日志结构合并树（LSM-tree）架构进行可扩展的键值缓存管理。SGLANG-LSM实现了一个分层系统设计，包含三个协调组件：（1）一个前缀保持存储引擎，通过键值分离高效存储大型键值缓存张量，同时保持词元序列的局部性；（2）一个自适应控制器，根据动态变化的工作负载特性优化LSM-tree配置；（3）运行时服务，包括批处理操作和自动资源管理，以支持生产部署。在大规模动态工作负载上的评估表明，与最先进的系统相比，SGLANG-LSM将缓存命中率显著提升高达143%，并将TTFT降低高达24%，这代表了数据库存储架构在大规模LLM缓存管理中的首次系统性应用。