KV cache quantization can improve Large Language Models (LLMs) inference throughput and latency in long contexts and large batch-size scenarios while preserving LLMs effectiveness. However, current methods have three unsolved issues: overlooking layer-wise sensitivity to KV cache quantization, high overhead of online fine-grained decision-making, and low flexibility to different LLMs and constraints. Therefore, we theoretically analyze the inherent correlation of layer-wise transformer attention patterns to KV cache quantization errors and study why key cache is generally more important than value cache for quantization error reduction. We further propose a simple yet effective framework KVTuner to adaptively search for the optimal hardware-friendly layer-wise KV quantization precision pairs for coarse-grained KV cache with multi-objective optimization and directly utilize the offline searched configurations during online inference. To reduce the computational cost of offline calibration, we utilize the intra-layer KV precision pair pruning and inter-layer clustering to reduce the search space. Experimental results show that we can achieve nearly lossless 3.25-bit mixed precision KV cache quantization for LLMs like Llama-3.1-8B-Instruct and 4.0-bit for sensitive models like Qwen2.5-7B-Instruct on mathematical reasoning tasks. The maximum inference throughput can be improved by 21.25\% compared with KIVI-KV8 quantization over various context lengths. Our code and searched configurations are available at https://github.com/cmd2001/KVTuner.

翻译：键值缓存量化可在长上下文和大批量场景中提升大语言模型的推理吞吐量与延迟，同时保持模型效能。然而，现有方法存在三个未解决的问题：忽视了不同层对键值缓存量化的敏感度差异、在线细粒度决策开销高，以及对不同大语言模型和约束条件的适应性低。为此，我们从理论上分析了Transformer各层注意力模式与键值缓存量化误差的内在关联，并研究了为何在减少量化误差时，键缓存通常比值缓存更为重要。我们进一步提出了一个简单而有效的框架KVTuner，通过多目标优化自适应地搜索面向粗粒度键值缓存的最优硬件友好型层间量化精度对，并在在线推理时直接应用离线搜索到的配置。为降低离线校准的计算成本，我们利用层内键值精度对剪枝和层间聚类来缩减搜索空间。实验结果表明，对于如Llama-3.1-8B-Instruct的大语言模型，我们能在数学推理任务上实现近乎无损的3.25位混合精度键值缓存量化；对于如Qwen2.5-7B-Instruct的敏感模型，则能达到4.0位量化。在不同上下文长度下，与KIVI-KV8量化相比，最大推理吞吐量可提升21.25%。我们的代码及搜索到的配置已公开于https://github.com/cmd2001/KVTuner。