Knowledge Distillation (KD) is a promising technique for reducing the high computational demand of large language models (LLMs). However, previous KD methods are primarily applied to white-box classification models or training small models to imitate black-box model APIs like ChatGPT. How to effectively distill the knowledge of white-box LLMs into small models is still under-explored, which becomes more important with the prosperity of open-source LLMs. In this work, we propose a KD approach that distills LLMs into smaller language models. We first replace the forward Kullback-Leibler divergence (KLD) objective in the standard KD approaches with reverse KLD, which is more suitable for KD on generative language models, to prevent the student model from overestimating the low-probability regions of the teacher distribution. Then, we derive an effective on-policy optimization approach to learn this objective. The student models are named MiniLLM. Extensive experiments in the instruction-following setting show that MiniLLM generates more precise responses with higher overall quality, lower exposure bias, better calibration, and higher long-text generation performance than the baselines. Our method is scalable for different model families with 120M to 13B parameters. Our code, data, and model checkpoints can be found in https://github.com/microsoft/LMOps/tree/main/minillm.

翻译：知识蒸馏（KD）是一种有前景的技术，用于降低大型语言模型（LLMs）的高计算需求。然而，以往的KD方法主要应用于白盒分类模型，或训练小模型以模仿如ChatGPT等黑盒模型API。如何有效地将白盒LLMs的知识蒸馏至小模型中，目前仍研究不足，而随着开源LLMs的繁荣，这一问题变得更为重要。在本工作中，我们提出了一种将LLMs蒸馏为较小语言模型的KD方法。我们首先将标准KD方法中的前向Kullback-Leibler散度（KLD）目标替换为反向KLD，后者更适用于生成式语言模型的KD，以防止学生模型高估教师分布的低概率区域。随后，我们推导出一种有效的在线策略优化方法来学习此目标。所得学生模型命名为MiniLLM。在指令跟随设置下的广泛实验表明，与基线方法相比，MiniLLM生成的响应更精确、整体质量更高、暴露偏差更低、校准性更好，且长文本生成性能更优。我们的方法可扩展至不同模型系列，参数规模从1.2亿到130亿不等。代码、数据及模型检查点可在https://github.com/microsoft/LMOps/tree/main/minillm获取。