Large Language Models (LLMs) achieve state-of-the-art performance across a wide range of applications, but their massive scale poses significant challenges for both efficiency and interpretability. Structural pruning, which reduces model size by removing redundant computational units such as neurons, has been widely explored as a solution, and this study devotes to input sparsification, an increasingly popular technique that improves efficiency by selectively activating only a subset of entry values for each input. However, existing approaches focus primarily on computational savings, often overlooking the representational consequences of sparsification and leaving a noticeable performance gap compared to full models. In this work, we first reinterpret input sparsification as a form of dynamic structural pruning. Motivated by the spontaneous baseline firing rates observed in biological neurons, we introduce a small set of trainable spontaneous neurons that act as compensatory units to stabilize activations in sparsified LLMs. Experiments demonstrate that these auxiliary neurons substantially reduce the sparsification-induced performance gap while generalizing effectively across tasks.

翻译：大型语言模型（LLMs）在广泛的应用中实现了最先进的性能，但其庞大的规模对效率和可解释性均构成了显著挑战。结构剪枝通过移除冗余计算单元（如神经元）来减小模型规模，已被广泛探索为一种解决方案；本研究专注于输入稀疏化，这是一种日益流行的技术，通过为每个输入选择性地仅激活一部分输入值来提高效率。然而，现有方法主要关注计算节省，往往忽视了稀疏化的表征后果，导致与完整模型相比存在明显的性能差距。在本工作中，我们首先将输入稀疏化重新解释为一种动态结构剪枝。受生物神经元中观察到的自发基线放电率启发，我们引入了一小组可训练的自发神经元，作为补偿单元来稳定稀疏化LLMs中的激活。实验表明，这些辅助神经元显著减小了由稀疏化引起的性能差距，并在不同任务中有效泛化。