Transformers have become the de facto architecture for a wide range of machine learning tasks, particularly in large language models (LLMs). Despite their remarkable performance, many challenges remain in training deep transformer networks, especially regarding the position of the layer normalization. While Pre-Norm structures facilitate more stable training owing to their stronger identity path, they often lead to suboptimal performance compared to Post-Norm. In this paper, we propose $\textbf{HybridNorm}$, a simple yet effective hybrid normalization strategy that integrates the advantages of both Pre-Norm and Post-Norm. Specifically, HybridNorm employs QKV normalization within the attention mechanism and Post-Norm in the feed-forward network (FFN) of each transformer block. We provide both theoretical insights and empirical evidence to demonstrate that HybridNorm improves the gradient flow and the model robustness. Extensive experiments on large-scale transformer models, including both dense and sparse variants, show that HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches across multiple benchmarks. These findings highlight the potential of HybridNorm as a more stable and effective technique for improving the training and performance of deep transformer models. Code is available at https://github.com/BryceZhuo/HybridNorm.

翻译：Transformer已成为广泛机器学习任务（尤其是在大型语言模型（LLMs）中）的事实架构。尽管其性能卓越，但在训练深度Transformer网络时仍存在诸多挑战，特别是关于层归一化的位置问题。虽然Pre-Norm结构因其更强的恒等路径而有助于更稳定的训练，但与Post-Norm相比，其性能往往欠佳。本文提出$\textbf{HybridNorm}$，一种简单而有效的混合归一化策略，它整合了Pre-Norm和Post-Norm两者的优势。具体而言，HybridNorm在注意力机制中采用QKV归一化，并在每个Transformer块的前馈网络（FFN）中使用Post-Norm。我们提供了理论分析和实证证据，证明HybridNorm改善了梯度流和模型鲁棒性。在大规模Transformer模型（包括稠密和稀疏变体）上的大量实验表明，HybridNorm在多个基准测试中始终优于Pre-Norm和Post-Norm方法。这些发现凸显了HybridNorm作为一种更稳定、更有效的技术，在提升深度Transformer模型的训练和性能方面的潜力。代码可在https://github.com/BryceZhuo/HybridNorm获取。