As large language models have grown larger, low-precision numerical formats such as NVFP4 have become increasingly popular due to the speed and memory benefits they provide. However, to accelerate computation with NVFP4, all matrix multiplication operands--weights and activations in the forward pass, and weights, activations, and gradients in the backward pass--must be quantized to NVFP4, often leading to divergence during training and performance degradation during inference. To address this issue, in this work we introduce Four Over Six (4/6), a modification to the NVFP4 quantization algorithm that evaluates two potential scale factors for each block of values. Unlike integer formats, floating-point formats such as FP4 have the most quantization error on near-maximal values in each block, which we find to be primarily responsible for downstream performance degradation. We find that for some blocks, scaling to smaller FP4 values makes the distribution of representable values more uniform, improving representation of near-maximal values. Importantly, 4/6 can be implemented efficiently on NVIDIA Blackwell GPUs, making it viable to use while training LLMs with NVFP4. In pre-training experiments with transformer and hybrid model architectures, we find that 4/6 prevents divergence in several cases, bringing training loss significantly closer to BF16 compared to models trained with current state-of-the-art NVFP4 training recipes. We also find that 4/6 can be easily incorporated into many different post-training quantization methods and generally improves downstream accuracy. We hope this inspires future work in training and deploying models with NVFP4. Our code is available at http://github.com/mit-han-lab/fouroversix.

翻译：随着大语言模型规模的不断增大，低精度数值格式（如NVFP4）因其带来的速度与内存优势而日益普及。然而，为利用NVFP4加速计算，所有矩阵乘法操作数——前向传播中的权重与激活值，以及反向传播中的权重、激活值与梯度——均需量化为NVFP4格式，这常导致训练过程中的发散及推理时的性能下降。为解决此问题，本研究提出“四比六”（4/6）方法，对NVFP4量化算法进行改进，为每个数值块评估两种可能的缩放因子。与整数格式不同，浮点格式（如FP4）在每个块中接近最大值的数值上会产生最大的量化误差，我们发现这是导致下游性能下降的主要原因。研究表明，对于某些数值块，缩放至较小的FP4值可使可表示值的分布更均匀，从而改善接近最大值数值的表示效果。重要的是，4/6方法可在NVIDIA Blackwell GPU上高效实现，使其能够在大语言模型的NVFP4训练中实际应用。在基于Transformer及混合模型架构的预训练实验中，我们发现4/6方法在多种情况下避免了训练发散，相较于采用当前最先进NVFP4训练方案训练的模型，其训练损失显著更接近BF16精度下的结果。我们还发现，4/6方法可轻松集成到多种不同的训练后量化方案中，并普遍提升下游任务精度。我们希望这项工作能启发未来在NVFP4模型训练与部署方面的进一步研究。代码已开源：http://github.com/mit-han-lab/fouroversix。