Rongchai Wang
Feb 23, 2026 18:39
NVIDIA’s NVFP4 4-bit coaching format achieves 59% sooner AI mannequin coaching than BF16 whereas matching accuracy on Llama 3 8B benchmarks, per new analysis.
NVIDIA’s NVFP4 low-precision coaching format delivers as much as 1.59x sooner throughput in comparison with commonplace BF16 coaching whereas sustaining equal mannequin accuracy, in accordance with new benchmarks printed by the corporate’s analysis staff on February 23, 2026.
The outcomes mark a major milestone for 4-bit AI coaching, demonstrating that aggressive numerical compression does not require sacrificing mannequin high quality when correct strategies are utilized.
The Numbers That Matter
Testing on Llama 3 8B fashions educated throughout 1 trillion tokens, NVIDIA’s staff measured throughput at 1,850 TFLOP/s per GPU with NVFP4 versus 1,165 TFLOP/s for BF16 baseline—a 59% enchancment. The assessments ran on GB200 NVL72 {hardware} utilizing the corporate’s Blackwell structure.
Downstream benchmark scores inform the true story. On MMLU, NVFP4-trained Llama 3 8B scored 45.64% in comparison with 45.98% for BF16. HellaSwag confirmed 75.59% versus 76.44%. These variations fall inside noise margins for sensible purposes.
Reminiscence effectivity features enabled doubling the micro-batch dimension from 2 to 4 throughout pretraining, instantly enhancing scalability for large-scale coaching runs.
Why 4-Bit Coaching Works Now
Earlier makes an attempt at ultra-low-precision coaching typically resulted in mannequin divergence or vital accuracy degradation. NVIDIA’s strategy sidesteps these points by way of a selected recipe that is emerged from intensive testing.
The important perception: preserving roughly 15% of the community in greater precision prevents coaching collapse. Particularly, the ultimate 4 transformer layers should stay in BF16. Ablation research confirmed that absolutely NVFP4 fashions diverge throughout coaching.
The format makes use of a two-level scaling technique—micro-block scaling for teams of 16 components mixed with world FP32 scaling throughout full tensors. This hierarchical strategy manages the restricted dynamic vary inherent in 4-bit representations.
Random Hadamard transforms easy tensor spectrums and scale back outliers that might in any other case trigger coaching instability. Stochastic rounding for gradients eliminates systematic quantization bias.
Comparability With Different Low-Precision Codecs
NVFP4 is not the one choice. FP8 with present scaling (FP8-CS) achieved 1.33x speedup over BF16, whereas MXFP8—a block-level scaling variant optimized for Blackwell—hit 1.32x. Each codecs confirmed barely higher convergence monitoring than NVFP4 throughout coaching, although last accuracy metrics remained comparable throughout all approaches.
MXFP8 demonstrated marginally higher efficiency than commonplace FP8, probably as a result of finer-grained scaling that higher captures native dynamic vary inside tensors.
Manufacturing Deployment
The strategies can be found now by way of NeMo Megatron Bridge, NVIDIA’s open PyTorch-native library. Switching between precision codecs requires altering a single configuration flag—no mannequin code or optimizer logic modifications wanted.
For groups working large-scale coaching workloads on Blackwell {hardware}, the throughput features translate on to decreased coaching time and compute prices. A mannequin that beforehand required 10 days of coaching may doubtlessly full in below 7 days with NVFP4.
The beneficial recipe for NVFP4: AdamW optimizer with epsilon=1e-8, studying price decaying from 6e-4 to 6e-6, and world batch dimension of 768. These parameters characterize the empirical candy spot from NVIDIA’s intensive testing throughout a number of architectures and datasets.
Picture supply: Shutterstock
