Thinking Machines Lab Trains LLM in 4-bit NVFP4

🤖 Thinking Machines Lab Trains LLM in 4-bit NVFP4

The company published a recipe for end-to-end MoE-LLM training on November 13, 2026: forward pass in NVFP4, backward pass in MXFP8 with stochastic rounding, and MoE communication via random Hadamard transformation in S2F6 format. They claim an 8% speedup in pretraining and 20% in RL compared to MXFP8, with an efficiency loss of ~2%. No independent replication or code is available.

🌍 4-bit NVFP4 is becoming a viable format not only for inference but for the full training cycle on Blackwell—reducing the cost of pretraining and RL. The recipe combining NVFP4 forward pass, MXFP8 backward pass, and quantized MoE communication sets a reference for other labs.

👤 On NVIDIA GB300, this delivers 15 PFLOPS compared to 5 for MXFP8 and 2.5 for BF16—4-bit is now a training mode, not just inference. The foundation is an NVIDIA paper (arXiv:2509.25149): a 12B model, 10 trillion tokens, with FP8-level accuracy.

Source 1: https://thinkingmachines.ai/blog/end-to-end-4-bit-training/ Source 2: https://arxiv.org/abs/2509.25149