Thinking Machines Lab has published a recipe for end-to-end LLM training in the 4-bit NVFP4 format: the model goes through the entire cycle from pretraining to RL in a hybrid low-precision scheme. According to the company's figures, this provides an 8% speedup in pretraining and a 20% speedup in RL relative to the MXFP8 baseline, turning 4-bit precision from a quantization tool for inference into a full-fledged training mode.

image
image

What Happened

On November 13, 2026, Thinking Machines Lab published a blog post titled "End-to-End 4-Bit Training" with a recipe for LLM training in reduced precision. The forward pass in MoE layers is executed in the 4-bit NVFP4 format, which occupies 4.5 bits per value; on NVIDIA GB300 accelerators, it delivers up to 15 PFLOPS compared to 5 PFLOPS for MXFP8 and 2.5 PFLOPS for BF16. Backward passes are performed in MXFP8 with stochastic rounding, while communications between MoE experts are compressed via random Hadamard transforms into the S2F6 format. According to the company, this is the largest known model trained in this configuration from pretraining to RL; the loss in computational efficiency is estimated at approximately 2%, with no noticeable impact on the reward metric.

Context

Until recently, 4-bit formats were primarily associated with quantizing already trained models for inference, and low precision in training was hindered by three bottlenecks: quantization noise in activations during the forward pass, gradient instability in the backward pass, and the cost of all-reduce exchanges in MoE architectures. The scientific foundation was laid by the NVIDIA paper "Pretraining Large Language Models with NVFP4" (arXiv:2509.25149): there, a 12-billion parameter model was pre-trained on 10 trillion tokens with accuracy comparable to the FP8 baseline. The Thinking Machines Lab publication extends this approach to MoE layers and carries it through to the full cycle including RL, with a specific technique addressing each bottleneck in the scheme: the 4-bit forward pass handles activation quantization noise, stochastic rounding in MXFP8 ensures gradient stability, and Hadamard transform with compression to S2F6 reduces the cost of MoE exchanges. The evidence base is two-tiered: the NVIDIA paper describes a 12B scale result on 10T tokens, while the blog post details the transfer of the recipe to MoE and the end-to-end pipeline up to RL.

Why This Matters for the Industry

For the industry, this shifts the cost curve for training: NVFP4 becomes a working format not only for inference but for the full training cycle on Blackwell-class hardware (GB300), directly reducing the cost of pretraining and especially RL, and with them the future economics of custom models and agent post-training. The combination of NVFP4 in the forward pass, MXFP8 with stochastic rounding in the backward pass, and quantized MoE communications sets a reference recipe that other labs are expected to reproduce and adapt. Since the recipe is published openly, cheap training itself ceases to be a defensible advantage, and the competitive moat shifts from owning the technique to data, infrastructure, and execution. For existing FP8 and BF16 pipelines, nothing changes automatically: a Blackwell cluster, NVFP4 support in the stack, and recipe re-tuning for specific models are required. Further dynamics—a wave of independent reproductions, comparisons of MXFP8 with the NVFP4 hybrid, competing recipes, and the maturation of tooling up to NVFP4 options in clouds and frameworks—remain interpretation rather than established fact.

Why This Matters for Users

There is no direct product impact from this news: the recipe is aimed at owners of Blackwell-class (GB300) accelerator clusters, and the sources contain no public API, checkpoints, or pricing. It is practically useful for teams planning training or RL on Blackwell: the recipe can already be incorporated into benchmarks and pilots, provided the current framework supports NVFP4. The figures should be read carefully: peak 15 PFLOPS does not translate into equivalent speedup because part of each step involves communications and memory-bound operations that 4-bit tensor cores do not accelerate; the significantly larger gain in RL is consistent with the higher proportion of forward passes at this stage. Both primary sources—the Thinking Machines Lab blog and the NVIDIA paper on arXiv—are open and allow for independent understanding of the recipe. Cheaper fine-tuning and RFT in cloud APIs is an expected but not yet confirmed consequence.

What Is Still Unknown / Limitations

There is no independent run yet: the sources contain no code, configs, or third-party reproductions, so the publication makes the recipe replicable in principle, but no reproduced result exists. The speedup figures of 8% and 20%, the efficiency loss of about 2%, and the lack of impact on reward are claims by the company itself. The statement of no noticeable impact on reward is weak without a disclosed evaluation protocol: potential degradation from 4-bit precision may manifest on specialized or long-context tasks not included in the test set. The behavior of the recipe on hardware other than GB300, as well as on models of different scales and architectures, is also unknown.

Sources

Author

Look at AI, editorial team