Skip to main content
Back

Thinking Machines Trains Inkling 2 Small With 4-Bit Precision, Hitting 20% RL Speedup

Thinking Machines detailed how they pretrained Inkling 2 Small with a native 4-bit forward pass, achieving an 8% pretraining and 20% RL speedup.

Paper, Full Access
  • Thinking Machines trained Inkling 2 Small end-to-end with a native 4-bit NVFP4 forward pass.
  • Recipe delivers 8% pretraining speedup and 20% RL speedup over full MXFP8 training.
  • Forward pass runs NVFP4; backward pass uses MXFP8 with stochastic rounding to preserve gradient expectations.
  • Random Hadamard transform plus S2F6 quantization cuts MoE combine traffic from 16 to 8.25 bits per value.
  • Custom BBBK kernels saturate NVLink, ALUs, and Tensor Cores simultaneously on GB300 hardware.
  • Total cost: around 2% compute efficiency hit in pretraining, no measurable RL reward difference. Full post.

Thinking Machines has detailed how it trained Inkling 2 Small with a native 4-bit forward pass from the first pretraining step through the final reinforcement learning rollout. Here, native means the forward matrix multiplications use NVIDIA Blackwell’s NVFP4 hardware path. “End-to-end” describes its use across the training lifecycle; the backward pass still runs in MXFP8. Thinking Machines calls Inkling 2 Small the largest model it knows of trained with this configuration.

On its largest jobs, the company reports the following gains over full MXFP8 training:

Phase Wall-clock speedup Primary benefit
Pretraining 8% Faster forward computation in MoE layers
Reinforcement learning 20% Higher rollout sampling throughput

The larger reinforcement learning gain reflects the workload: RL repeatedly generates candidate responses before scoring them and updating the model, so sampling consumes a substantial share of runtime. The company’s technical write-up explains how it preserved accuracy while accelerating both computation and communication.

Four bits leave little room for error

Low-precision formats represent fewer values, increasing rounding error as tensors pass through layers and optimization steps. NVFP4 combines a 4-bit payload with block-level scaling to recover some of the range lost at that precision.

  • Storage and throughput: Thinking Machines says NVFP4 requires about 28% as many stored bits as BF16, a common 16-bit training format, and runs up to six times faster on GB300 GPUs.
  • Block scaling: Each group of 16 values shares an FP8 E4M3 scale. MXFP8 and MXFP4 use groups of 32 with E8M0 scales, which provide coarser adjustment.
  • Representable values: Before scaling, FP4 can encode only ±{0, 0.5, 1, 1.5, 2, 3, 4, 6}. Shared scales map each block’s real values onto that small set.

Applying NVFP4 throughout the initial design raised cross-entropy loss enough for the team to reject the configuration. The production recipe therefore targets the operations that consume the most time and tolerate quantization best.

A split-precision path through training

The production system uses NVFP4 for the forward pass of mixture-of-experts layers. An MoE model routes each token through a subset of specialized feed-forward networks called experts, making those layers expensive enough to benefit substantially from faster matrix multiplication. Other operations retain higher precision, and the backward pass uses MXFP8 with fresh quantization and stochastic rounding.

Transpose scrambles the scale map

Backpropagation requires transposed forms of weights and activations. NVFP4 assigns a scale to each 1 × 16 block, but transposition moves values into new blocks with different neighbors. Reusing the original scales would quantize the transposed tensors incorrectly.

NVIDIA addressed this problem in an NVFP4 study by assigning scales to 16 × 16 tiles, which remain unchanged after transposition. Thinking Machines found those 256-value tiles too coarse for its workload. A single outlier could set a large scale and force many smaller values to round to zero.

Thinking Machines preserves the tensors statistically instead of requiring identical bits after transposition. It dequantizes the forward tensors, transposes them at higher precision, and quantizes them to MXFP8 with stochastic rounding. Values are rounded up or down with probabilities based on their distance from adjacent representable numbers, making the expected rounded value equal to the original. The design draws on work including TetraJet-v2 and Quartet II, and Blackwell accelerates the required stochastic rounding in hardware.

NVFP4 matrix multiplication shifted the bottleneck from arithmetic to communication between expert GPUs. On a GB300 NVL72 system, NVLink provides 900 GB/s in each direction, while NVFP4 general matrix multiplications, or GEMMs, can reach 15 PFLOPS. That ratio supplies roughly 16,000 floating-point operations of compute capacity per incoming byte, leaving Tensor Cores idle when expert traffic cannot feed them quickly enough.

Thinking Machines reduces collective traffic in both directions, using separate encodings for dispatching tokens to experts and combining expert outputs:

  1. Dispatch: Tokens cross NVLink in NVFP4 form. The backward pass already consumes quantized activations, so transmitting a higher-precision copy would add traffic without helping that computation.
  2. Combine: Expert outputs receive a 32-dimensional randomized Hadamard transform and are quantized into a block-scaled fixed-point format called S2F6. Including scaling overhead, this lowers traffic from 16 bits to 8.25 bits per value with no measurable accuracy loss in the company’s tests.

The Hadamard rotation spreads large outliers across dimensions, producing a more uniform distribution that fixed-point values can encode efficiently. Blackwell exposes PTX instructions such as cvt.rn.scaled.s2f6x2.bf16x2 for direct hardware conversion between BF16 and block-scaled S2F6.

BBBK fuses the hot path

Thinking Machines implemented a custom MoE stack called BBBK, short for Big Beautiful Blackwell Kernel. On GB300, it uses the largest two-SM UMMA matrix-multiply instruction, with a reduction dimension of K=96. A six-stage pipeline supplies matrix operands, while a seven-stage pipeline supplies scale factors, keeping the Tensor Cores fed.

Within the second expert projection’s GEMM epilogue, the kernel also performs the Hadamard transform and S2F6 quantization. Fusing those operations avoids additional memory passes and allows NVLink transfers, integer ALU work, and Tensor Core computation to overlap.

Three guardrails preserve accuracy

  • Higher precision near the output: The final 16.6% of layers remain in MXFP8, close to NVIDIA’s 15% heuristic for protecting quantization-sensitive late layers.
  • Adaptive activation scaling: Activations use Four Over Six, which selects between two scale factors to reduce quantization error. Weights retain the standard scaling path.
  • Independent outer scales: Above the block-level scales, weights use one outer scale per expert and activations use one per token. Per-token scaling keeps quantization independent of batch composition and prevents future tokens from affecting earlier positions.

A 2% FLOP penalty in pretraining

The NVFP4 pretraining recipe incurs about a 2% compute-efficiency penalty. An MXFP8 model reaches the same loss with roughly 2% fewer floating-point operations, according to Thinking Machines. The reported benefit comes from completing those operations faster, producing the 8% wall-clock improvement on the company’s largest pretraining jobs.

During reinforcement learning, the team measured a 20% speedup with no discernible difference in reward. Because the numerical configuration remains consistent from pretraining through RL, the resulting quantized checkpoint requires no separate post-training calibration step.

Blackwell defines the current boundary

Thinking Machines reports that Inkling 2 Small completed pretraining in 68 hours on GB300 NVL72 racks. It says a comparable model trained several months earlier required several weeks. That historical comparison includes changes in hardware and system design, so the direct MXFP8 comparisons of 8% for pretraining and 20% for RL provide the clearer measure of this recipe’s contribution.

The current implementation relies on specific Blackwell capabilities, including accelerated stochastic rounding, S2F6 conversion instructions, and the GB300’s two-SM matrix-multiply path. Porting the design to another accelerator would require equivalent numerical primitives, new communication formats, or replacement kernels.

Thinking Machines identifies FP4 backward computation and support for NVIDIA Vera Rubin systems as its next targets. Backward FP4 would extend low precision to gradient matrix multiplications that currently use MXFP8. The published work includes no performance or accuracy results for that extension yet.