Inkling 2 Small uses four-bit training, Thinking Machines reports 8% and 20% savings

• Inkling 2 Small used native four-bit expert forward passes from pretraining through the final RL rollout, according to Thinking Machines.
• Reported wall-clock speedups over full MXFP8 training were 8% in pretraining and 20% in RL, figures quoted by rohanpaul_ai.
• Backward passes remain MXFP8, with stochastic re-quantization to reduce gradient bias in the published training breakdown.
• Weights are Apache 2.0-licensed in the lab’s release; the NVFP4 checkpoint requires at least 500 GB of aggregate GPU memory, according to rohanpaul_ai.

A 32-dimensional Hadamard transform sits in the GPU communication path. The lab calls its kernel stack BBBK, Big Beautiful Blackwell Kernel, and the shipping checkpoint keeps its final eight expert layers in eight bits.

Thinking Machines described three choices in its November 13 training post, also shared by soumithchintala:

• Forward computation: NVFP4 expert-layer matrix multiplications, supported by a dedicated kernel.
• Backward computation: MXFP8 matrix multiplications, with re-quantization and stochastic rounding of saved weights and activations.
• GPU communication: Quantized transfers to relieve bottlenecks exposed by faster computation.

The lab targeted MoE layers because they are computationally expensive and, in its experiments, tolerate quantization better than other layers. Initial experiments using NVFP4 for both forward and backward passes caused unacceptable losses in compute efficiency.

Keeping the same numerical recipe throughout training eliminates a separate calibration step when delivering the quantized checkpoint.

NVFP4 shares an eight-bit E4M3 scale across each block of 16 four-bit values. Including that block scale, storage averages 4.5 bits per value, as shown in the lab’s format comparison.

| Format | Bits per value, including block scales | Quoted GB300 compute |
| --- | ---: | ---: |
| BF16 | 16 | 2.5 PFLOPS |
| MXFP8 | 8.25 | 5 PFLOPS |
| NVFP4 | 4.5 | 15 PFLOPS |

An additional FP32 outer scale extends the representable range. Thinking Machines uses two granularities for those outer scales:

• Weights: One outer scale per expert.
• Activations: One outer scale per token, keeping quantization independent of other batch members and avoiding dependence on future tokens.

Transposing a block-scaled tensor changes which values share a scale, so quantizing the two orientations independently can produce different represented weights. The lab tried a shared scale across each 16×16 tile, but found it too outlier-sensitive: one large value could force the remaining values to round to zero.

The clever bit in the replacement recipe is preserving forward-pass values in expectation:

1. De-quantize the saved forward-pass weights and activations.
2. Transpose them in higher precision.
3. Re-quantize to MXFP8 with hardware-accelerated stochastic rounding.
4. Compute gradients with MXFP8 matrix multiplications.

In the post’s FP4 illustration, a represented value of 4.5 becomes 4 with 75% probability or 6 with 25% probability after regrouping. Its expectation remains 4.5, avoiding the systematic downward bias of always rounding to 4.

Random rounding error can average out across a training batch. Consistent rounding bias can instead compound across layers and optimization steps.

Faster expert matrix multiplications exposed communication bottlenecks, Thinking Machines reports in its communication analysis. Its GB300 figures pair 15 PFLOPS of NVFP4 compute with 900 GB/s of NVLink bandwidth in each direction.

The resulting communication path has three important mechanics:

• Dispatch: Tokens travel to their routed experts in NVFP4. The backward pass also uses the quantized forward activations, avoiding a separate high-precision activation transfer.
• Combine: Expert outputs undergo a 32-dimensional random Hadamard transform, then quantization into block-scaled S2F6, an eight-bit fixed-point format. The lab found no measurable training accuracy degradation from this combination.
• Fusion: BBBK folds the transform and S2F6 quantization into the output matrix multiplication’s epilogue.

An earlier MXFP8 combine path lost too much accuracy because of outliers. The Hadamard rotation spreads their magnitude across dimensions before quantization; S2F6 allocates two signed integer bits and six fractional bits.

Thinking Machines reports two learning-quality results alongside its wall-clock gains in the training comparison:

• Pretraining: About a 2% compute-efficiency penalty relative to MXFP8, roughly 2% more compute to reach comparable loss.
• RL: No discernible difference in reward.

Pretraining completed in 68 hours on GB300 NVL72 racks, according to the closing passage shared by teortaxesTex. The lab attributes the improvement over earlier, weeks-long runs to changes across the stack.

Inkling 2 Small’s weights are available under Apache 2.0, with their release also announced by VictoriaLinML. Thinking Machines lists two aggregate VRAM requirements in the model card:

| Checkpoint | Minimum aggregate VRAM | Example configurations |
| --- | ---: | --- |
| Native NVFP4 | 500 GB | 2× B300 or 4× H200 |
| MXFP8 | 800 GB | 4× B300 or 8× H200 |

The lab reports a 1–2 percentage-point evaluation advantage for the MXFP8 checkpoint over NVFP4.

Hosted access also arrived on Vercel AI Gateway, while Prime Inference announced its own endpoint.

The 594B-parameter MoE activates 17B parameters per token, according to the model card. Released expert weights use NVFP4 in layers 0–39 and MXFP8 in layers 40–47, as eliebakouch noted.

Width-four short convolutions add a rolling three-token cache per layer alongside the KV cache, as andrew_n_carr pointed out.

vLLM’s day-zero implementation supports the bundled DSpark speculator, which uses an RNN head and proposes seven tokens per step. The team reported an acceptance length of about 3.5 with probabilistic drafting at temperature 1.0.

The serving recipe documents several deployment constraints:

• TP4 on one GB200 tray: Verified contexts up to 262,144 tokens.
• TP8 on two GB200 trays: Verified full context of 1,048,576 tokens.
• Draft sampling: Default greedy drafting produces about 15–25% lower acceptance at temperature 1.0 than probabilistic drafting.
• LoRA: Routed-expert adapters require BF16 expert weights. The native NVFP4/MXFP8 checkpoint supports attention-only adapters under this recipe.