JetSpec++ reports lossless decoding above 4,000 tokens per second

• JetSpec++ exceeds 4,000 tokens/s on Qwen3-8B with NVFP4 and kernel optimizations across two GB200 GPUs, according to the release announcement.
• Drafting moves largely off the critical path: residual drafting time on Qwen3-8B falls from 3.76 to 0.61 ms in the lab's GSM8K breakdown.
• Self-refinement raises average accepted tokens per round from DSpark's 6.89 to 8.94 across seven benchmarks, per the refinement results.
• Code, models and the inference engine are public through the release links.

JetSpec++'s neatest trick is a cache of possible futures, keyed by how many tokens verification accepts and which bonus token follows. A drafter trained on 16-token blocks also handles blocks of 32.

Hao AI Lab reports these peaks with the target model on one GB200 GPU and the drafter on another:

| Target | BF16 | NVFP4 with kernel optimizations |
|---|---:|---:|
| Qwen3-8B | 1,361 tokens/s | 4,117 tokens/s |
| Qwen3.8-27B | 367 tokens/s | Over 900 tokens/s |

The 4,117-token/s result uses NVFP4 weights and activations on MATH-500 at batch size 1, according to the project's GB200 results. On that workload, the BF16 result is 7.45× faster than autoregressive decoding and 1.51× faster than DSpark; the announcement's maximum across results is 1.54× over DSpark.

“Lossless” means rejection sampling preserves the verified target model's output distribution, as defined in the paper. That guarantee is relative to the target used for verification, including its precision configuration.

The project labels its batch-size 2–4 results preliminary and lists higher-batch, realistic serving workloads as ongoing work.

JetSpec++ starts drafting block t+1 while the target verifies block t. Its foresight machinery prepares for several possible verification outcomes:

1. Predict hidden states. An adapter estimates the target states that verification has not yet returned.
2. Score outcomes. A decoder reuses frozen final target layers, the final normalization and LM head to predict accepted-prefix lengths and bonus anchor tokens.
3. Cache next blocks. The drafter batches likely outcomes. A matching outcome selects an already-drafted block; a miss falls back to an ordinary speculative step.

On GSM8K, the prepared outcomes match verification in 92.9% of rounds on 8B and 89.0% on 27B. The lab reports these per-round latency changes:

• Qwen3-8B: 9.92 → 6.84 ms overall; residual drafting time falls from 3.76 → 0.61 ms.
• Qwen3.8-27B: 31.88 → 28.47 ms overall; residual drafting time falls from 4.46 → 0.71 ms.

While verification runs, the draft GPU uses its remaining time for three correction operations:

• Jacobi iteration: each position conditions on the preceding tokens from the previous draft pass.
• Context refresh: the adapter re-predicts target hidden states for the current draft.
• Anchor replacement: the decoder replaces the first token where it disagrees with the draft; the next pass redrafts later positions.

At temperature 0 and verification budget 16, Qwen3-8B averages 8.94 committed tokens per round across seven benchmarks, versus 6.89 for DSpark, according to the paper. Its acceptance-length definition includes the accepted prefix plus one bonus token.

The larger target's verification window fits three refinement passes instead of one. On GSM8K, accepted tokens per round improve over DSpark by 22% on 27B versus 6% on 8B, according to the lab's follow-up.

Despite training on blocks of 16, JetSpec++ increases acceptance when decoding with blocks of 32. On MATH-500, the lab reports:

• Qwen3-8B: 14.55 tokens per round, versus 8.66 for DSpark.
• Qwen3.8-27B: 14.05 tokens per round, versus 8.49 for DSpark.

The GitHub repository hosts the code and inference engine. The Qwen3-8B checkpoint and Qwen3.8-27B checkpoint include:

• Drafter weights: the 8B target's drafter has 1.05B parameters in BF16.
• Adapters: weights and configuration for predicting target hidden states.
• Serving calibration: thresholds, explicitly for block size 16 in the 8B release.
• Decoder manifests: pinned target files from which the engine extracts the decoder's final layers.
• File manifests: SHA-256 hashes and sizes for every released file.