vLLM reports 842 tokens per second for Kimi decoding with NVIDIA DSpark

• Kimi K3 reached 842 decode tokens/s on 16 GB300 GPUs: vLLM's announcement links to a controlled DSpark benchmark with synthetic acceptance, showing 1.73× the stock implementation's throughput.
• AgentX interactivity rose from 223.7 to 403.2 tokens/s/user, but the announced 1.8× gain compares 16 GPUs against eight, according to the benchmark write-up.
• Engineers disputed the “megakernel” description: marksaroufim characterized the implementation as a CUDA graph engine with persistent kernels and programmatic dependent launch, or PDL.

A compiler audit caught activation reads moving ahead of a synchronization wait. The merged pull request adds 12,805 lines across 50 files, with the optimized execution path isolated from ordinary serving.

NVIDIA engineer Benjamin Chislett reported 488 → 842 decode tokens/s under the same speculative-decoding setup. The benchmark configuration is specific:

• Hardware: 16 GB300 GPUs, using 16-way tensor parallelism, or TP16.
• Batch and sequence lengths: batch one, one input token, roughly 1,000 output tokens.
• Speculation: DSpark with seven draft tokens and synthetic acceptance length of six.
• Measurement: aggregate decode throughput, excluding prefill.

The acceptance length is imposed for the performance measurement, rather than measured from ordinary requests.

Without speculative decoding, the blog reports 256.4 tokens/s at TP16 batch one, 1.88× stock vLLM. At batch 64, throughput reaches 3,606.1 tokens/s across the entire TP group, 1.62× stock.

AgentX replays coding-agent traffic patterns while replacing private content with synthetic text. The supplied results contain three different comparisons:

• vLLM's AgentX comparison: 223.7 → 403.2 P90 tokens/s/user at concurrency one, 1.8×, using 16 GB300 GPUs for the optimized run versus eight for the earlier baseline, according to the write-up.
• SemiAnalysis's Kimi update: 423 tokens/s/user and a 1.68× improvement, described as a same-hardware comparison. Its chart compares September 15 and October 18 serving curves.
• A separate DeepSeek cost result: SemiAnalysis reports 1.3 billion tokens per dollar of TCO in another post. Its chart concerns DeepSeek V4.1 Flash 552B, comparing disaggregated GB300 NVL72 serving against aggregated B300 serving at large-hyperscaler ownership prices, with roughly 4× better token economics.

The kernel timeline shrinks from 130 to 69 microseconds per layer, with 23 → 13 kernels. Chislett describes four optimization areas:

1. Workload-specific shapes. Attention input projections use small-matrix kernels at batches one and two, then split-K kernels at larger batches. Decode context parallelism distributes the KV cache across GPUs for long contexts.
2. Communication overlap. Attention output projection fuses with all-reduce and residual computation. Larger batches switch to a two-stage reduction to reduce inter-GPU data movement.
3. MoE scheduling. Router scores and latent input projections are divided across tensor-parallel ranks instead of duplicated on every GPU. The shared expert runs on a separate stream, overlapping other work.
4. Weight prefetching. The neatest trick: a side-stream kernel brings upcoming weights into cache while communication runs.

SemiAnalysis called the roughly 12,000-line implementation a megakernel. The objection from marksaroufim concerned that architectural label, describing a graph engine coordinating persistent kernels and PDL instead.

The implementation merged October 16 and remains opt-in:

--additional-config '{"decode_backend": "latency_specialized"}'

Its supported envelope includes:

• GB300 with TP8 or TP16 inside one NVLink domain with multicast support.
• Non-speculative decode batches up to 64.
• DSpark verification with three to seven draft tokens, subject to a 64-token verification-batch limit: batch eight with seven drafts, or batch 16 with three.
• Optional decode context parallelism.

No model-weight changes are required. Prefills, mixed batches and oversized batches fall back to stock vLLM execution.

Chislett describes the work as agent-assisted development and Kimi K3 as the first contribution following the proposed fast-track optimization approach. The blog sets out four principles:

1. Isolate changes within model-specific code.
2. Bound supported hardware and workloads while preserving production features.
3. Validate numerical correctness and complete-model behavior systematically.
4. Scale validation through reproducible evaluations on contributors' GPUs.

Correctness evaluations used probabilistic drafting and standard rejection sampling, without forced acceptance. The PR reports AIME25, GPQA-Diamond and MRCR results matching stock within about one standard error; a larger MRCR matrix scored 0.296 for the optimized build versus 0.290 for stock.

CI coverage remained unsettled in the merged PR's checklist. Chislett said he was satisfied with validation but was still determining how much belonged in CI, potentially including a GSM8k sanity check.