vLLM reports 58× DeepSeek V4.1 Flash throughput on Rubin

• The 58× gain applies at a matched P90 generation rate of 387 tokens/s/user: vLLM reports 1.7 million total tokens/s/GPU on Vera Rubin NVL72 running DeepSeek V4.1 Flash.
• Rubin-specific kernel rewrites cut profiled prefill steps by about 28% and decode steps by about 25%, according to vLLM's kernel breakdown.
• Rack-wide KV and Engram pools sustain 96.5–97% prefix-cache hits at high concurrency, while freeing requests from rank affinity, as described in vLLM's thread.

Rubin's packed-FP4 layout fits 11 weight-pipeline stages in shared memory where the previous layout fit seven. The cache work is the sleeper feature: moving Engram tables into HBM frees host memory for roughly 1,500 additional live sessions.

The million-token figures count cached input, uncached input and output, according to vLLM's measurement notes. The rack-scale comparison uses prefill/decode disaggregation and Wide-EP, with Rubin results labeled as previews.

AgentX replays growing conversations and bursts of subagents. Each plotted point aggregates thousands of requests with changing cache state and in-flight load.

AgentX total throughput per GPU versus P90 generation tokens/s/user. The peak-throughput improvement is 2.7×; the 58× result sits farther along the latency-throughput curve.

Inferact led the optimization work, vLLM said, crediting DeepSeek, NVIDIA, Mooncake and SemiAnalysis.

Each Rubin GPU has two memory domains, and SMs read their own domain faster than interleaved memory or the other domain. The kernel changes exploit that locality alongside new tensor-core instructions:

• Local weights and residuals: expert weights are split by output columns across the memory domains; residual processing drains local stripes before helping the other domain.
• Wider MMA instructions: SM107's K64 FP8×FP4 instructions do twice the work per instruction. MegaMoE gains 8–11% throughput at 8K-plus tokens per rank; the low-batch routed GEMM runs 12% faster.
• More work in flight: decode attention keeps three blocks of index loads and four gather batches outstanding, and starts dependent kernels early.

The 28% prefill step-time reduction was measured at 32K tokens per rank. The 25% decode reduction compares against the first Rubin build, using a DEP32 target-forward profile simulated on a four-GPU node.

The specialized low-latency path spreads tensor-parallel decode across two compute trays. Cross-tray all-reduce adds only 0.3 microseconds over an in-tray operation; TP8 halves per-GPU weight reads, with a reported 35% reduction in step time from that change.

The decode implementation also removes work at operator boundaries:

• Fused reductions: GEMM epilogues publish directly into NVLink multicast mailboxes. Receiving kernels reduce the ranks, apply mHC, collapse residual routes and compute RMSNorm.
• Inline shared expert: below 16 rows, the shared expert runs inside routed GEMMs, saving 220 microseconds per step.
• Early KV gathers: sparse attention fetches existing KV rows while the previous kernel still runs, saving 16–21 microseconds per step.

At concurrency one, the tuning comparison holds DSpark acceptance fixed at 3.51 tokens per step. P50 generation rises from 1,117 to 1,844 tokens/s/user; the headline single-session result is 1,810 at P90.

The GB300 setup kept prefixes in per-rank CPU caches and pinned sessions to ranks. Rubin's higher concurrency outgrew those caches and left some ranks idle while others queued.

Prefill ranks can now load any session's prefix through the shared pools described by vLLM:

• Mooncake KV: one pool spans every tray's DRAM and spare prefill-GPU HBM, accessed over RDMA. The HBM extension remains experimental.
• Engram: two 94.4 GiB hashed n-gram tables are sharded across GPU HBM. The prefill peer group uses 10.5 GiB on each of 18 GPUs and reads peers over NVLink.
• Dynamo routing: each request goes to the worker and data-parallel rank with the least uncached work.
• NIXL transfer: KV moves from prefill to decode through a separate push path.

In vLLM's three-instance test, at concurrency 4,800, pooling plus routing served 68% more requests and cut median time to first token from 29.8 seconds to 4.2 seconds.

A separate 12 × DEP2 simulation reduced P90 time to first token from 0.98 seconds to 0.38 seconds. Time spent idle while another rank queued fell from 10.3% to 1.3%.

GB300 also reached 1,725 tokens/s/user using TP24 and TileRT MegaKernels in a separate InferenceX result. SemiAnalysis described decode as twice as fast as general-purpose engines and said multi-concurrency results were still to come.

The GB300 baseline in vLLM's 58× chart uses Dynamo plus vLLM, a different serving stack from this TileRT result.