vLLM reports 10.2× tokens per second per chip on Rubin

• MiniMax-M3.1-Flash reached 10.2× total tok/s/chip, including cached tokens, on Rubin versus GB300 at 546 P90 tok/s/user, according to vLLM's results.
• Throughput per TCO dollar improved about 6.5× in the initial NVIDIA and Inferact Rubin submission, SemiAnalysis reported.
• The latest gains combine CPU-overhead reductions with Rubin-tuned sparse-attention and decode kernels, according to vLLM's follow-up.

Rubin's higher modeled hourly cost makes the cost-normalized gain the standout. Its attention indexer switches to FP16 to reduce conversion work, while locality-aware MoE steers expert-weight reads toward nearby HBM.

vLLM reported three operating points for MiniMax-M3.1-Flash on Vera Rubin NVL72 versus GB300 NVL72 in its November 11 update:

• 546 P90 tok/s/user: 10.2× total tokens per second per chip, including cached tokens.
• 376 P90 tok/s/user: 6.5× on the same cache-inclusive throughput metric.
• Low concurrency: up to 1,205 P90 tok/s/user on Rubin.

SemiAnalysis defines P90 interactivity as 1,000 divided by P90 full-response inter-token latency in milliseconds. The percentile is taken over latency before inversion, so this represents slower-tail streaming performance.

That metric excludes the initial wait before streaming. Matching tok/s/user therefore leaves time to first token unconstrained.

NVIDIA and Inferact submitted the initial Rubin result less than three weeks after the model's launch, SemiAnalysis said. Its chart compares Dynamo plus vLLM on both platforms, with configurations that include CPU KV offloading.

The cost model assumes ownership at large-hyperscaler volume, using SemiAnalysis's July 2026 pricing surveys. Rubin is priced at $3.61 per chip-hour, versus $2.31 for GB300, roughly 56% higher.

At those hourly rates, a 10.2× per-chip throughput ratio converts to approximately 6.5× throughput per TCO dollar. That dollar-normalized ratio uses a different denominator from the 6.5× per-chip throughput comparison at 376 tok/s/user.

The Rubin submission uses CUDA 13.4's locality-aware MoE feature, which raises realized HBM bandwidth by approximately 1.2×, according to SemiAnalysis.

vLLM describes the implementation around two locality domains, each pairing a group of streaming multiprocessors with nearby HBM:

• Weight placement: Split the expert weights column-wise for both FC1 and FC2, placing each shard in its corresponding domain's memory.
• Kernel placement: Use Green Contexts and CUDA streams to keep each domain's computation on its local weight shard.
• Activations: Leave the comparatively small input and output activations accessible across both domains.

MoE decode is dominated by reading expert weights in this setting. The placement scheme eliminates most cross-domain weight reads.

The October write-up also documents a resource-partitioning wrinkle: default equal partitions use 200 of Rubin's 212 SMs. Enabling cudaDevSmResourceGroupBackfill makes all 212 available.

vLLM says the latest results extend the first wave of optimizations published with M3.1-Flash support.

The linked October 23 implementation already documented several Rubin-specific paths:

• Indexer arithmetic: Convert NVFP4 directly to FP16 and perform the score matrix multiplication in FP16. This cuts dequantization instructions by about one-third and avoids the FP8 re-rounding used by the alternative path.
• Tiny decode batches: With four queries or fewer, the one-head indexer applies block scales per matrix-multiply result rather than per key value. It runs three thread blocks per SM, exploiting Rubin's larger shared memory and Tensor Memory capacity.
• Attention instructions: Prefill and decode attention use dense FP8 matrix-multiply instructions with K=64, halving the instruction count compared with the Blackwell path.

Those mechanisms belong to the earlier implementation. The new post additionally credits eliminating CPU overhead, without assigning separate end-to-end gains to each optimization.

AgentX reconstructs opt-in Claude Code traffic for the workload behind vLLM's comparison. Its published methodology and benchmark guide define controls that matter for interpreting the results:

• Private content: Session-scoped hashes in 64-token blocks preserve shared prefixes. Deterministic synthetic coding and tool-use tokens replace the original content during replay.
• Concurrency: The setting counts live agent clients. Subagent fan-out can put more HTTP requests in flight than the client count.
• Speculative decoding: Acceptance lengths come from SPEED-Bench's coding category for each model and speculative configuration, then are applied through engine controls. Synthetic text alone would not reproduce realistic draft acceptance.
• Measurement window: Prefix priming and ten additional warmup requests per replay lane precede a one-hour profiling window.

AgentX measures serving capacity, latency, cache behavior, and cost. Generated-code correctness and agent task success sit outside its scope.