NVIDIA reports Rubin delivers up to 5.7× GB300 throughput per watt

• Rubin delivers up to 5.7× GB300’s throughput per megawatt at matched per-user decode speeds, according to ArtificialAnlys.
• Comparing each system at peak throughput gives smaller gains: 1.14× to 1.45× across four models in the initial results.
• DeepSeek V4.1 Flash reaches 906 output tokens/s per user on four GPUs in the Flash results.
• Headline comparisons use p25 output speed, as ArtificialAnlys explains; its full report finds GB300 matches or beats Rubin’s efficiency at most end-to-end speeds on DeepSeek V4 Pro.

The eye-popping token totals count cached inputs, which contribute about 96% of measured throughput. Flash’s Rubin runs also used a 2,048-token reply cap, versus 16,384 for the other Flash results.

NVIDIA submitted every Rubin result in Artificial Analysis’s October 15 release. The largest throughput-per-MW advantages occur at these per-user output speeds:

| Model | Rubin advantage over GB300 | Output speed per user |
| --- | ---: | ---: |
| DeepSeek V4 Pro | ~5.7× | ~190 tok/s |
| DeepSeek V4.1 Flash | ~5.4× | ~580 tok/s |
| GLM-5.2 | ~3.0× | ~165 tok/s |
| Kimi K3 | ~2.8× | ~67 tok/s |

Increasing DeepSeek V4 Pro’s per-user speed from roughly 120 to 165 tok/s costs GB300 81% of its throughput per MW, versus 16% for Rubin.

The GB300 comparison changes configuration along that curve. NVIDIA’s GB300 rack configurations top out around 163 tok/s per user; beyond that, AA uses smaller four-GPU configurations it built.

Comparing each system at its own throughput maximum produces these efficiency ratios. The maxima can occur at different per-user speeds.

| Model | GB300, million tokens/s/MW | Rubin, million tokens/s/MW | Ratio |
| --- | ---: | ---: | ---: |
| DeepSeek V4 Pro | 239 | 272 | 1.14× |
| DeepSeek V4.1 Flash | 395 | 571 | 1.45× |
| GLM-5.2 | 201 | 272 | 1.35× |
| Kimi K3 | 56 | 70 | 1.26× |

For Flash, Rubin reaches its peak at 68 output tok/s per user, versus 23 tok/s at GB300’s peak.

DeepSeek V4 Pro takes a median 1.65 seconds to return its first token on Rubin at roughly 165 output tok/s, versus 1.08 seconds on GB300. Including that wait changes the efficiency comparison: GB300 matches or beats Rubin at most end-to-end speeds.

Other model-specific limits in the same report:

• DeepSeek V4 Pro: Above roughly 255 output tok/s per user, AA’s four-GPU GB300 configurations deliver higher throughput per MW. They reach 340 tok/s, versus Rubin’s maximum of 298.
• GLM-5.2: Including first-token latency leaves Rubin with a 1.1× to 1.8× throughput-per-MW advantage across the measured range.
• Kimi K3: At peak throughput, p95 first-token latency is 9.3 seconds for Rubin’s 2,048-agent configuration and 19 seconds for GB300’s 768-agent configuration.

Three Rubin configurations set AA-AgentPerf leaderboard records as of October 15:

• DeepSeek V4.1 Flash: 906 tok/s on one four-GPU server serving one agent, versus GB300’s fastest 596 tok/s.
• GLM-5.2: 408 tok/s, versus GB300’s 330, with the new record reported in ArtificialAnlys’s GLM-5.2 results.
• Kimi K3: 255 tok/s on one eight-GPU vLLM server, versus GB300’s 194, with the record reported in ArtificialAnlys’s Kimi K3 results.

Flash’s lower response cap affected about 2% of replies. Average replies were roughly a tenth shorter on Rubin, which AA says modestly favors its throughput.

AA-AgentPerf replays coding-agent trajectories generated while resolving issues in public repositories. Each simulated agent completes three trajectories sequentially, pausing between requests for tool-call durations sampled from real runs.

The workload and metric definitions explain what these numbers measure:

• Throughput: Cached input tokens, newly prefilled input tokens and generated output tokens all count.
• Per-user speed: The 25th-percentile output rate across requests, excluding time to first token. Three quarters of requests exceed this rate.
• Power: GPU and HBM consumption reported through NVIDIA DCGM/SMI telemetry. CPUs, networking and cooling are excluded.
• Context: Requests reach 132K input tokens; a single turn can add up to 69K new tokens.
• Cache reuse: On average, 97% of a request’s input repeats the previous request, with tool pauses lasting up to five seconds.

The published workload statistics cover 500 trajectories and 18,997 requests, tokenized with o200k_base.

NVIDIA’s Vera Rubin NVL72 integrates 72 Rubin GPUs and 36 Vera CPUs. AA’s October 15 hardware comparison lists these changes from GB300:

| Specification | GB300 generation | Vera Rubin generation |
| --- | ---: | ---: |
| Memory per GPU | 288 GB HBM3E | 288 GB HBM4 |
| Memory bandwidth per GPU | 8 TB/s | 19.2 TB/s |
| NVLink bandwidth per GPU | 1.8 TB/s | 3.0 TB/s |
| Dense NVFP4 compute per GPU | 15 PFLOPS | 35 PFLOPS |
| CPU-to-GPU bandwidth per superchip | 0.9 TB/s | 1.8 TB/s |

Measured GPU power across the configurations on the performance frontiers ranges from 0.5 to 1.9 kW for Rubin, versus 0.3 to 1.0 kW for GB300.

Every Rubin configuration runs behind NVIDIA Dynamo. The configuration breakdown identifies the engines and checkpoints:

| Model | Engine | Weights |
| --- | --- | --- |
| DeepSeek V4 Pro | TensorRT-LLM | NVIDIA NVFP4 |
| DeepSeek V4.1 Flash | vLLM | DeepSeek MXFP4 release checkpoint |
| GLM-5.2 | TensorRT-LLM | NVIDIA NVFP4 |
| Kimi K3 | TensorRT-LLM; vLLM for one agent | NVIDIA NVFP4 |

NVIDIA described six techniques behind the submissions:

1. Memory-tier caching. Most configurations offload KV cache to CPU memory over NVLink-C2C, overlapping transfers with GPU work. DeepSeek V4 Pro prefill workers reserve 120 to 200 GiB of host memory for cache; Flash instead places its 196B-parameter Engram memory in CPU memory.
2. Session-aware routing. Dynamo keeps an agent’s turns on the same prefill worker and uses prefix-cache-aware routing for Pro, GLM and Kimi. Kimi’s recurrent linear-attention state requires snapshots at reusable boundaries.
3. Disaggregated prefill and decode. Separate workers prevent a large tool response from blocking other agents’ decoding. Pro scales to six four-GPU prefill workers and spreads experts across up to 32 decode GPUs.
4. Batch-tuned speculation. All four models use speculative decoding. Pro drafts three to seven tokens depending on concurrency; the configuration producing its 5.7× result drafts seven.
5. Rubin-specific kernels. NVFP4 checkpoints use tuned four-bit MoE kernels alongside architecture-specific fused kernels. Attention-kernel tuning spans variable input shapes rather than one fixed prompt length.
6. Host-side request handling. Dynamo’s router uses Rust. Pro and GLM add tokenization caching and streamlined transport; the other models pin workers to nearby CPU cores and memory.

AA describes the Rubin and GB300 configurations as very similar, attributing most of the difference to upgraded hardware alongside early Rubin-specific kernels and tuning.

A separate InferenceX submission claims Rubin delivers 4.23× GB300 NVL72’s performance per dollar using SGLang. That metric incorporates cost, whereas AA-AgentPerf’s headline comparison uses measured GPU power.

The submission remains a preview on Qwen3.8-Flash-Next, described in SemiAnalysis_’s follow-up as roughly 180B total parameters with 6B active. The linked InferenceX dashboard covers agentic and fixed-sequence inference; SemiAnalysis_ says a fuller public SGLang Rubin NVL72 submission with larger models, including GLM-5.3, is forthcoming.