vLLM reports 196 tokens per second for DeepSeek V4.1 Flash on AgentX

• vLLM reports 196 tokens/s per user at P90 and 954K total tokens/s/GPU for DeepSeek V4.1 Flash on AgentX, according to its benchmark announcement.
• The throughput includes cached input, fresh input and output. The 7.9× gain is against vLLM's earlier single-node result, as the team's thread explains.
• vLLM also tracks DeepInfer's published decode curve across batch sizes 8 to 256, using different parallelism, in its comparison with DeepSeek's engine.
• The reported $38.96/GPU-hour is revenue calculated at DeepSeek's off-peak token prices, per vLLM's pricing example.

Widening decode from 16 to 32 GPUs initially made it slower, a serving-stack horror story buried in the engineering write-up. The AgentX workload preserves coding-session fan-out and prefix reuse while replacing the original prompts with synthetic tokens.

vLLM's AgentX submission combines prefill/decode disaggregation with wide expert parallelism on a GB300 NVL72 rack. Inferact led the engineering, with NVIDIA collaboration, according to vLLM's announcement.

The engineering post defines the measurements and rack allocation:

• Throughput: cached input tokens + uncached input tokens + generated output tokens, per second per GPU.
• Interactivity: the chart labels this as the reciprocal of P90 inter-token latency.
• P/D configurations: multiple two-GPU prefill instances feeding one 16- or 32-GPU decode instance, using 34 to 64 GPUs within the rack.
• DEP terminology: DEPn means data-parallel attention plus expert parallelism across n GPUs.

AgentX reconstructs workloads from 393 opt-in Claude Code sessions, according to its methodology. Its concurrency setting counts live agent clients; subagents can push the number of in-flight HTTP requests above that setting.

The replay has several controls that shape these results:

• Session graphs: main-agent chains, subagent branches, joins and inter-turn delays preserve the recorded request structure.
• Prefix reuse: session-scoped hashes identify shared 64-token blocks. Deterministic synthetic coding and tool-use tokens replace the original content.
• Warm caches: a primer establishes active prefixes, followed by 10 warmup requests per replay lane before the one-hour measurement window.
• Speculative acceptance: forced-acceptance controls use measurements from SPEED-Bench's coding category, rather than acceptance behavior on the synthetic payloads.
• Closed-loop execution: faster servers progress farther through sessions during the measured hour, so the exact request mix can vary, especially at low concurrency.

Model answer quality falls outside AgentX's scope. The team's follow-up says the serving results were verified in the benchmark.

At 131K context with three DSpark draft tokens, vLLM reports decode throughput within 6% of DeepInfer across batch sizes 8 to 256 in its engineering comparison.

Published vLLM and DeepInfer curves, with hardware and parallelism differences documented beneath the charts.

The setups differ:

• Decode: vLLM uses 32 GB200 GPUs in DEP32. DeepInfer's published configuration is EP64DP64. The vLLM test fixes accepted tokens per step at 3.25.
• Prefill: vLLM uses GB300 DEP2, no redundant experts and a 128-token replay. DeepInfer uses EP4DP4, 32 redundant experts and a 140-token replay; its chart does not identify the hardware.
• Reference values: vLLM reads DeepInfer's results from DeepSeek's published charts.

DeepInfer was originally built on vLLM. vLLM has since integrated DeepSeek's open-source kernels, while the DeepInfer reference measured here is an offline engine.

DeepSeek V4.1 Flash's global KV cache occupies just 890 bytes per token, through cross-layer sharing and FP4 compression, according to vLLM's model analysis. A two-GPU prefill instance can hold the weights and cache for many concurrent sessions.

At prefill-only concurrency 256, DEP2 delivered 64.4K tokens/s/GPU versus 53.3K for DEP4. Separate lagged-rank experiments found throughput losses of 5.9% for DEP2 and 13.6% for DEP4 when ranks fell out of step.

vLLM then reduced prefill work with three changes, measured sequentially at 32K new tokens per rank after a 128K cached prefix:

1. Bounded replay through layer 20: write global KV for every new token, but run the layer's remaining computation on the last 128 replay tokens. Throughput increased 19.9%.
2. Larger Engram mappings: replace 64 KB GPU pages with 2 MB pages for shared host tables, reducing GPU TLB misses. Another 2.8% gain.
3. DeepSelect top-k: use DeepSeek's selection kernel to rank candidate blocks in the prefill indexer. Another 15.6% gain.

Together, those changes increased throughput 42.5% at that operating point.

At 128K context and 64 requests per rank in DEP4, MoE consumed 62% of the target-model step; attention and indexing together consumed 13%, according to vLLM's profiling. MegaMoE decode is weight-read-bound, so spreading experts across more GPUs reduces the dominant cost.

With expert load balancing enabled, widening DEP4 to DEP32 on GB200 reduced target MegaMoE time from 13.4 to 6.0 ms. The remaining work stayed around 10 ms, and draft-model communication began offsetting the gains at DEP32.

The expert load balancer tracks the target and draft separately:

• Target: 384 routed experts, top-6 routing; measured balancedness rose from 0.35 to 0.94.
• DSpark draft: 128 routed experts, top-3 routing; balancedness rose from 0.21 to 0.89.

Here, balancedness is mean rank load divided by maximum rank load. Experts are rearranged every 3,000 steps using the previous 1,000 steps of load history; an NVLink-based symm_mem backend rebalances DEP32 in under a second without stopping decode.

An unoptimized DEP32 configuration took 17.57 ms per step, versus 13.84 ms for DEP16, despite having 1.1 ms faster GPU execution. Every layer synchronizes ranks, allowing one late CPU worker to stall the instance, as vLLM documents in its overhead analysis.

The fixes came from separate experiments with different concurrency and topology settings:

| Bottleneck | Change | Reported effect |
| --- | --- | --- |
| Synchronous KV reads on the decode worker | NIXL push with UCX scatter-gather; packed KV rows sent as one region | 141K → 440K tokens/s/GPU; P90 interactivity approximately 40 → 159 tokens/s |
| Gloo all-reduce latency growing with rank count | Recursive-doubling CPU all-reduce | DP synchronization 10.8 → 1.5 ms per step |
| Bursts admitting up to 140 requests in one step | Opt-in cap of four new KV loads per step | 460K → 516K tokens/s/GPU; P90 109 → 153 tokens/s |
| Admission work growing with prompt length | One int32 prompt buffer, skipped null sliding-window blocks and chunked prompt copying | CPU admission time 38.8 → 6.0 ms for an 855K-token prompt |

To separate GPU work from serving overhead, the team used vllm bench forward through the actual execute_model path and DecodeBenchConnector for engine-level end-to-end decode measurements.

At DeepSeek's off-peak token prices, vLLM calculates $38.96 in revenue per GPU-hour in its announcement. That calculation prices the benchmark's cached-input, uncached-input and output mix.

Utilization and serving costs govern the resulting margin, according to deedydas, whose illustration assumes $5 per GB300 GPU-hour and model list prices.

Some Chinese model labs are also seeking revenue sharing from providers at scale or for pre-release access, deedydas reports, adding another cost beyond rented hardware.

Specialized small-batch decode kernels raised concurrency-one AgentX interactivity from 575 to 826 P90 tokens/s, a 44% increase, on four B300 GPUs in TP4, according to vLLM's low-latency measurements. Those tests also covered concurrency settings of eight and 16.