DeepSeek releases DeepInfer serving engine for V4.1 Flash

• DeepInfer opens DeepSeek's V4.1 Flash inference engine under Apache 2.0, with its vLLM origins acknowledged by the vLLM team.
• The public release provides offline inference and omits several production serving components, according to a breakdown of the release.
• DeepSeek reports 155 trillion served tokens per day and $38 in revenue per GPU-hour at off-peak API prices in figures quoted by swyx.
• vLLM subsequently reported decode performance within 6% of DeepInfer in its follow-up benchmarks.

A 100 PiB SSD cache with days-long prefix retention is the engineering prize here. DeepSeek also runs production without tensor or context parallelism, while its deterministic-sampling mode requires disabling variable-length speculative verification.

DeepInfer packages optimized kernels, model execution, sampling, distributed inference, speculative decoding, and structured outputs. DeepSeek says future models' inference code and subsequent optimizations will also land in the project.

The repository specifies:

• Model: DeepSeek-V4.1-Flash, with deepseek-ai/DeepSeek-V4.1-Flash-1017 as the default checkpoint.
• Accelerators: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
• Build requirements: Linux, GCC 13+, Python 3.12, and CUDA 13 or CANN 9.2.
• Kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels.

DeepSeek credits vLLM as the framework's primary starting point, with additional inspiration from SGLang and TensorRT-LLM.

QuixiAI said the implementation would inform its SlimServe work in its response to DeepSeek.

DeepSeek explicitly lists three components outside the public release in the README:

1. A KV cache system.
2. Vision-encoder, prefill, and decode disaggregation, although most required engine functionality is implemented.
3. Load balancing across data-parallel replicas for prefill and decode.

DeepSeek's production architecture includes separate load balancers and external KV cache storage.

The deployment Q&A calls 64 prefill GPUs plus 64 decode GPUs the smallest efficient deployment unit under its recommended Wide-EP configuration. Smaller configurations are deployable.

Published engine curves were measured on four SM100 GPUs for prefill and 64 for decode, each stage within one scale-up domain. Ascend production performance remained an open question in teortaxesTex's follow-up.

The Ascend figures circulating in teortaxesTex's comparison come from a separate 32-chip CANN test using DSpark with five draft tokens and forced expert-parallel load balancing. Those settings differ from DeepInfer's published NVIDIA decode benchmark.

DeepSeek's production figures distinguish freshly computed input, cache-hit input, and generated output. Cache reuse is included in the served-input throughput accounting.

DeepSeek's reported production throughput, latency, cache hit rate, and revenue.

At the reported roughly 1:1 prefill-to-decode GPU allocation, applying off-peak tariffs to the rounded throughput figures yields $39.10 per average GPU-hour in a calculation shared by teortaxesTex. That is within 3% of DeepSeek's stated $38.

“Off-peak” refers to DeepSeek's API prices, as Yuchenj_UW clarified. The approximately 82% gross-margin estimate from Yuchenj_UW assumes $7 per GPU-hour in rental costs and notes that the production hardware was unspecified.

DeepSeek's $38 disclosure is revenue; storage, networking, and other operating costs are not itemized alongside it.

DeepInfer supports balanced context parallelism, but DeepSeek's production Q&A says neither context nor tensor parallelism is currently used in its deployment.

The Q&A describes three separate balancing mechanisms:

• Prefill routing: Requests fall into four context-length buckets: below 16K, 16–64K, 64–256K, and at least 256K. Chunk sizes come from a fitted quadratic compute-cost model and a per-step time budget.
• Decode routing: Placement balances request count and total KV cache length simultaneously, accounting for both batch size and attention work.
• Expert placement: GPUs retain redundant copies of hot experts. Recent routing statistics periodically trigger expert reassignment, with weights shuffled over NVLink.

DeepSeek reports achieving roughly 80% of a perfectly balanced offline benchmark through its production load-balancing system.

DeepSeek separates cache state into three storage lifecycles in its deployment documentation:

• GPU memory: In-flight block KV cache and state KV cache.
• Host memory: Sliding-window attention state, kept as a single copy and fetched across machines over a 400G VLAN when needed.
• Dedicated 3FS storage: Persistent block KV cache on roughly 100 PiB of SSD capacity, retained for three days for agent traffic and seven days for chatbot traffic.

Any prefill worker can retrieve any cached prefix. That removes prefix-affinity routing from the compute load balancer and lets storage scale independently of inference workers.

Cache fetching runs asynchronously into host shared memory. For swap-in to GPU memory, DeepSeek uses an RDMA NIC to drive PCIe transfers without consuming GPU streaming multiprocessors, overlapping transfers with layer execution.

DeepSeek reconstructs approximate sliding-window attention state from a short suffix, according to the deployment Q&A:

1. Reuse persistent block KV: A missing sliding-window cache does not require rebuilding the whole cached prefix.
2. Replay 128 recent tokens: Attention is restricted to the replayed segment, producing approximate SWA state at a cost independent of the full context length.
3. Shorten decoder reconstruction: Exact recovery would process the last 2,560 tokens through 20 decoder layers. Bounded replay processes the last 128 instead, which DeepSeek says nearly halves total prefill computation.

The engine's default configuration retains a 140-token suffix. That setting covers the attention window plus speculative backtracking; it is distinct from the Q&A's 128-token replay window.

DSpark drafts up to five tokens and predicts their acceptance probabilities. Confidence-scheduled verification chooses how much of that draft prefix to verify based on expected throughput.

DeepSeek reports a 6% production throughput gain over always verifying five draft tokens in the technical account summarized by ZhihuFrontier. CPU batch scheduling also overlaps device execution by default.

The README exposes the controls and their tradeoffs:

• Variable verification: DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE controls maximum draft length, defaulting to five.
• Batch invariance: DEEPINFER_BATCH_INVARIANT=1 keeps results identical across different batch compositions, on NVIDIA only.
• Deterministic sampling: Batch invariance enabled, variable-length verification disabled, and a fixed seq_seed in SamplingParams, also NVIDIA only.
• Constrained generation: SamplingParams.so_args accepts regex, JSON Schema, or EBNF grammar constraints.

vLLM reported decode throughput within 6% of DeepInfer across batch sizes 8–256 in its comparison. The test fixed 131K context and three DSpark draft tokens; vLLM used 32 GB200 GPUs against DeepSeek's published 64-GPU SM100 curve.

vLLM's decode measurements track DeepInfer's published curve under the stated benchmark settings.

vLLM attributed the close results to shared DeepSeek kernel libraries, including MegaAttn, MegaMoE, and Mega-mHC.

A separate AgentX result on GB300 reached $38.96 in estimated revenue per GPU-hour at P90 interactivity of 196 tokens per second per user. Its 954K tokens-per-second-per-GPU figure combines cached input, fresh input, and output.

The team explicitly cautioned that AgentX revenue estimates and DeepSeek's production revenue figures use different workloads.

DeepInfer defaults to the Flash 1017 checkpoint. Its model-quality evaluations are separate from the engine throughput measurements.

First-party

• Terminal-Bench 4.0: 31.2 → 42.3, +11.1 points, in DeepSeek's evaluation table.
• Automation-Bench: 54.8 → 60.5, +5.7 points, in the same company evaluation.

Third-party evaluators

• Artificial Analysis Intelligence Index: 39 → 45, +6 points, according to ArtificialAnlys.
• AA-Omniscience non-hallucination rate: 4% → 33%, +29 points, according to ArtificialAnlys.

DeepSeek's Terminal-Bench evaluation used the Claude Code framework. The model's advertised context window is 1M tokens; the engine's published throughput curves use 131K context.

DeepSeek separately released deepseek-recipe, a Rust library collection with Python bindings, to reproduce its official API's encoding and decoding behavior.

The MIT-licensed repository covers:

• Protocol conversion: Messages, Chat Completions, and Responses requests become a shared conversation representation, then model prompts and token IDs. Outputs convert back into complete or streaming responses.
• Control-token handling: Encoding separates genuine structural tokens from identical-looking user text, preventing special-token injection during tokenization.
• Tool handling: V4.1 supports deferred tool loading in Messages and client-executed tool search in Responses.
• Structured outputs: Schemas are validated and rendered, while the inference backend determines whether decoding constraints are enforced.

The caller supplies model inference, tool execution, and HTTP transport. The default image fetcher connects to any address; the library exposes a DestinationFilter hook for restricting destinations.