vLLM adds NVFP4 serving for MiniMax M3.1 Flash at 76 concurrent sequences

• Four B300 GPUs fit 76 concurrent 256K-token sequences under DEP4, versus 28 for M3, according to vLLM's capacity comparison.
• Day-zero serving combines NVFP4 MoE weights, NVFP4 attention and indexer caches, and DSpark speculative decoding, as vLLM announced.
• AMD gets up to 2.3× faster low-concurrency decode on MI350X at TP4, according to vLLM's AMD results.

NVFP4's 4-bit cache values cost 4.5 bits once block scales are included. The Hopper example disables video while retaining images. Inferact is contributing production improvements back to vLLM.

MiniMax published the weights for M3.1 Flash, a 428B-parameter model with 23B parameters activated per token. Its model card lists text, image and video inputs with up to 1M tokens of context.

The memory comparison in vLLM's engineering post has three parts:

• Checkpoint: 451.5 GB for M3-MXFP8 → 250.1 GB for M3.1 Flash, 44.6% smaller.
• KV memory per token: 61,440 bytes with FP8 → 34,560 bytes with NVFP4, 43.75% lower.
• Resident sequences: 28 → 76 at 256K tokens on four B300 GPUs, about 2.7× the capacity.

The 76-sequence figure is a memory-fit comparison using data-plus-expert parallelism, DEP4. The launch quickstarts use tensor parallelism, TP4.

MiniMax Sparse Attention follows three steps, described in the kernel breakdown:

1. Divide the KV cache into 128-token blocks.
2. Scan the index keys across the context and select 16 blocks, always retaining the most recent block.
3. Run exact main attention over those selected blocks, 2,048 keys per query.

Every token's K and V remain resident in memory. The indexer also scans the full context on each decode step, so long-context serving still has a context-dependent memory-access cost.

Decode context parallelism splits indexer scoring across GPUs by context. Each GPU scores all four index heads over a quarter of the pages, exchanges local candidates, and merges the exact top-16 selection; vLLM reports up to 3.3× faster indexing per sparse layer at 512K on four GB300 GPUs.

Blackwell's decode path keeps queries in FP8 and converts cached NVFP4 values to FP8 inside the kernel. CUDA 13.4 supplies the QMUL4 instruction used for that conversion.

The changes span three separate operations:

• Attention decode: direct NVFP4-to-FP8 conversion replaces the FP16 round trip, yielding up to 1.7× faster indexer QK decode and 1.3× faster sparse-attention decode.
• MegaMoE: fused MXFP8 shared experts and earlier Tensor Memory release cut prefill kernel latency by 20% and raise overall prefill throughput by 7%.
• QKV fusion: projection, normalization, RoPE and cache insertion run together, avoiding an intermediate QKV write to HBM. vLLM reports about 19% lower median time to first token at TP4.

The DSpark draft ships inside the checkpoint's dspark/ directory, so the serving recipe requires no separate draft-model path. Its attention uses non-causal grouped-query attention with a 4,096-token sliding window.

Switching draft attention from FlashAttention 4 to FlashInfer PrimiTS makes that operation 1.3–2× faster, reducing total forward time by 5%, according to the engineering post.

The AgentX results occupy different latency/throughput operating points:

| Hardware | Latency-end decode tok/s/user | Throughput-end total tok/s/GPU | Decode tok/s/user at throughput end |
| --- | ---: | ---: | ---: |
| NVIDIA GB300 | 418.9 | 153,800 | about 127 |
| NVIDIA Rubin | 724.9 | 191,500 | about 134 |

The GB300 curve uses seven speculative tokens and synthetic acceptance lengths of 3.16–3.26 tokens, including the first token. The headline per-user and aggregate maxima therefore describe separate points on the serving frontier.

AMD's CDNA4 lacks a native NVFP4 dequantization instruction. vLLM's implementation precomputes scaled values in a 2 KB lookup table, then uses byte permutation and sign handling to convert FP4 values to FP8.

The AMD implementation also changes:

• Indexer scoring: context parallelism delivers up to 2.8× faster indexing per sparse layer on four MI355X GPUs, measured across 45 requests of 30K–210K tokens.
• Low-concurrency decode: one FlyDSL launch per sparse MoE decoder layer yields up to 2.3× faster decode on MI350X at TP4, doubling throughput for the measured 8K-input/1K-output workload.
• Prefill and disaggregation: sequence parallelism, dual-batch overlap and a contiguous KV layout deliver 1.8–2.1× faster prefill for batches of at least 16K tokens on four MI355X GPUs. Each KV block moves in one RDMA read rather than 125 small reads.

The documented MI355X AITER recipe re-quantizes NVFP4 experts to MXFP8 at load time while retaining NVFP4 caches.

Day-zero support ships in dedicated containers. The launch post says nightly PyPI availability follows upstream review and merging, typically about a week later.

• NVIDIA image: vllm/vllm-openai:minimax-m31-flash.
• AMD image: vllm/vllm-openai-rocm:minimax-m31-flash.
• Blackwell quickstart: CUDA 13.4 recommended, TP4, mandatory 128-token blocks, NVFP4 attention and indexer caches, and seven DSpark speculative tokens with FLASHINFER_TS and probabilistic draft sampling.
• AMD quickstart: TP4, NVFP4 caches, three speculative tokens and TRITON_ATTN.
• Hopper recipe: TP8 on H100/H200, FP8 attention and indexer caches, Marlin MoE kernels, and DSpark disabled.

Inferact says it has served M3.1 Flash production traffic since launch.

Day-zero RL support combines veRL, FSDP2 and LoRA, as vLLM announced. The training implementation starts from the NVFP4 checkpoint and updates only rank-32 adapters.

The trainer uses NVFP4 routed experts and MXFP8 for other projections, then synchronizes only adapters after each step. Expert LoRA runs inside FlashInfer's TRTLLM NVFP4 MoE kernel, preserving the serving kernel for the base experts.