LocalLLaMA users report near-6x Qwen 3.6 27B speedups with MTP

• Qwen 3.6 27B reached 321.5 tok/s on an 18-turn coding session, where the LocalLLaMA benchmark measured a 53.5 tok/s baseline and a 6.01x speedup from DFlash plus n-gram drafting.
• The near-6x number belongs to the full stack, while the same benchmark put MTP around 2.7x, DFlash alone at 3.71x, and DFlash plus ngram-mod at 5.68x.
• The llama.cpp Gemma 4 path now has copy-pasteable draft-mtp commands, with the Gemma 4 command screenshot covering 31B, 26B-A4B, 12B, E4B, and E2B GGUFs.
• Local serving work is splitting prefill, decode, memory placement, and power into separate levers: one LocalLLaMA post proposed remote prefill, another NUMA test kept KV and dense layers on GPU, and a power sweep traded 16% speed for 53W.

The llama.cpp speculative docs describe DFlash as a block-diffusion drafter and ngram-map-k4v as a token-history lookup. Modal's Inkling deployment note says a custom DFlash speculator beat the model's MTP path by 67% at matched throughput, while the SGLang GLM 5.2 post reports TopK-V2 scaling from 2.33x at 80K ISL to 10.17x at 1M. The benchmarker published Docker commands, sweeps, CSVs, and charts.

The benchmark used unsloth/Qwen3.6-27B-GGUF on an RTX PRO 6000 Blackwell, CUDA 13, f16 KV cache, 256k context for the speed runs, greedy decoding, and one server on the GPU at a time.

Decode throughput on the 18-turn coding session:

• Baseline: 53.5 tok/s.
• DFlash alone: 198.7 tok/s, 3.71x.
• DFlash plus ngram-map-k4v: 200.6 tok/s, 3.75x.
• DFlash plus ngram-mod: 304.3 tok/s, 5.68x.
• Full stack: 321.5 tok/s, 6.01x.

The maintenance turns, where the model edited existing code, reached 385 tok/s and 7.5x over baseline. By the end of the session, 92% of output tokens came from accepted drafts.

The n-gram part cost no extra GPU memory in that run. The author measured 33,256 MiB baseline, 38,810 MiB with DFlash, and 38,810 MiB with DFlash plus n-gram.

The same test found n-gram drafting useful only after the conversation had enough repeated context to copy from.

In the 100-prompt LiveCodeBench replay, plain DFlash beat the full stack on varied one-shot prompts:

• Plain DFlash: 176.1 tok/s per user, 47% draft acceptance, 30.5 ms time to second token.
• DFlash plus n-gram: 170.3 tok/s per user, 42% draft acceptance, 69.1 ms time to second token.

The author's explanation was simple: the n-gram cache starts empty. Iterative coding gives it repeated code and edits to copy, while unrelated prompts do not.

The posted commands all use llama serve --spec-type draft-mtp against Gemma 4 GGUF targets.

The underlying llama.cpp Gemma 4 PR says Gemma 4 shipped official Multi-Token Prediction assistant checkpoints, and the change wires those assistants into the existing draft-mtp speculative path. The PR also added Gemma 4 assistant GGUF metadata and CUDA flash-attention shape support for E4B assistant decoding.

Gemma 4 work was still moving around the model too. In osanseviero's Gemma 4 update, the listed follow-up items were Flash Attention 4 support, tool-use bug patches, and token-budget resources for vision.

The LocalLLaMA proposal treated prompt processing as something a bigger GPU machine could do once, then hand back to a weaker local box for decode.

The motivating calculation in segmond's post was a 50k-token coding prompt: at 10 tok/s prompt processing, prefill takes 5000 seconds, about 1.5 hours. The post estimated that a fast GPU server could process the same prompt in under a minute, with transfer and startup bringing the path under three minutes.

The proposed requirements were concrete:

• Exact same quant across machines.
• Fast internet.
• A threshold calculation for when remote prefill is worth the transfer.
• A rough implementation with curl, scp, and bash.
• A cleaner future flag shaped like --pp-server http://ppserver.

The production analogue is visible in vLLM's TileRT post, which describes prefill as compute-bound, decode as memory-bandwidth-bound, and TileRT decode as a connector behind stock vLLM prefill. In vLLM's launch note, the same split keeps the OpenAI-compatible surface and prefix caching while routing latency-bound traffic to TileRT decode.

In the linked Medium analysis, Park Chan-sung framed RTX PRO 6000 inference as a serving-economics question rather than a cheaper-H100 swap.

The analysis centered on workload shape:

• H100 remained the fastest BF16 option when workloads fully used its tensor path.
• RTX PRO 6000 value depended on traffic mix, batching, and request shape.
• NVFP4 helped under batching and concurrency, but did not uniformly improve single-stream or serialized long-context serving.
• The experiments emphasized a two-GPU RTX PRO setup, with long 65,536-token serving tests and 8192-token quality tests.

That matches the LocalLLaMA mood: the hardware question has shifted from peak tok/s to which bottleneck a workload actually hits.

The NUMA test came from a custom nmoe engine on a Dell T440 with dual Xeon Gold 6226 CPUs, 256GB RAM, an RTX 3080 20GB, and an NVIDIA A2 16GB.

The engine kept KV cache, dense layers, and shared expert layers on GPU. Experts on the NUMA nodes were tensor-split without duplicating data.

Reported numbers:

• Qwen 3.5 35B A3B: nmoe reached 1400 prefill tok/s and 71 decode tok/s, versus llama.cpp at 800-900 prefill tok/s and 45 decode tok/s.
• Qwen 3.5 122B A10B: nmoe reached 400-500 prefill tok/s and 37-40 decode tok/s, versus llama.cpp at 300-400 prefill tok/s and 20-21 decode tok/s.
• Qwen 3.5 397B A17B: the run hit CUDA OOM, then reported CPU-side 15-17 tok/s decode.

DFlash also showed up outside llama.cpp experiments.

Modal said it trained a custom DFlash speculator for Inkling on SGLang. Its deployment note reported 250 tok/s per user at 2.5M TPM of per-GPU throughput on 8x B200s, 67% faster interactivity than Inkling's built-in MTP path at matched throughput.

SGLang reported a separate GLM-5.2 NVFP4 serving path: 500+ tok/s/user at batch size 1 on 8x B300, 18-34% higher single-user interactivity within two weeks of day-zero support, and 6-11% better peak throughput at high concurrency. Its blog attributed the gains to Spec V2 overlap execution, IndexShare reuse, TopK-V2, indexer fusion, and GEMM kernel work.

Drafters are becoming workload-specific serving components.

The power thread tested P100 frequency settings on Darwin-36B and reported that sudo nvidia-smi -pm 1 -ac 715,1063 kept 84% of speed at 67% of power, a 25% tokens-per-joule gain.

The same post said a 150W power cap cost 0.3% on the workload because the cap rarely bound during layer-split serving. The last 265 MHz bought 16% more speed and cost 53W.

Community replies matched that pattern: QuixiAI said he limits 3090s to 250W, and his follow-up framed the reason as stability and heat.