Speculative decoding tests report acceptance drop from 0.71 to 0.18 after ~32K context

• Cross-provider speculative decoding hit a sharp long-context failure: acceptance fell from about 0.71 to 0.18 past 32K tokens while independently sampled next-token KL stayed stable, according to hoyasgirl25's r/LocalLLaMA post.
• A single RTX PRO 6000 run found DSpark still helped under heavy CPU offload: FantasticNature7590's benchmark reported roughly 15% to 17% faster generation, with the drafter faster in system RAM than VRAM.
• Apple Silicon tests moved too: A-Rahim's mlx-dspark run put Muse Glimmer 30B at 8.2 tok/s without speculation and 18 to 26 tok/s with acceleration, depending on content.
• DSpark's stronger result was throughput, not accuracy: ZhihuFrontier's vLLM comparison reported 2.45x to 2.55x baseline throughput for DSpark versus 1.96x to 2.09x for DFlash.
• New open model releases are shipping drafters as part of the model package: mervenoyann's Nemotron note says Nemotron 3.5 Lightning comes with DSpark, DFlash, and NVFP4 checkpoints, while Rohan Paul's Muse Glimmer summary measured DFlash at 233 tok/s on an RTX 5090.

The DeepSpec repo is a full stack for training and evaluating speculative drafters, not just a benchmark notebook. Meta's Muse Glimmer model card exposes a 52-layer dense decoder, a 1.8B ViT-G/14 perception encoder, and a 131K-token context window. NVIDIA published separate DSpark and DFlash checkpoints for Nemotron 3.5 Lightning, and the vLLM launch post lists speculative decoding as a first-class deployment feature.

The weirdest report was a black-box boundary failure. A local draft deployment and a third-party verifier endpoint used essentially the same tokenizer and model family, but acceptance collapsed after about 32K context: 0.71 to 0.18 in the post's numbers.

The author said tokenizer and version skew, BOS/EOS handling, sampling params, prompt canonicalization, and fp16 versus bf16 differences were already ruled out. The remaining suspect was position dependence: different RoPE scaling, effective position IDs, KV-cache behavior, or hidden provider-side preprocessing.

That failure mode is nasty because speculative decoding is sensitive to the served distribution, not just the model name. Stable independent next-token KL can coexist with bad draft acceptance if the verifier's long-context serving path diverges position by position.

The most useful local result came from DeepSeek V4 Flash 284B plus DSpark on one RTX PRO 6000 96GB. The 144.4GB model did not fit in VRAM, so the final split kept 21 expert layers on GPU and 19 expert layers in system RAM.

The run is reproducible in the GitHub repo. The headline numbers:

| Setup | Generation speed |
| --- | ---: |
| No drafter | 26.52 tok/s |
| DSpark in VRAM | 29.86 tok/s |
| DSpark experts in RAM | 31.16 tok/s |

The surprising result was memory placement. Moving the 10.15GB Q8_0 drafter into system RAM freed enough VRAM for three more target-model expert layers, and that beat keeping the drafter in VRAM.

Other findings from the same run:

• Speculative guesses: 3 guesses was the fastest point at 33.40 tok/s with 74.7% mean acceptance.
• Longer guesses lost: 4 guesses fell to 31.32 tok/s and 5 guesses to 30.16 tok/s.
• q8_0 KV cache: 256K, 512K, and 768K context stayed near 32 tok/s decode in the reported split.
• 1M context dropped to 29.22 tok/s because another expert layer had to move out of VRAM.
• Retrieval sanity checks worked around 240K and around 900K tokens in the author's tests.
• The LiveCodeBench sanity check was 28/30, with the author warning that the slice was older and easier.

Christmas came early for local-inference nerds: the optimal placement was not “put every helper in VRAM.”

A-Rahim reported Muse Glimmer 30B running on an M4 Pro with mlx-dspark: 8.2 tok/s without speculation, then 18 to 26 tok/s depending on workload. The reported speedups were 3.27x on math, 2.5x on code, and 2.22x on chat.

The memory trade-off was explicit. The 8-bit run peaked around 40GB, while the 4-bit run needed about 18GB and reached roughly 25 tok/s with about a 1.7x speedup.

Meta's official Muse Glimmer post says full precision would require more than 55GB, while 4-bit quantization shrinks the language model under 20GB and leaves room for KV cache, the perception encoder, and the DFlash drafter inside a 24GB or 32GB envelope.

The llama.cpp path had a small day-one wrinkle. mervenoyann's update said the linked prebuilt binary initially lacked Muse Glimmer support, and mervenoyann's follow-up added the missing --hf-repo-draft meta-models/Muse-Glimmer-30B-GGUF:dflash-kquant flag.

The core loop is still the classic speculative-decoding contract:

1. A cheap draft model proposes a block of tokens.
2. The target model verifies that block in one forward pass.
3. The system keeps the longest accepted prefix.
4. On rejection, it resamples from the leftover distribution.
5. The final output remains distributed like the target model.

wafer_ai framed draft quality as distribution overlap, not top-1 token prediction. In that framing, acceptance equals one minus total variation distance between draft and target distributions.

The DSpark twist is two extra pieces on top of a parallel drafter:

• A tiny sequential head adds local dependence between guessed tokens, so suffixes decay less.
• A survival-probability head plus hardware-aware scheduler decides how many draft tokens are worth verifying under current load.

DFlash takes the parallel-drafting side further by using target hidden states and block-style proposals. DSpark's bet is that a small amount of sequential structure and dynamic verification beats blindly sending the full draft to the target.

ZhihuFrontier summarized a vLLM test that compared DSpark and DFlash on the same Qwen3-4B target model. The setup used 8 NVIDIA A800-SXM4-80GB GPUs, DSpark block-7, DFlash block-16, and vLLM 0.26.0.

The throughput result was the clean part:

| Configuration | Throughput |
| --- | ---: |
| Baseline | 229 tok/s |
| DSpark, original setting | 584 tok/s |
| DFlash, original setting | 449 tok/s |
| DSpark, swapped setting | 561 tok/s |
| DFlash, swapped setting | 480 tok/s |

That put DSpark at 2.45x to 2.55x baseline throughput, while DFlash landed at 1.96x to 2.09x. In matched comparisons, DSpark stayed 1.17x to 1.30x faster than DFlash.

The accuracy numbers were weaker evidence. ZhihuFrontier noted that MMLU varied by about one point and GSM8K baseline runs shifted from 29.2% to 34.4%, which the post attributed to vLLM batch composition changing floating-point accumulation order even at temperature zero.

Speculative decoding moved from inference trick to packaged model feature this week.

Muse Glimmer bundled DFlash for local agent workloads. Rohan Paul's DFlash note said the RTX 5090 decode path moved from 74.9 tok/s to 233 tok/s, while M4 Max moved from 23.7 to 38 tok/s.

The benchmark picture was mixed. Rohan Paul's benchmark summary said Muse Glimmer led Gemma4-31B and Qwen3.6-27B on MCP Atlas at 75.5 versus 54.2 and 62.5, but Qwen3.6-27B was higher on OSWorld-Verified, TerminalBench 2.1, and SWE-Bench Verified.

The Rundown AI also tied the release to Meta's larger open-weight posture: Muse Spark 1.2 weights were promised “soon,” alongside Zuckerberg's “Future Is for Everyone” essay.

Nemotron 3.5 Lightning pushed the same pattern into a 30B MoE execution model. NVIDIA's technical blog describes a 30B model with 3B active parameters, speculative decoding, NVFP4 and BF16 checkpoints, and up to 4x output speed versus similar-sized models.

The ecosystem support landed immediately: mervenoyann pointed to DSpark, DFlash, and NVFP4 checkpoints; vLLM's launch post described OpenAI-compatible serving; and Ollama's post listed local launch targets for Claude Code, Hermes Agent, and OpenClaw.

The next systems problem is keeping the drafter aligned with real traffic. wafer_ai argued that every speculative verification pass already computes target logits for accepted and rejected draft tokens, then discards a useful training signal.

The thread quantified the sensitivity: with lookahead 5 and draft cost 0.15, acceptance drifting from 0.75 to 0.65 costs about 24% throughput. That gives the 32K cliff a production analogue: small acceptance drops compound quickly.

The proposed training loop uses target outputs without loading the target model into the trainer. In the Aurora-style setup described by wafer_ai, the inference server streams hidden states from three target layers plus top-1024 verifier logits, compressing about 256KB to 2KB per token.

The infrastructure hooks are starting to look real. The same thread points to vLLM hidden-state extraction, /collective_rpc for atomic draft reloads, SGLang's enable_aux_hidden_states and update_weights_from_disk, LMSYS SpecForge, and Snowflake Arctic's vLLM plugin recipes.