Qwen3.8-Flash-Next runs from SSD on M4 Max at 40 tokens per second

• Full 4-bit Qwen3.8-Flash-Next weights reportedly ran with 37 GB resident memory and 40 tokens per second on an M4 Max after 60% of experts were streamed from SSD, according to EyalToledano's full-expert benchmark.
• The result depends on an unreleased pure-MLX engine, while EyalToledano's pMLX status update said pMLX was still non-public.
• A separate, concrete mlx-vlm change sends the 51B n-gram PLE table to disk, cutting an expert-pruned 4-bit build from 68.17 GB to 38.37 GB with bit-identical logits, as documented in EyalToledano's runtime update.
• Public REAP-288 artifacts trade some quality for footprint: EyalToledano's GGUF and BF16 release lists GGUF and BF16 builds, while the project reports 91.5% HumanEval versus 93.9% for the stock model.
• A 128 GB M5 Max test kept decode interactive through 169K tokens, but Artistic_Okra7288's M5 Max context test reported role confusion beyond roughly 100K tokens.

Qwen's official model card describes a 125B sparse backbone plus a 51B n-gram embedding table, the unusual memory layout behind these experiments. The disk-offload pull request reads only the needed n-gram rows through mmap. The 37 GB full-expert result belongs to a different layer of the stack, expert paging in pMLX.

Qwen3.8-Flash-Next carries 125B parameters in its main model but activates roughly 6B per token. Its n-gram embedding table adds 51B parameters that can sit outside accelerator memory, as the official model card explains.

The stored model is therefore far larger than its per-token compute budget. EyalToledano's architecture note describes 48 layers with 512 routed experts per layer, top-10 routing, and a 6B-active-parameter path.

• Three Gated DeltaNet layers alternate with one Qwen Sparse Attention layer.
• A Gated Residual expands the residual stream into four branches with gated reads and writes.
• The n-gram layer appears at layer 2, while MTP modules reuse sparse-attention indices during speculative decoding.
• The official checkpoint has a 262,144-token native context window and documents YaRN extension to 1M tokens.

The PLE table is already a tractable SSD-offload target because only a few hundred bytes are read per token. The mlx-vlm pull request puts the quantized table behind a row-granular NumPy memmap, then dequantizes a row on demand.

It exposes three opt-in surfaces:

• load_model(..., ple_on_disk=True) or load(..., ple_on_disk=True)
• mlx_vlm.server --ple-on-disk
• mlx_vlm.generate --ple-on-disk

The pull request records a stock 4-bit conversion falling from about 97 GB resident to 68 GB. Its REAP-288 test falls from 68.17 GB to 38.37 GB, with maximum absolute logit difference of 0.0 and the same argmax.

The original PR was closed after a maintainer said the work was covered by a pending release, and EyalToledano's follow-up said the 39 GB path would arrive with an mlx-vlm upgrade. This path offloads the PLE table only.

The 37 GB headline comes from paging experts as well as table data. EyalToledano's full-expert benchmark says 60% of the model's experts stream from SSD into memory on demand, retaining full weights rather than pruning them.

That benchmark reported 40 tokens per second on an M4 Max. EyalToledano's pMLX explanation attributes the result to a pure-MLX forward pass, custom or fused kernels, and renewed MTP gains after removing verification overhead that had made speculative decoding a poor trade on earlier chips.

The reported residency ratios expose the trade:

• 60% of experts on SSD: 37 GB resident and 40 tokens per second in the initial benchmark.
• 70% on SSD: an estimated 27 GB resident and about 20 tokens per second, according to EyalToledano's paging estimate.
• 40% resident in the attached test matrix: 37.3 GB with a drafter, 18.5 bare tokens per second, and 39.5 with MTP.

EyalToledano's tiered-streaming description describes a per-layer ledger that selects the experts to activate while the rest remain on NVMe. In an earlier comparison, EyalToledano's full-versus-pruned comparison reported that full experts beat a pruned model on both throughput and quality at similar residency, 26 GB and 19 tokens per second versus 23 GB and 15 tokens per second.

Later tuning reached 73 tokens per second in a short-context update from EyalToledano, while the author had also said long-context testing remained unfinished in EyalToledano's benchmark caveat.

The downloadable artifacts currently use REAP pruning, not the unreleased full-expert paging engine. REAP scores experts by saliency and removes low-use experts, taking each layer from 512 to 288 experts.

The REAP-288 MLX model card lists a 68 GB 4-bit artifact, 68 GB resident on stock runtime or about 39 GB with table streaming. It reports 91.5% HumanEval for REAP-288, a 2.4-point drop from the 93.9% stock result.

Published formats include:

• GGUF Q4_K_M, 78 GB
• GGUF Q5_K_M, 87 GB
• GGUF Q8_0, 116 GB
• BF16, 231 GB

The GGUF repository says its quants derive directly from the BF16 pruned source, avoiding a second quantization pass. Its stated calibration distribution is agentic coding, and post-pruning vision input remains unevaluated.

An independent LocalLLaMA Mac optimization report used SSD streaming for tensors, engrams, and MTP on an M1 Max. It reported a custom Q4 build, sparse attention, and dynamically disabling MTP when context size turned speculative decoding negative. The original 28-token-per-second M4 result was from a MacBook Pro, after EyalToledano's hardware correction fixed an earlier MacBook Air label.

Weights are only one part of the local memory budget. In a 3.5-hour test using a 79 GB 2-bit GGUF on a 128 GB M5 Max, Artistic_Okra7288's M5 Max context test configured a 358,400-token YaRN slot and reached 169,425 tokens in one conversation.

Cold prefill fell from 1,561 tokens per second at 5.6K context to 318 at 111K. Effective decode fell from roughly 30 to 35 tokens per second at short context to 11.5 at 169K, with prefix reuse keeping normal turns much faster than the one 333-second cold prefill after an idle gap.

The same tester reported role confusion past about 100K tokens despite FP16 KV cache and staying below the model's native 262K context, leaving the 2-bit quantization or preview-model long-context quality as suspects. Separately, QuixiAI reported that a Turboquant KV cache broke multi-turn conversation tracking and that switching the cache to BF16 fixed the observed failure in QuixiAI's follow-up.