Developer reports Qwen3.8-Flash-Next runs in 37GB on M4 Max

• A custom MLX build reportedly keeps the full Qwen3.8-Flash-Next Q4 model at 37GB of resident memory by streaming 60% of its experts from SSD, while generating 40 tokens per second on an M4 Max, according to EyalToledano's report.
• The preceding local route pruned the model from 512 to 288 experts, reaching 39GB and 91.5% HumanEval versus 93.9% for stock in EyalToledano's REAP-288 release.
• Server-side self-hosting is already using a different stack: svonava's SIE report pairs Qwen3.8 with EAGLE-3 speculative decoding on H100s.

An M1 Max test found that the model's n-gram table needs only 16 rows per token. The latest oMLX release added SSD-mapped PLE storage, continuous batching, prefix-cache preservation, and Lightning MTP for Qwen3.8-Flash-Next.

EyalToledano reports that his custom engine leaves 40% of the routed experts in memory and pages the other 60% from disk on demand. The 37GB figure is resident memory, while the complete Q4 expert set remains available rather than being structurally pruned.

The result is a single M4 Max developer measurement, but its distinction is unusually important: sparse MoE routing selects experts per token, so the build is attempting to trade storage latency for full-model quality without reducing the expert pool. The report puts decode at 40 tokens per second.

Before expert paging, the same work took the conventional route: REAP kept 288 of 512 experts per layer, reducing the Q4 model to 39GB resident memory. Its reported HumanEval result was 91.5%, against 93.9% for the stock conversion.

The author said 288 was the best quality-per-GB point in an 11-step pruning ladder. In a rare-name check, EyalToledano's pruning ladder reported intact output in 9 of 10 seeded generations at 288 experts, versus 1 of 10 at 256.

HumanEval is still the only reported benchmark, after about 14 hours of local testing, as EyalToledano's benchmark caveat noted. A later comparison put the full model at 26GB resident and 19 tokens per second, versus 23GB and 15 tokens per second for the pruned model at 91.5% accuracy, according to EyalToledano's follow-up.

Expert paging is separate from the model's built-in offload opportunity. Qwen3.8-Flash-Next combines a 125B main model with a 51B n-gram embedding table, while activating 6B main-model parameters per token; the SGLang implementation post describes host-memory offload and asynchronous prefetching for that table.

The n-gram table is addressed by local-context lookup, not MoE routing. mlx-vlm's disk-offload patch adds a ple_on_disk path that reads and dequantizes table rows on demand, while UnslothAI's local guide lists 75GB RAM for its separate GGUF route. Quantization, runtime, and what is offloaded differ across those figures.

The M4 Max result is not a stand-in for server throughput. svonava's SIE report describes an H100 profile with EAGLE-3 speculative decoding, 160 tokens per second at a 256K context window, and more than 1,000 tokens per second under concurrent load.

SIE also published profiles for multiple other GPU types. That setup is a serving-engine profile, rather than the custom SSD expert-paging path reported on the Mac.