Qwen 122B runs on older laptop with llama.cpp in user test

• A LocalLLaMA user got a 122B Qwen GGUF to run on a laptop, but the post measured 5 minutes to load, 2 minutes of prompt processing, and 14 minutes of generation.
• Slow output was treated as part of the point: one commenter compared local model work to hot-rodding, where the hobby is squeezing a machine under ugly constraints.
• A separate Radeon 7600 run pushed Qwen3.6 35B-A3B from 18 tok/s to 21 tok/s after a ROCm rebuild and VRAM overclock, according to the hardware post.
• Active-parameter MoE is winning the home-hardware argument: onusoz reported 60 decode tok/s on Qwen3.x-35B-A3B on GB10, while 27B dense hit about one-third of that.
• The 2.4T Qwen thread sat at the other extreme, with LocalLLaMA users asking how to run Qwen3.8-2.4T-A95B locally while NVIDIA described the same model as data-center-scale compute.

The laptop command used -ctk q4_0 -ctv q4_0, -c 8192, four threads, and --reasoning off, according to the run log. The Qwen3.5-122B-A10B model card puts the base model at 122B total parameters but 10B activated per token. Unsloth's Qwen3.6 notes say 35B-A3B fits in 23 GB at 4-bit, and llama.cpp's August 11 release notes moved release builds to ROCm 7.14, the same ROCm line that showed up in the Radeon tuning report.

The run used Qwen3.5-122B-A10B-Uncensored-HauhauCS-Aggressive-IQ4_XS.gguf through llama.cpp. The user reported a 5-minute model load, 2 minutes of prompt processing, and 14 minutes of generation.

The official Qwen3.5-122B-A10B card describes the model as a vision-language MoE with 122B total parameters, 10B activated parameters, 48 layers, and compatibility with Transformers, vLLM, SGLang, and KTransformers.

That active-parameter count explains why the run is not as absurd as the headline number sounds. The harder part is memory footprint, not per-token activation.

The most useful part of the LocalLLaMA post is the command, because it shows the trade-offs chosen to make the run fit:

• Model: Qwen3.5-122B-A10B-Uncensored-HauhauCS-Aggressive-IQ4_XS.gguf
• KV cache quantization: -ctk q4_0 -ctv q4_0
• Context: -c 8192
• CPU threads: --threads 4
• Reasoning: --reasoning off

A commenter in the same thread pointed to CPU-thread tuning as a live variable, not a solved default. Another said slow local inference can still be useful if the answer is good enough.

The faster report came from a Ryzen 5600 box with 64 GB DDR4 and a Radeon 7600. The user said Qwen3.6 35B-A3B-Q8_0 moved from 18 tok/s to upper 19 tok/s after rebuilding llama.cpp for ROCm 7.14, then to 21 tok/s after maxing the VRAM overclock.

The llama.cpp b10356 release notes say the project switched ROCm builds from 7.2.1 to 7.14 and moved Windows ROCm jobs to ROCm wheels.

The reported flags were the kind of setup detail local-model threads exist for:

• --n-gpu-layers 999
• --n-cpu-moe 36
• --no-mmap
• -ctk q8_0 -ctv q8_0
• -fa 1
• -c 9000

The oddest measurement was UI-dependent: the same post said watching tokens generate dropped output to 13 tok/s, while minimizing the window brought it back to 21 tok/s.

Onusoz argued that home inference is moving toward DDR or LPDDR memory plus MoE-like architectures, because smaller boxes and lower power draw fit more users than GPU clusters.

The concrete comparison was Qwen3.x-35B-A3B versus Qwen3.x-27B. On GB10, onusoz reported 60 decode tok/s for a single A3B session, or 6 parallel sessions at 20 tok/s, while the 27B model reached about one-third of that.

Unsloth's Qwen3.6 docs line up with the fit argument: Qwen3.6 27B is listed at 18 GB in 4-bit, while 35B-A3B is listed at 23 GB in 4-bit, with both supporting 256K context.

The 27B question has shifted from “can it fit” to “which speed path ships.” One LocalLLaMA post asked whether Qwen 3.8 27B would ship with a DFlash or MTP head, then cited about 8 tok/s for Qwen3.6 27B with MTP on 32 GB unified memory and a Radeon 780M.

That detail matters because the 27B model is described by users as stronger but slower than 35B-A3B. The trade-off is no longer parameter count alone, it is dense quality versus active-parameter throughput plus whatever speculative path the runtime can use.

A separate RTX 5080 comparison used a custom llama.cpp build with CUDA Toolkit 13.1, MSVC 19.44, native Blackwell PTX, CUDA flash attention, CUDA graphs, and sm_120a-real.

The user preferred Muse Glimmer 30B for precision and rule-following, but said Qwen3.6 35B produced richer voxel worlds with higher design quality. That is a useful split for local evals: throughput numbers do not settle creative-output preference.

LocalLLaMA immediately applied the same “how far can we push it” instinct to Qwen3.8-2.4T-A95B. The thread framed the target as a challenge for people who had already tried to wrestle 70B, Mistral Large, Goliath, DeepSeek, GLM, MiMo, and Kimi-class models onto local hardware.

NVIDIA's Qwen3.8-2.4T-A95B serving post gives the scale: 2.4T total parameters, 95B activated per token, up to 1 million tokens of context, up to 128K output, and a hybrid of full and linear attention.

NVIDIA also says deploying the model requires data-center-scale accelerated compute. Local hobbyists are already asking the consumer-hardware version of that question anyway.