Unsloth reports UD3.1 beats GSQ-RCO in a revised 35.1B-token quantization test

• UD3.1 leads GSQ-RCO by roughly 10-20% in Unsloth’s revised DeepSeek GGUF tests, according to its announcement.
• The rerun used 35.1B tokens, with 512K context for agentic tasks and 128K for general tasks, replacing the earlier 32K setup described in the benchmark update.
• More sampling also lowered some of Unsloth’s own scores: Aider Polyglot got worse with additional runs, as a follow-up acknowledged.

GSQ-RCO IQ1_M can ace a 30-question AIME run about 43% of the time in Unsloth’s analysis, despite getting 51 of 1,920 sampled answers wrong. The Terminal Bench rerun also excludes seven of 89 tasks because they could not pass on Unsloth’s infrastructure.

Unsloth’s November 2 update covers Dynamic V3.1 preview GGUFs for DeepSeek-V4.1-Flash-1017. The team changed four parts of its evaluation:

1. Context: 512K for agentic tasks and 128K for general tasks, up from 32K.
2. Coverage: Added unsaturated agentic benchmarks, including DeepSWE, Terminal Bench and Automation Bench.
3. Sampling: Increased benchmarking from 1.3B to 35.1B tokens, about 27x more. AIME25 received 64 samples per question; DeepSWE and Terminal Bench received three runs each.
4. Grading: Used Math-Verify, revised answer graders, gpt-6-luna at xhigh for HLE, and official graders for benchmarks including Terminal Bench through Harbor and DeepSWE.

For the baseline, Unsloth says it ran the original weights directly, using DeepSeek’s released FP4/FP8 checkpoint.

Two different recovery calculations appear in the comparison:

• Unsloth’s overview: An unweighted average of accuracy recovery across 14 benchmarks, according to its methodology.
• GSQ-RCO’s advertised recovery: The sum of AIME25, GPQA-Diamond and LiveCodeBench v6 scores divided by the baseline’s sum, following its model card.

Using the second definition, Unsloth’s detailed rerun table reports:

| GSQ-RCO quant | Card claim | Measured recovery, 95% CI | p versus claim |
| --- | --- | --- | --- |
| IQ1_M | 98.70% | 97.24% [96.23, 98.21] | 0.0013 |
| IQ2_XXS | 100.00% | 98.38% [97.52, 99.22] | 0.0001 |
| IQ2_XS | 99.80% | 99.54% [98.88, 100.18] | 0.21 |

The claim comparison uses a one-sided paired bootstrap over questions. IQ2_XS remains compatible with its advertised recovery under Unsloth’s test.

The announcement’s prose swaps the two IQ2 claimed recoveries and repeats 98.38% for IQ2_XS. The table above follows the model card and Unsloth’s detailed tables.

DeepSWE v1.1 and Terminal Bench 2.1 both ran through mini-swe-agent. Their reported step settings differ:

• DeepSWE: A 500-step cap; the longest observed run reached 334 steps.
• Terminal Bench: A reported 150-step setting.

The sharpest caveat is the denominator: seven of 89 Terminal Bench tasks were excluded because they could not pass on Unsloth’s infrastructure, leaving an 82-task evaluation. Unsloth discloses that exclusion in its benchmark notes.

Unsloth acknowledged in a follow-up that its Aider Polyglot scores worsened with more trials. Its updated 128K LiveCodeBench estimates incorporate six to 12 additional runs beyond the original single run.

Unsloth says it requested ISTA-DASLab’s benchmark protocol and quotes a now-deleted Reddit reply telling it to reproduce the numbers itself.

ISTA-DASLab’s model card specifies reasoning effort 50 but leaves repetition counts and grading configuration unspecified. Its GSQ and RCO repositories provide quantization implementations.

Unsloth has promised its own benchmark code and said it will upstream its Math-Verify fixes.