Vals tests: four models retain at least 80% of scores at half cost or less

• Four of seven models retained at least 80% of their scores at half the cost or less in ValsAI's study. All seven scored worse overall at their lowest reasoning setting.
• Cheaper settings matched or exceeded leaderboard scores in 22 of 56 model-and-benchmark pairs, according to ValsAI's results.
• Extra reasoning delivered median gains of 19.2 points on Legal Research and 15.6 on Code Migration in ValsAI's benchmark breakdown.
• Perfect-hindsight setting selection added a median 6 points at 64% of leaderboard cost in ValsAI's calculation, with settings chosen after seeing each task's results.

Gemini 4 Argon's middle setting cost less than its lowest setting and scored slightly above its leaderboard run. The Vals Index weights scores by sector GDP, while the study's cost figure averages its eight benchmarks equally.

Vals AI's November 3 study tested one model from each of seven labs at three reasoning levels: lowest, middle, and the configuration used on its leaderboard, usually the highest available.

Every percentage below is relative to that model's own leaderboard result. Opus and Grok delivered the sharpest discounts, both running at 16% of baseline cost.

| Model | Lowest → leaderboard effort | Score retained | Cost paid |
| --- | --- | ---: | ---: |
| Gemini 4 Argon | low → high | 99% | 84% |
| DeepSeek V4.1 Flash | low → max | 89% | 50% |
| GPT-6 Astra | low → max | 88% | 18% |
| Grok 4.7 | low → xhigh | 86% | 16% |
| GLM-5.3 | low → max | 81% | 53% |
| Claude Opus 5.5 | low → max | 80% | 16% |
| Muse Spark 1.3 | minimal → xhigh | 77% | 29% |

GLM narrowly missed the half-cost threshold; Muse fell below 80% score retention. Gemini's existing leaderboard runs specified no effort level, so Vals used high for this experiment.

The median score increase from lowest to highest reasoning effort varied substantially across the eight benchmarks:

• Legal Research: +19.2 points.
• Code Migration 1.1: +15.6 points.
• Terminal-Bench 4: +12.1 points.
• Tax Agent Bench: +11.5 points.
• Vibe Code Bench: +10.6 points.
• Finance Agent 2.1: +5.0 points.
• HLAB 1.1: +3.3 points.
• EMB 1.1: +3.3 points.

None of the cheaper configurations scored significantly higher than the leaderboard setting. Of the cheaper settings tested, 33 scored significantly lower, according to the study's statistical tests.

The 22 matches count individual model-and-benchmark pairs. Across the aggregate Vals Index, zero models matched their leaderboard score at lower cost using their lowest setting.

Vals retrospectively selected the cheapest setting that attained the best observed score on each individual task. That tie-breaking rule rewards lower cost whenever multiple settings achieve the same result.

Always using the lowest setting cost a median 29% of leaderboard pricing, compared with 64% for the hindsight-selected runs. For this calculation, Vals used plain averages across the eight benchmarks, rather than the Index's GDP-weighted score.

The reported 6-point gain is an upper bound: the selector used each task's own results to make its choice. An actual routing system would have to choose before those results existed; the study did not demonstrate such a router.

Every setting ran on the same fixed subset, covering about half of each benchmark's tasks, according to Vals's methodology.

• Subset selection: Vals chose subsets that preserved the model ranking from the full benchmark. Most benchmarks had only one run per setting.
• Run dates: Most leaderboard-setting runs came from earlier months; the other settings ran in October. Provider-side changes between runs could affect the comparisons. Gemini's leaderboard-setting runs were all fresh.
• Significance testing: Vals used paired bootstrap tests, Benjamini–Hochberg correction, and an adjusted p-value threshold below 0.05. With half-sized samples, a nonsignificant difference can still conceal a small real effect.
• Pricing: Costs use provider list prices. Code Migration costs were estimated from token usage.