METR introduces expenditure horizon for cost-aware agent evals

• METR's new eval unit is a crossover budget: agent and human performance are plotted against spend, and the expenditure horizon is where human labor becomes more cost-effective, as METR's launch thread defines it.
• The metric is built for continuously scored optimization tasks where token cost, experiment compute, and human labor all change the comparison, a cost-normalization problem METR's follow-up calls out.
• NanoGPT is the proof point: METR's example puts the best model horizons around $2K-$3K, with a public-challenge overfitting caveat.
• The human baseline changed by three orders of magnitude on correction: METR's correction says the intended estimate was $2.5K per 1% optimization, not the $2,500K wording in the original example.

The official METR post prices the whole agent run, not just tokens. The NanoGPT speedrun repo makes the task concrete: train a 124M model to 3.28 FineWeb validation loss on 8 H100s, now under 90 seconds. Toby Ord's earlier cost note is the background itch: horizon charts without run cost leave engineers guessing whether longer autonomous work is actually getting cheaper.

METR defines expenditure horizon as the budget where an agent's improvement equals the improvement a human would buy with the same dollars, according to the official post. In the schematic, the horizon is the x-axis distance from the starting point to the intersection of the agent curve and the human curve.

The ingredients are simple:

1. Agent curve: performance as agent spend rises.
2. Human curve: expected returns from paid human labor.
3. Crossover: the point where the cheaper source of progress flips.

The cost target is broader than API tokens; METR's follow-up names experimental compute and agent tokens alongside human cost. That is the useful shift for agent evals where running experiments can dominate the bill.

METR's time-horizon page measures task duration at a reliability threshold. Expenditure horizon keeps the human comparison, but changes the x-axis to dollars and uses continuous scores instead of a binary pass or fail.

METR's claimed advantages in the official post are threefold:

• Continuous scores can detect model differences with fewer observations.
• Frontier optimization problems make the score easier to map to AI R&D usefulness.
• Scaling curves expose how much extra spend buys before the agent plateaus.

The NanoGPT speedrun repo defines the benchmark as training a 124M model to 3.28 FineWeb validation loss on 8 H100s. METR picked record #78, a March 2026 state with 85.56 seconds of training time, as the starting point for six high-expenditure agent runs, according to the official post.

The corrected human curve in METR's correction is about $2.5K per 1% optimization. METR got that from two interviews with prolific NanoGPT contributors, which implied roughly 16 hours of labor at $150/hour for a one-percentage-point gain, according to the official post.

The chart in METR's NanoGPT example gives these horizons:

• GPT 5: $0
• GPT 5.2: $840
• GPT 5.5: $2,347
• Opus 4.1: $0
• Opus 4.5: $613
• Opus 4.8: $3,333

The raw and re-validated lines in METR's chart are not a cosmetic detail. METR's official post says raw trajectories tracked cumulative best scores before re-validation, which overstated progress because of statistical noise.

The re-checks found:

• GPT-5 and Opus-4.1 showed no increase over baseline after re-validation.
• GPT-5.5 and Opus-4.8 showed significant improvements.
• For the best-performing models, only about 70% of contributions looked mergeable to the maintainer.

The harness was expensive by construction: METR's official post says agents had continuous access to 4 H100 nodes for validation. Experiment compute made up about 70-90% of the cost of most trajectories.

That detail fits the cost frame in METR's follow-up: an agent eval can look different once experiment spend sits on the same axis as tokens and labor. METR said a more optimized harness would likely lower cost for a given optimization, but horizontally shifting the curves did not look likely to raise the maximum speedup achieved.

The whole metric depends on the crossover shown in METR's schematic. METR's official post lists several failure modes that do not show up in the summary chart:

• Hybrid human-plus-AI optimization is outside the metric.
• Lumpy returns make the crossover hard to measure.
• Agent work already baked into the starting state shortens later agent horizons.
• If a model lab trained directly on the benchmark, frontier usefulness can be overstated; METR says OpenAI and Anthropic have not publicly disclosed whether they train on NanoGPT.
• If agent returns stop diminishing faster than human returns, the intersection disappears and expenditure horizon is no longer well-defined.