Practitioners propose a standard harness for agent benchmarks

• Agent scores combine model weights with tool schemas, transcript structure, and control-flow logic: ZhihuFrontier's analysis separates the resulting overfitting into format, context-structure, and control-flow failures.
• A common, stable runtime could yield a comparable fixed-harness score, the proposal in onusoz's post argues, while native vendor systems answer a separate performance question.
• NVIDIA reported that AVO with Claude Opus 5 reached 100.00 RHAE on ARC-AGI-3's 25-environment public set, a contrast imjustnewatai's breakdown places against a 30.16% baseline; fchollet's clarification says the public demonstration set is not the full benchmark.
• Runtime mechanics can also change the bill: reach_vb's rate-limit update attributed unexpected Codex usage to images in long sessions, Computer History, and conversation-title generation.

A controlled MCP-versus-CLI study ran one six-operation repository task across seven scaffolds and five models, then verified the repository state instead of trusting agent self-reports. Harness-Bench keeps task environments, budgets, and evaluation protocol shared across 106 sandboxed tasks while preserving native execution behavior. DeepSeek's own minimal-preset commit is titled “align minimal agent with RL composition.”

An agent harness supplies the system prompt, tool schema, history layout, result truncation, planning hooks, reflection timing, and stopping policy. The same underlying model can therefore retain task understanding while failing to operate once that surrounding interface changes.

The failure modes divide into three layers:

1. Format: a changed tool-call schema or syntax produces malformed calls or parsing failures.
2. Context structure: reordered history, truncated results, summaries, or compaction alter the model's next action.
3. Control flow: planning and stopping behavior relies on runtime primitives, such as a todo tool.

The DeepSeek Harness repository presents its runtime as a plugin architecture. Its minimal preset's explicit alignment with the RL composition makes “minimal” a description of training-interface fidelity, not merely tool count.

State also lives in project instructions and working notes. A study summarized by dair_ai tracked 94,813 development events across 557 coding sessions and found that instruction files and agent working notes accounted for 60.5% of documentation interactions, versus 10.6% for classical technical documents and 1.3% for API references.

kunchenguid's Backpass announcement treats a project-level AGENTS.md or CLAUDE.md as a trainable artifact, sampling past sessions and proposing evidence-backed edits for review. The author also called actual task-execution traces valuable in a reply.

NVIDIA's AVO result is the headline-sized version of the harness effect. Its loop wraps a general-purpose coding agent with:

1. Persistent memory across context windows.
2. An inspect, plan, implement, evaluate cycle.
3. Execution feedback.
4. A supervisor that detects stagnation and redirects work.
5. External transcript state for tool calls and artifacts.

NVIDIA's technical writeup reports all 183 public-set levels solved in 6,624 environment actions. daniel_mac8's follow-up called the harness delta the larger story than the perfect public score.

The 30.16% and 100.00 figures came from differing settings, so the comparison is system-level evidence rather than an isolated AVO ablation. fchollet's clarification adds that clearing the public environments leaves semi-private and private ARC-AGI-3 evaluation unresolved.

onusoz proposed a simple, stable common interface, naming mini-swe-agent, Terminus 2, and vanilla Pi as possible candidates. The point is a score for a known environment and known resource budget.

A onusoz follow-up draws the boundary directly: a fixed-harness result answers which model performs best through a common interface, while a native-system result identifies the strongest complete vendor system. Both measurements retain useful information when their leaderboard columns are labeled.

steipete argued that modern agents may depend on retained reasoning across turns, server-side compaction, and code-oriented tooling. A harness that removes those elements measures a model under a distinct body, not merely a neutral testing room.

Cost figures already expose harness behavior. The reach_vb update cited inefficient image handling in long sessions, high Computer History usage, and conversation titles while announcing a subscription-wide reset.

koltregaskes's reset timing put that reset at 2 p.m. Pacific, while reach_vb's support reply asked an affected user for a support ID. Those incident details turn cache behavior, compaction, auxiliary features, and accounting rules into variables alongside completion rate.

Harnesses also decide what behavioral data a model provider can observe. thinkymachines's announcement made Inkling and Inkling-Small free on OpenRouter only when used through agentic harnesses, saying it would use data disassociated from accounts to improve agentic performance.

Harness-Bench already supplies one concrete experimental shape: 106 offline, sandboxed tasks, six configurable harnesses, eight API model backends, and 5,088 model-harness trajectories under shared environments, budgets, and evaluation protocols. Its reported harness scores span 52.4% to 76.2% across tested configurations in the paper's main results.

The MCP-versus-CLI experiment supplies a narrower companion design: one fixed task, seven scaffoldings, five models, and repository-state verification. Its authors found scaffolding to be the dominant effect on cost, while the fixed task makes the interface comparison reproducible.