Agent builders trace tool failures to harness schemas and retries

• AlphaSignalAI's harness summary framed the agent harness as the runtime that calls the model, dispatches tools and state, retries failures, and logs actions; the linked paper screenshot describes that layer as the model's operating system.
• TheTuringPost's MCP summary says MCP itself added almost no latency, 0.01 ms over stdio and 0.39 ms over HTTP, while tool-selection accuracy fell below 90% after roughly 10 to 30 tools.
• Better Models: Worse Tools described newer Anthropic models inventing extra fields for Pi's edit tool schema, a concrete failure mode at the model and harness boundary.
• the production effort-level test found high reasoning leaked a forbidden revenue total once in four runs, while low effort was slower than medium on a heavy database agent because it under-planned and looped.
• the cache analysis modeled Claude Code subagent prompt cache costs at 14% too high, mostly from static prompt context and parent caches expiring during child runs.

The 603-page Hitchhiker arXiv paper gives the harness its cleanest definition: orchestrator, context manager, memory layer, tool executor, observability, and state store. The MCP architecture paper adds a practical ceiling for tool counts, plus five server patterns that look more like backend architecture than prompt craft. The weirdest bug report came from Simon Willison's Better Models: Worse Tools: stronger Claude models became worse at one third-party edit schema, apparently because they had learned a neighboring harness too well.

The boring layer got interesting. The paper screenshot in AlphaSignalAI's linked follow-up defines an agent harness as runtime infrastructure around an LLM, with six components:

• Orchestrator
• Context manager
• Memory layer
• Tool executor
• Observability
• State store

_avichawla's layer breakdown separated the stack into prompt engineering, context engineering, harness engineering, and loop engineering. In that framing, prompt and context tune one model call; the harness handles tool definitions, parsing, retries, routing, and verification around that call.

The practical split from the Hitchhiker paper thread is workflows versus agents: a workflow runs written steps, while an agent decides its own next step in an observe, reason, act loop.

In Better Models: Worse Tools, Simon Willison summarized Armin Ronacher's Pi bug: Opus 4.8 and Sonnet 5 sometimes called Pi's edit tool with invented extra fields in nested edits[], causing Pi to reject otherwise-correct edits.

The reported pattern was narrower than generic malformed tool use. Older Claude models did not show it, while the newer models did. Ronacher's theory, as summarized in the writeup, was that recent Anthropic models had been trained to use Claude Code's baked-in edit tools, then generalized the wrong schema into Pi.

badlogicgames on edit tools added the tool-designer side: Pi's edit tool is multi-edit, so one successful call can save several provider round trips; Codex-style patch tools have similar benefits, but many models generate bad patches. Tool shape is now part of model selection.

The MCP paper surfaced a simple constraint: do not expose every possible operation as a flat tool list. TheTuringPost's thread reported that tool-selection accuracy drops below 90% after roughly 10 to 30 tools, depending on model.

The paper's five architecture patterns are useful as a naming system:

1. Resource Gateway: exposes existing data sources through MCP.
2. Tool Orchestrator: coordinates multiple tools behind a simpler interface.
3. Stateful Session Server: keeps conversation or workflow state across requests.
4. Proxy Aggregator: combines multiple MCP servers behind one endpoint.
5. Domain-Specific Adapter: wraps a complex external API in LLM-friendly tools.

The same thread said MCP overhead was tiny in the paper's measurements: 0.01 ms over stdio and 0.39 ms over HTTP. The bottleneck moved back to network calls, API design, and tool selection.

One Reddit user ran 26 known production questions across low, medium, and high effort, about 80 model calls total. The access-control tests used the same role identity injection as the production agent, so the guardrail path was real rather than a unit-test stand-in.

The results split by job type:

• Heavy database agent: one duplicate-hunting query took around 187 seconds on low effort and around 100 seconds on medium because low effort under-planned and recovered through extra tool loops.
• Access-control agent: low and medium refused a forbidden company-wide revenue request, while high returned the forbidden total once in four runs.
• Final configuration in the post: routing on low, access enforcement on low, heavy database work on medium.

petergostev's table showed the same dial expanding runtime and tool use: low ran 73.2K tokens, 27 tool calls, and 12 minutes; max ran 367.1K tokens, 122 tool calls, and 1 hour 50 minutes.

Claude Code's 2.1.199 changelog reads like a harness failure catalog. The detailed changelog listed fixes for discarded partial streaming output, subagents cut off by rate limits, subagents reporting API errors as successful results, background daemons self-killing on corrupted worker records, and transient 429 retries for subscribers.

The next release moved safety defaults. The 2.1.200 changelog changed default permission mode to Manual across CLI, VS Code, JetBrains, and help output; it also made AskUserQuestion stop auto-continuing by default unless configured with an idle timeout.

Claude Code 2.1.201 then removed mid-conversation system-role harness reminders for Sonnet 5 sessions, a small example of the harness changing once the model needed fewer reminders.

A Reddit analysis parsed roughly two weeks of local Claude Code transcripts: about 95 sessions, 1,800 subagents, and 6.8B input tokens. The author modeled subagent prompt cost at about 14% too high, or roughly 8% of total spend for that workflow.

The failure shape was specific:

• Subagent cold starts were around 37K tokens.
• Only about 950 tokens, roughly 3%, were the actual task.
• The remaining 97% was mostly static boilerplate: system prompt, tool definitions, and project rules.
• Dynamic fields such as date, cwd, branch, and reminders appeared early enough to break prefix-cache reuse.
• Parent caches expired while waiting on child subagents for more than five minutes.

The modeled fixes were also specific: a one-hour TTL write before child dispatch saved about 6%, while putting the identical per-type static prefix behind a one-hour TTL and moving dynamic content later saved about 7.6%. The same post said that naive one-hour caching for all subagents made costs 8.6% worse.

OctoTools is the cleanest research example in the evidence pool: a training-free agent harness for complex reasoning that plugs in tools, routes tasks, and composes tool-use workflows without additional model training. lupantech said it became an ACL 2026 Oral, Top 3.4%, with 1.5K GitHub stars and 100+ citations.

The product side is converging on similar portability. cramforce described AI SDK's HarnessAgent as one interface for Pi, Claude, Codex, OpenCode, and other harnesses, using official SDKs underneath so subscription-backed tokens still work.

browser_use's CLI 3.0 launch is the browser version of the same move: direct Chrome DevTools Protocol control, cloud or real Chrome, and a 6x smaller CLI with fewer tokens than the old wrapper layer.

Harness reliability is getting its own data plane. alexgshaw launched harbor exec as an agentic map-reduce CLI for executing and aggregating agents in sandboxes to analyze traces, mine sessions, search, and more.

[Vtrivedy10's LangSmith note] framed coding-agent evaluation as three infrastructure problems: storing and rerunning evals, understanding failure modes across rollouts, and turning production failures into new evals. The same post said LangSmith and Harbor had integrated for large-scale containerized evals with reproducible traces.

alexgshaw on Harbor evals put the trust argument plainly: in the era of agents, evals are the source of truth, and reproducible evals are how claims become auditable.