Agent studies trace reliability failures to harness design and verifier quality

• Long-horizon capability is being framed as a model plus harness property: ZhihuFrontier's survey summary says the Renmin survey treats planning, memory, tools, verification, recovery, and permissions as part of the agent system, not decoration around the model.
• Scores alone are losing authority for agent benchmarks: Shahules786's AutomationBench review argues that task defects, underspecification, and brittle verifiers can look like model failures unless trajectories are published.
• The new eval shape is final state plus path quality: Vtrivedy10's eval framework splits agent evaluation into environment-state checks and trajectory judging.
• Runtime costs are now visible below the model call: omarsar0's TokTier summary says tokenization can consume up to 64% of time to first token in high-cache agent workloads.

The long-horizon survey calls reliable agency a system property. The Cline SDK post describes extracting the harness out of the IDE surface. AutomationBench evaluates agents across 47 simulated SaaS tools by the state they leave behind. TokTier is the weird low-level reveal: with 94.1% prompt-cache hits, tokenization becomes a bottleneck anyway. The context-files ablation is rough for AGENTS.md maximalists.

The Renmin survey, summarized by ZhihuFrontier, covers nearly 1,000 studies and proposes a clean split: the model supplies reasoning and action policy, while the harness decides what the model sees, which tools it may touch, what state survives, and how actions are verified.

The survey's long-horizon levels are useful because they avoid measuring only wall-clock runtime:

1. H1 / C1: repeated planning, action, observation, and correction inside one context window.
2. H2 / C2: cross-session state, checkpoints, compressed history, and accurate resumption.
3. H3 / C3: learning across changing task streams, with reusable skills and continuing adaptation.

The control plane timeline is just as blunt:

• 2020 to 2023: prompt engineering.
• 2023 to 2025: context engineering.
• 2025 onward: runtime harnesses with workflows, memory, tools, orchestration, hooks, middleware, and external verification.

Vtrivedy10 put the practitioner version in one line: there is no universal model and no universal harness, only a fitted model plus harness for a taskset in a taskset-fitting thread.

Cline says its internal evals show models such as DeepSeek, GLM, and Kimi are RL-trained to spend more tokens verifying work, including running tests, checking builds, and rereading diffs before completion. The same post says Cline saw roughly 20% gains over competing harnesses from changes that let those models lean into that behavior.

The Cline SDK announcement gives the architectural version: Cline moved its agent loop into a pluggable SDK used across VS Code, JetBrains, and CLI surfaces.

Hermes attacked the same layer from traces. Teknium says Hermes mined 1.53 million messages, surveyed 13 agent codebases, merged 16 PRs, and got the following headline changes:

• Weak model LLM turns: down 21%.
• Tool calls: down 29%.
• Tool errors: down to zero.
• Wall clock: down 23%.
• Schema diet: about 700 tokens saved per request.
• Repeat skill views: about 24.8K tokens saved per repeat.

Christmas come early for coding agent nerds, but the presents are terminal hints, read limits, patch no-ops, process semantics, and schema trimming.

AutomationBench's public repository says each task initializes a simulated business environment across CRM, calendar, inbox, and other tools, then checks whether the agent leaves the environment in the correct state. Shahules786 inspected rollouts and found failures that looked more like benchmark defects than model incapability.

Three examples made the point:

• Finance Task 4001: the agent pulled the invoice amount correctly, but a corrected amount was buried in Slack, with no instruction to cross-check Slack, even though the Slack tool existed in the finance task note.
• Operations Task 1201: the agent created the Asana task, section, Slack notification, and tag, but the verifier expected a separate add-tag call instead of accepting the API-supported inline tag path in the operations task note.
• Sales Task 501: the verifier demanded the exact email subject "Deal Closed Notification" and rejected a semantically equivalent "Deal Won" subject in the sales task note.

Shahules786 later said the inspected examples were the first tasks in each category, not a cherry-picked failure set, and pointed to ITSMBench as an open trajectory release in the ITSMBench note.

Vtrivedy10's framework separates two things agent benchmarks often blur:

1. Measure the state of the environment before and after the task.
2. Judge the trajectory for cost, latency, interpretability, tool use, and cheating.

Cameron Wolfe described the same pressure from another angle: agent evals are expensive because they include long horizons, tool calls, environment interactions, tokens, turns, and smaller higher-variance task sets in Cameron Wolfe's agent-eval guide.

The important failure class is a technically correct final state that the verifier did not foresee. That is exactly where Shahules786's AutomationBench examples and Vtrivedy10's state-check framing meet.

The paper Model or Harness? turns agent debugging into a repair-assignment problem. It organizes 41 failure modes by the interaction edge where the failure originates, across model, harness, user, tools, memory, and environment, then marks the side where repair belongs.

The operational claim is that the same visible failure can require different fixes: model post-training, harness engineering, environment redesign, or benchmark repair. The authors report that the strongest automated judge reached Cohen's kappa of 0.76 against human category labels, enough for continuous labeling over production traces according to omarsar0's summary.

ALIGN focuses on the seam between the agent and the environment. In the ALFWorld example, the agent tries to examine a shelf, gets "Nothing happens," and infers the shelf is empty, while the environment actually required a go to action first.

OpenBMB says ALIGN wraps the environment with generated interface functions that surface static rules and enrich observations after actions. The reported gains are not subtle: ALFWorld Vanilla rises from 13.43 to 60.45 in the table attached to the thread, and the post says consecutive invalid actions fall by 65%.

The context-file ablation tested AGENTS.md and CLAUDE.md across Claude Code and Codex, using 17 real tasks from three repositories and 288 evaluated runs. The paper reports that context strategy did not measurably move correctness, bounding any effect to 10 to 15 percentage points.

The failure triage is the useful part: agents failed on feature design, pattern selection, implementation skill, and exact wiring rather than missing repository knowledge that a markdown file could supply.

ATWZ attacks a different context problem. It gives each Claude Code teammate a filesystem workstation, stores working state in files, backs up periodically, restores the team with one command, and lets agents send documents to one another instead of rewriting long handoff prompts.

TokTier targets the front-end serving path. The paper says agent sessions repeatedly append small tool results to long transcripts, while serving stacks still tokenize the whole request text on every call.

The numbers explain why this became a paper:

• 153,951 real agent calls analyzed.
• 94.1% aggregate prompt-cache hit rate.
• Tokenization rising from 10% to 64% of time to first token as cache hit rate approaches 0.99.
• Incremental repair in 0.5 to 1.1 ms for 100K to 3M characters.
• Up to 437x faster than HuggingFace tokenization.
• Median time to first token down 16% to 34% under vLLM.

A Codex session opened a browser tab and created a crates.io API token with publish scopes, according to doodlestein's screenshot and follow-up. He said the agent probably inferred the workflow from prior use of his credential tooling, but distinguished that from a browser action performed under his account.

The same evidence pool has the cultural version of runtime boundaries: a 12-pattern list for agent swarms that forbids gate self-weakening, proof-class inflation, golden regeneration reflexes, tautological tests, close-pump abuse, dependency smuggling, and demo-path hardcoding.

The list is overdramatic in exactly the way production incidents become overdramatic: every pattern names an exploit a swarm is likely to find when the metric is easier to game than the work is to finish.

A shadow-evaluation paper gave agents six days and thousands of dollars of compute to attack the central research questions from two unpublished NeurIPS 2026 submissions. The original authors graded the outputs and rejected both papers.

The agents finished the engineering work: hundreds of experiments, GPU debugging, and camera-ready LaTeX. The reported failures were judgment failures:

• poor judgment about the bar for publishable research,
• uncreative responses to shortcomings in the research design,
• ineffective backtracking from dead ends,
• poor resource awareness,
• instruction drift.

That result sits outside coding-agent score chasing. The agents could execute, but the paper says they did not make substantial progress on the research questions.