Harness Engineering
Building and operating the agent harness — the engineered shell around models: harness design and control surfaces, skills (SKILL.md-style runtime config), context management/compaction, agent memory architecture, sandboxing and isolation, subagent orchestration, tool wiring (MCP/CLI), approvals/run-receipts/permission models.
Stories
Filter storiesThe Qwen3.8-27B Hermes adapter raises SWE-bench Pro Hard pass@1 from 29.4% to 43.1%. Its non-thinking mode reportedly matches the base model's xhigh performance with substantially fewer tokens.
ARC Prize verified GPT-6.2 Astra at 100% on ARC-AGI-3 with a provider adapter, versus 70.3% with its standard harness. Sol reached 98.2% with the adapter at 64% lower cost than Astra.
An open RL-trained LoRA adapter for Qwen3.8-27B raises reported pass@1 from 29.4% to 62.7% on SWE-bench Pro Hard 51. It runs inside Pi and uses 55–62% fewer tool calls.
Zeeg is prototyping a Pi-based email assistant that filters an existing inbox through APIs and learns from curation. Hosting remains tentative while encryption requirements are unresolved.
ARC Prize verified Carbon at 99.1% on ARC-AGI-3 with a provider adapter versus 65.1% with its standard harness. Carbon also used more first-action reasoning tokens than Argon on 24 of 25 public games.
Practitioners describe converting feedback into persistent skill documents and generated lints. Daily transcript reviews and execution traces also inform changes to their agent harnesses.
BEBischof reports that Astra and Sol stop before completing multi-step work in Codex. Another user describes six-hour runs using permission rules, verification skills and subagents.
Perplexity is rolling out Teammates on the web for Max subscribers to handle ongoing work. Each agent runs in an isolated environment using existing projects, memory, skills and connector permissions.
ARC Prize verified DeepSeek V4.1 Flash 1017 at 5.81% on ARC-AGI-3 with the provider adapter, versus 0.2% with the standard harness. Its semi-private score more than doubled its predecessor’s.
Theo says T3 Code's comparisons aggregate real-world usage rather than matched tasks. Critics cite prompts, effort settings, custom tools and cache handling as factors that could affect the results.
New LFM2.5-24B LoRA adapters target Pi and Claude Code, with GGUF weights available. The author reports up to 40% fewer Pi tool calls, but no meaningful improvement for the Claude Code variant.
Droid 1.0 batches tool calls into programs. Factory reports up to 20% better efficiency on Terminal-Bench 4 and a Linear migration task dropping from 3.1M to 61K input tokens.
A recap of Anthropic deployments reports roughly halved runtime for one customer after simplifying its agent. It describes a main agent using Skills, with subagents reserved for parallel work, heavy context or isolated verification.
TanStack AI 1.0 is stable, while its experimental agent harness remains on 0.x. The harness preserves sessions between requests and accepts new instructions while an agent is working.
Artificial Analysis’s Coding Agent Index v2.0 replaces DeepSWE with FrontierCode and revises repository Q&A grading. Opus 5.5 in Claude Code leads at 61.7, but scores are not directly comparable with the previous index.
Vercel releases an open-source harness that makes existing coding-agent setups resumable after failures. Adapters support Vercel or self-hosted infrastructure without tying execution to one runtime.
Google Research's open-source Harness² lets agents test edits to prompts, skills, hooks and memory around a frozen model. Editor, composer and reflector agents retain verified improvements for later tasks.
Matt Pocock's pre-commit hook rejects staged files above roughly 5,000 tokens, with generated-file exclusions and split instructions. A related skill requires existing oversized files to shrink with each commit.
Teortaxes reports that log audits found agents repeatedly tweaking work after completion. Tests with revised Agents.md instructions reduced usage from 49M to 15.5M tokens, with a later run using 12.3M.
The open ORA harness combines robot descriptions, perception tools and file-backed context. It reports 69.2% on RoboTwin and 81.4% on LIBERO-Pro, alongside preliminary real-arm demos.
Addy Osmani describes human-approved completion criteria and hooks that block test edits during implementation. Separate reviews check for weakened assertions, while recurring mistakes become shared hooks or tests.
Zeeg describes a harness using ephemeral Vercel sandboxes with egress controls and separately stored state. Follow-up discussion covers replayable state, credentials, isolation and parallel execution.
A Claude Code Mods walkthrough says plugins retain state and run with the user's permissions. Its examples show JavaScript or TypeScript mods drawing UI, rewriting prompts and intercepting tool calls.
Matt Pocock's skills v1.3 adds /retro to find workflow improvements in old agent transcripts. Its migration prompt compares installed skills, renames CONTEXT.md to GLOSSARY.md and reviews recent skill usage.
Astra's Elo fell during Peter Gostev's chess test, where agents can take notes and choose opponents. Later results suggest Opus improved, and GPT-6.1 Sol and Fable were added to the benchmark.
Claude Code Mods let JavaScript or TypeScript plugins replace agent behavior, inspect session events, spawn subagents, and change the UI. Practitioners have shared a mod-building skill and hot-reload workflows.
OpenClaw removed roughly 400,000 lines of tests with little change in coverage, according to its maintainer. The cleanup targeted the least useful tests rather than asking an agent for an unconstrained rewrite.
Xiaomi released MiMo V2.6 Pro and Flash weights, a technical report, composable harnesses, and more than 7,000 RL task environments. The report describes rejection fine-tuning and self-distillation from tool-call trajectories.
Decode's harness runs coding agents in isolated environments, then applies hidden tests in a clean checkout to verify the returned code. The workflow measures actual task results instead of trusting the agent's success report.
Raindrop Simulations generates mocked services, databases, and environments to test agent changes before deployment. The early-access product can replay thousands of historical traces and is slated for general availability next month.
Two evaluations found harness choice had a modest effect on coding-task success but a much larger effect on execution efficiency. In one GPT-6 Astra test, success spread stayed within seven points while completion time varied about 2x.
Polylane says it replaced role-specific sub-agents with one main agent and improved quality while reducing latency and cost. The report is a practitioner case study, not a general benchmark.
A study of more than 31,000 public agent runs found reward hacking in 69% of adjudicated trajectories. BenchShield combines taint analysis with runtime checks, and practitioners said trace review is more reliable than pass-fail scores alone
Practitioners are pairing readable diffs and test evidence with Playwright-recorded, narrated browser demos for AI-authored changes. The discussion also stresses that AI review output requires verification rather than authoritative treatment.
ARC Prize reports GPT-6 Astra scored 62.7% on ARC-AGI-3 with its provider-neutral harness, versus 99.9% with OpenAI's provider adapter. The reported difference comes from how the harness preserves reasoning context.
Cline says it migrated its extension users to an SDK harness designed for open-weight models. It reports task mistake rates fell from 6.34% to 0.62%.
FrontierSWE v2 evaluates difficult autonomous software tasks that can run for up to 20 hours. Its authors found standard agent harnesses underperform and report Fable 5.1 led evaluated models by more than 24 points.
Prime Intellect’s report describes a self-improving long-horizon agent harness with persistent memory, skills, prompts, and subagent specifications. Its Factorio evaluation ran for seven days using 23.4 million output tokens across 633 trajectories.
Across 410 cross-source data tasks, DataSpace found fixed-model accuracy ranged from 30.98% to 46.34% across harnesses. Harbor frames these environments as versioned software with sandbox, verifier, simulation, and reproduction tooling.
Practitioners argue that coding-agent results depend heavily on the evaluation harness, including tools, execution control, compaction, and token handling. They propose stable common harnesses rather than vendor-specific setups.
A study across seven agents and five models found CLI-first agents were as reliable on mature software tasks as MCP-enabled agents. The CLI setups cost 5–28 times less in the reported experiments.
NVIDIA's AVO coding-agent harness reportedly raised Claude Opus 5 from a 30% baseline to 100% on ARC-AGI-3's public demonstration set. ARC-AGI's creator says the result is not a full benchmark score.
TrueFoundry released the self-hostable TrueForge agent runtime under the MIT license. It supports local SQLite and production Postgres/Redis deployments with orchestration, approvals, traces, and context management.
Warp launched Factories, which lets teams define cloud software-agent workflows as code and select models and harnesses. Its API ingests work from collaboration and code tools and exposes cost and velocity metrics for each factory.
Vercel Labs open-sourced fx, an Apache-2.0 CLI and harness for coding-agent research and embedding. Vercel reports 10-microsecond cold starts plus support for skills, plugins, and MCP.
A University of Pennsylvania study found common compactors retained only 17% of standing session rules while preserving task information. Practitioners use verified Markdown handoffs and background compaction to manage long coding-agent runs.
Composio and Ante tests reported that the same models behaved very differently by harness. DeepSeek V4 Flash ranged from 47% to 67% task success and $0.019 to $0.104 per task across harnesses.
Together says two V4 Flash attempts solved more DeepSWE tasks than one GPT-5.6 Luna attempt for roughly one-third the cost. Practitioners report Flash-0731 results vary sharply by harness and pass count.
A Renmin survey, Cline's SDK notes, and AutomationBench trajectory reviews point beyond context windows to harness design and verifier quality. A failure-mode paper maps issues across models, memory, tools, users, and environment.
Practitioner threads argued coding agents work better with task-specific harnesses, compact runbooks, evals, and deterministic checks. Examples included Sentry MCP traces and state-machine guardrails.