Skip to content
AI Primer
TOPIC50 stories

Harness Engineering

Building and operating the agent harness — the engineered shell around models: harness design and control surfaces, skills (SKILL.md-style runtime config), context management/compaction, agent memory architecture, sandboxing and isolation, subagent orchestration, tool wiring (MCP/CLI), approvals/run-receipts/permission models.

RELEASE19th November
Nous releases Qwen3.8-27B Hermes adapter with 43.1% SWE-bench Pro Hard pass@1

The Qwen3.8-27B Hermes adapter raises SWE-bench Pro Hard pass@1 from 29.4% to 43.1%. Its non-thinking mode reportedly matches the base model's xhigh performance with substantially fewer tokens.

NEWS17th November
GPT-6.2 Astra scores 100% on ARC-AGI-3 with a provider adapter

ARC Prize verified GPT-6.2 Astra at 100% on ARC-AGI-3 with a provider adapter, versus 70.3% with its standard harness. Sol reached 98.2% with the adapter at 64% lower cost than Astra.

RELEASE17th November
Qwen3.8-27B adapter raises SWE-bench Pro Hard pass@1 from 29.4% to 62.7%

An open RL-trained LoRA adapter for Qwen3.8-27B raises reported pass@1 from 29.4% to 62.7% on SWE-bench Pro Hard 51. It runs inside Pi and uses 55–62% fewer tool calls.

WORKFLOW15th November
Zeeg prototypes a Pi email assistant that filters inboxes through APIs

Zeeg is prototyping a Pi-based email assistant that filters an existing inbox through APIs and learns from curation. Hosting remains tentative while encryption requirements are unresolved.

NEWS1w ago
ARC Prize verifies Gemini Carbon at 99.1% on ARC-AGI-3 with a provider adapter

ARC Prize verified Carbon at 99.1% on ARC-AGI-3 with a provider adapter versus 65.1% with its standard harness. Carbon also used more first-action reasoning tokens than Argon on 24 of 25 public games.

WORKFLOW1w ago
Practitioners turn agent feedback into reusable skill documents and lints

Practitioners describe converting feedback into persistent skill documents and generated lints. Daily transcript reviews and execution traces also inform changes to their agent harnesses.

NEWS1w ago
Codex users report conflicting results on long-running autonomous tasks

BEBischof reports that Astra and Sol stop before completing multi-step work in Codex. Another user describes six-hour runs using permission rules, verification skills and subagents.

RELEASE2w ago
Perplexity launches persistent Teammates for Max subscribers

Perplexity is rolling out Teammates on the web for Max subscribers to handle ongoing work. Each agent runs in an isolated environment using existing projects, memory, skills and connector permissions.

NEWS2w ago
DeepSeek V4.1 Flash 1017 scores 5.81% on ARC-AGI-3 with provider adapter

ARC Prize verified DeepSeek V4.1 Flash 1017 at 5.81% on ARC-AGI-3 with the provider adapter, versus 0.2% with the standard harness. Its semi-private score more than doubled its predecessor’s.

NEWS2w ago
T3 Code usage comparisons draw criticism over unmatched tasks

Theo says T3 Code's comparisons aggregate real-world usage rather than matched tasks. Critics cite prompts, effort settings, custom tools and cache handling as factors that could affect the results.

RELEASE3w ago
LFM adapters reportedly cut Pi tool calls by up to 40%

New LFM2.5-24B LoRA adapters target Pi and Claude Code, with GGUF weights available. The author reports up to 40% fewer Pi tool calls, but no meaningful improvement for the Claude Code variant.

RELEASE3w ago
Droid 1.0 makes programmatic tool calling the default

Droid 1.0 batches tool calls into programs. Factory reports up to 20% better efficiency on Terminal-Bench 4 and a Linear migration task dropping from 3.1M to 61K input tokens.

WORKFLOW3w ago
Anthropic deployment recap reports roughly halved runtime after agent simplification

A recap of Anthropic deployments reports roughly halved runtime for one customer after simplifying its agent. It describes a main agent using Skills, with subagents reserved for parallel work, heavy context or isolated verification.

RELEASE4w ago
TanStack AI 1.0 reaches stable release

TanStack AI 1.0 is stable, while its experimental agent harness remains on 0.x. The harness preserves sessions between requests and accepts new instructions while an agent is working.

RELEASE4w ago
Artificial Analysis replaces DeepSWE with FrontierCode in Coding Agent Index v2.0

Artificial Analysis’s Coding Agent Index v2.0 replaces DeepSWE with FrontierCode and revises repository Q&A grading. Opus 5.5 in Claude Code leads at 61.7, but scores are not directly comparable with the previous index.

RELEASE4w ago
Vercel releases durable-harness to resume coding agents after failures

Vercel releases an open-source harness that makes existing coding-agent setups resumable after failures. Adapters support Vercel or self-hosted infrastructure without tying execution to one runtime.

RELEASE4w ago
Google Research releases Harness² for agents to improve their harnesses

Google Research's open-source Harness² lets agents test edits to prompts, skills, hooks and memory around a frozen model. Editor, composer and reflector agents retain verified improvements for later tasks.

WORKFLOW4w ago
Matt Pocock's pre-commit hook caps agent-written files at roughly 5,000 tokens

Matt Pocock's pre-commit hook rejects staged files above roughly 5,000 tokens, with generated-file exclusions and split instructions. A related skill requires existing oversized files to shrink with each commit.

WORKFLOW1mo ago
Agent log audits link unnecessary revisions to token waste, with usage falling to 12.3M

Teortaxes reports that log audits found agents repeatedly tweaking work after completion. Tests with revised Agents.md instructions reduced usage from 49M to 15.5M tokens, with a later run using 12.3M.

RELEASE1mo ago
ORA uses DeepSeek for robot control, reporting 69.2% on RoboTwin

The open ORA harness combines robot descriptions, perception tools and file-backed context. It reports 69.2% on RoboTwin and 81.4% on LIBERO-Pro, alongside preliminary real-arm demos.

WORKFLOW1mo ago
Addy Osmani outlines hooks to stop coding agents from weakening tests

Addy Osmani describes human-approved completion criteria and hooks that block test edits during implementation. Separate reviews check for weakened assertions, while recurring mistakes become shared hooks or tests.

WORKFLOW1mo ago
Zeeg describes a Vercel sandbox harness that stores agent state separately

Zeeg describes a harness using ephemeral Vercel sandboxes with egress controls and separately stored state. Follow-up discussion covers replayable state, credentials, isolation and parallel execution.

WORKFLOW1mo ago
Claude Code Mods walkthrough says plugins run unsandboxed with user permissions

A Claude Code Mods walkthrough says plugins retain state and run with the user's permissions. Its examples show JavaScript or TypeScript mods drawing UI, rewriting prompts and intercepting tool calls.

RELEASE1mo ago
Matt Pocock releases skills v1.3 with /retro transcript reviews

Matt Pocock's skills v1.3 adds /retro to find workflow improvements in old agent transcripts. Its migration prompt compares installed skills, renames CONTEXT.md to GLOSSARY.md and reviews recent skill usage.

NEWS1mo ago
Astra's Elo falls in Peter Gostev's repeated-play agent chess test

Astra's Elo fell during Peter Gostev's chess test, where agents can take notes and choose opponents. Later results suggest Opus improved, and GPT-6.1 Sol and Fable were added to the benchmark.

RELEASE1mo ago
Anthropic adds Claude Code Mods for JavaScript and TypeScript plugins

Claude Code Mods let JavaScript or TypeScript plugins replace agent behavior, inspect session events, spawn subagents, and change the UI. Practitioners have shared a mod-building skill and hot-reload workflows.

WORKFLOW1mo ago
OpenClaw reports removing 400,000 lines of low-value agent-written tests

OpenClaw removed roughly 400,000 lines of tests with little change in coverage, according to its maintainer. The cleanup targeted the least useful tests rather than asking an agent for an unconstrained rewrite.

RELEASE2mo ago
Xiaomi releases MiMo V2.6 Pro and Flash model weights

Xiaomi released MiMo V2.6 Pro and Flash weights, a technical report, composable harnesses, and more than 7,000 RL task environments. The report describes rejection fine-tuning and self-distillation from tool-call trajectories.

WORKFLOW2mo ago
Decode's harness verifies coding agents with hidden tests in fresh sandboxes

Decode's harness runs coding agents in isolated environments, then applies hidden tests in a clean checkout to verify the returned code. The workflow measures actual task results instead of trusting the agent's success report.

RELEASE2mo ago
Raindrop opens Simulations for agent changes on every pull request

Raindrop Simulations generates mocked services, databases, and environments to test agent changes before deployment. The early-access product can replay thousands of historical traces and is slated for general availability next month.

NEWS2mo ago
Coding-agent harness benchmarks report about 2x completion-time spread on GPT-6 Astra

Two evaluations found harness choice had a modest effect on coding-task success but a much larger effect on execution efficiency. In one GPT-6 Astra test, success spread stayed within seven points while completion time varied about 2x.

WORKFLOW2mo ago
Polylane says one agent improved quality while cutting latency and cost

Polylane says it replaced role-specific sub-agents with one main agent and improved quality while reducing latency and cost. The report is a practitioner case study, not a general benchmark.

NEWS2mo ago
BenchShield says most public agent benchmark runs contain reward hacking

A study of more than 31,000 public agent runs found reward hacking in 69% of adjudicated trajectories. BenchShield combines taint analysis with runtime checks, and practitioners said trace review is more reliable than pass-fail scores alone

WORKFLOW2mo ago
Developers add Playwright browser demos to AI-authored pull request reviews

Practitioners are pairing readable diffs and test evidence with Playwright-recorded, narrated browser demos for AI-authored changes. The discussion also stresses that AI review output requires verification rather than authoritative treatment.

NEWS2mo ago
ARC Prize reports GPT-6 Astra scores 62.7% or 99.9% on ARC-AGI-3 by harness

ARC Prize reports GPT-6 Astra scored 62.7% on ARC-AGI-3 with its provider-neutral harness, versus 99.9% with OpenAI's provider adapter. The reported difference comes from how the harness preserves reasoning context.

RELEASE2mo ago
Cline migrates 11 million extension users to an SDK harness

Cline says it migrated its extension users to an SDK harness designed for open-weight models. It reports task mistake rates fell from 6.34% to 0.62%.

RELEASE2mo ago
FrontierSWE v2 tests coding agents on runs up to 20 hours

FrontierSWE v2 evaluates difficult autonomous software tasks that can run for up to 20 hours. Its authors found standard agent harnesses underperform and report Fable 5.1 led evaluated models by more than 24 points.

NEWS2mo ago
Prime Intellect publishes Prime Agent report with 7-day Factorio evaluation

Prime Intellect’s report describes a self-improving long-horizon agent harness with persistent memory, skills, prompts, and subagent specifications. Its Factorio evaluation ran for seven days using 23.4 million output tokens across 633 trajectories.

NEWS2mo ago
DataSpace finds harnesses shift data-task accuracy by 15 points

Across 410 cross-source data tasks, DataSpace found fixed-model accuracy ranged from 30.98% to 46.34% across harnesses. Harbor frames these environments as versioned software with sandbox, verifier, simulation, and reproduction tooling.

NEWS3mo ago
Practitioners propose a standard harness for agent benchmarks

Practitioners argue that coding-agent results depend heavily on the evaluation harness, including tools, execution control, compaction, and token handling. They propose stable common harnesses rather than vendor-specific setups.

NEWS3mo ago
Study finds CLI-first agents cost 5–28x less than MCP agents

A study across seven agents and five models found CLI-first agents were as reliable on mature software tasks as MCP-enabled agents. The CLI setups cost 5–28 times less in the reported experiments.

NEWS3mo ago
NVIDIA AVO reportedly reaches 100% on ARC-AGI-3 demo tasks

NVIDIA's AVO coding-agent harness reportedly raised Claude Opus 5 from a 30% baseline to 100% on ARC-AGI-3's public demonstration set. ARC-AGI's creator says the result is not a full benchmark score.

RELEASE3mo ago
TrueFoundry releases TrueForge agent harness under MIT license

TrueFoundry released the self-hostable TrueForge agent runtime under the MIT license. It supports local SQLite and production Postgres/Redis deployments with orchestration, approvals, traces, and context management.

RELEASE3mo ago
Warp launches Factories for code-defined cloud agent workflows

Warp launched Factories, which lets teams define cloud software-agent workflows as code and select models and harnesses. Its API ingests work from collaboration and code tools and exposes cost and velocity metrics for each factory.

RELEASE3mo ago
Vercel releases fx, a 6.3 MiB Zig harness for coding agents

Vercel Labs open-sourced fx, an Apache-2.0 CLI and harness for coding-agent research and embedding. Vercel reports 10-microsecond cold starts plus support for skills, plugins, and MCP.

WORKFLOW3mo ago
Study finds context compactors retain only 17% of standing agent rules

A University of Pennsylvania study found common compactors retained only 17% of standing session rules while preserving task information. Practitioners use verified Markdown handoffs and background compaction to manage long coding-agent runs.

NEWS3mo ago
Composio and Ante benchmark coding agent harnesses with 47%–67% success range

Composio and Ante tests reported that the same models behaved very differently by harness. DeepSeek V4 Flash ranged from 47% to 67% task success and $0.019 to $0.104 per task across harnesses.

NEWS3mo ago
DeepSeek V4 Flash benchmarks claim lower DeepSWE cost than GPT-5.6 Luna

Together says two V4 Flash attempts solved more DeepSWE tasks than one GPT-5.6 Luna attempt for roughly one-third the cost. Practitioners report Flash-0731 results vary sharply by harness and pass count.

WORKFLOW3mo ago
Agent studies trace reliability failures to harness design and verifier quality

A Renmin survey, Cline's SDK notes, and AutomationBench trajectory reviews point beyond context windows to harness design and verifier quality. A failure-mode paper maps issues across models, memory, tools, users, and environment.

WORKFLOW3mo ago
Agent builders compare thin harnesses with large skill files for coding agents

Practitioner threads argued coding agents work better with task-specific harnesses, compact runbooks, evals, and deterministic checks. Examples included Sentry MCP traces and state-machine guardrails.

AI PrimerAI Primer

Your daily guide to AI tools, workflows, and creative inspiration.

© 2026 AI Primer. All rights reserved.