Coding-agent harness benchmarks report about 2x completion-time spread on GPT-6 Astra

• Harness choice barely moved pass rates in Composio's GPT-6 Astra test: Codex, Hermes Agent, and Command Code each passed 21 of 29 tasks, while OpenCode and Pi Agent passed 19, according to Composio's task breakdown.
• Median task time ranged from 99 seconds in Pi Agent to 201 seconds in Claude Code, as Composio's timing report found.
• Failed runs consumed 3 to 5 times the cumulative input tokens of successful ones in Composio's failure analysis, while Composio's cost estimate put solved-task cost between $1.06 and $2.62.
• A separate Arena study tested 21 model-harness pairs and likewise reported modest correctness differences alongside much larger spending differences.

Arena's HarnessTax report sampled SWE-bench Lite and Terminal-Bench 2.0. Polylane's engineering post describes an Autofix issue that could involve as many as 18 agents before a pull request. Composio's separate Kimi K3 comparison fixes the model, provider, reasoning level, and tools across 25 tasks, a more fully specified experiment than the Astra thread.

Composio ran GPT-6 Astra through Codex, Claude Code, OpenCode, Hermes Agent, Pi Agent, and Command Code on 29 tasks it called challenging and agentic in its test announcement. The announcement names the model, harnesses, and task count, but does not specify the task corpus, repeated-run count, provider settings, or tool configuration.

Only three tasks produced different outcomes across harnesses. Eighteen passed everywhere and eight failed everywhere, Composio's result breakdown reported.

• Codex, Hermes Agent, Command Code: 21/29, 72%
• Claude Code: 20/29, 69%
• OpenCode, Pi Agent: 19/29, 66%

The result card measures median cumulative input tokens in failed runs relative to passed runs. Composio's failure chart reported the following ratios:

• Codex: 5.4x
• Pi Agent: 4.4x
• Hermes Agent: 4.3x
• Claude Code: 3.9x
• Command Code: 2.6x
• OpenCode: token use unavailable

Composio attributed the tail to more retries combined with larger contexts being resent on later calls in a follow-up. It said context per call grew by 1.4x to 2.1x on failed runs in another reply, and attributed leaner harnesses' lower spend to stopping earlier after a task went sideways in its explanation.

Pi Agent's estimated cost per successful task was $1.06, versus $2.62 for Claude Code, a 2.5x spread that Composio's estimate attributes to harness choice.

The Astra thread labels those figures estimates but does not publish a price card, cache assumptions, or per-harness token totals. Composio's Kimi K3 study uses eight harnesses rather than six and 25 tasks rather than 29, so its protocol and results are a separate comparison.

Pi Agent, Codex, and Hermes Agent recorded median task times of 99, 100, and 102 seconds. Claude Code took 201 seconds, Composio's timing figures said.

Claude Code also had the highest median tool-call count on both universally passed and failed tasks, while OpenCode had the fewest Composio's tool-call comparison. The thread does not establish whether those calls were productive work, retries, or different tool-selection behavior.

Arena's HarnessTax report tested seven models across Claude Code, Codex CLI, and Pi on 30 randomly sampled tasks from each of SWE-bench Lite and Terminal-Bench 2.0.

Its reported pattern was similar but broader:

• 21 model-harness pairs produced relatively close task-success results.
• Spending differed by as much as 5x for similar success rates.
• Pi's four built-in tools, read, write, edit, and bash, often sat on the cost-accuracy Pareto frontier.
• Some models performed better in a non-native harness than in their bundled one.

Polylane's Autofix case study says its earlier issue-fixing workflow could involve up to 15 subagents plus an orchestrator, with a confirmed issue reaching 18 agents. On September 3, it replaced triage, coordination, hypothesis generation, and coding handoffs with one agent that investigated and opened the pull request end to end.

The post attributes the change to one trace and the removal of cross-agent summaries. Separately, pvncher described wide parallelism as a shared-context tax because each subagent must relearn context. thdxr argued that Polylane's result also reflects the cost of an overly prescriptive workflow, and that background delegation remains useful in a multi-user Slack setting; boristane's terminology note distinguishes that pattern from parallel subagents that return summaries to a main agent.