Composio and Ante benchmark coding agent harnesses with 47%–67% success range

• The same DeepSeek V4 Flash model landed between 47% and 67% task success depending on harness, with cost per task spanning $0.019 to $0.104 in Composio's eight-harness conclusion.
• Pi Agent led Composio's new batch at 20/30 tasks passed, while Composio's cost table put it at $0.028 per successful task.
• Claude Code's Kimi K3 run showed the ugly side of rich context replay: pauliusztin_'s breakdown reported 340K median tokens per task, versus 61K for Kimi Code and 67K for Hermes.
• Ante pushed the same harness conversation in the opposite direction, with ai_for_success's Ante note describing a 15 MB self-contained binary with embedded grep, git, PDF, and OCR.
• The benchmark still has reproducibility gaps: Composio's prompt disclosure reply said exact prompts and grading criteria are not public, while bcherny said the results did not match internal evals.

The DeepSeek model card frames V4-Flash-0731 as the official Flash release with stronger agentic behavior and the same model structure as the DSpark variant. Composio's Pi vs Claude Code post says its hard 30-task setup used DeepSeek V4 Flash, hosted Composio MCP tools, and a 900-second ceiling per task. Ante's README publishes a checkable Terminal-Bench 2.1 run at 82.7%, and its methodology page says each run pins a public binary and links raw Harbor trials.

DeepSeek-V4-Flash-0731 is a useful testbed because the model stayed fixed while wrappers changed. The Hugging Face model card says the release supersedes the preview, keeps the same structure as DeepSeek-V4-Flash-DSpark, and posts an 82.7 on Terminal Bench 2.1.

Composio used the same model across harnesses, which turns the harness into the measured variable. In Composio's Pi vs Claude Code post, the company says the setup used real app tasks, the same hosted MCP router, and the same DeepSeek V4 Flash model.

That makes this a clean benchmark shape for agent builders: model, tools, and task set fixed, loop behavior not fixed.

In Composio's new-batch scorecard, its cost table, and its timing table, the four added harnesses sorted like this:

• Pi Agent: 20/30 passed, $0.028 per success, 132.2s median time.
• Deep Agents: 16/30 passed, $0.045 per success, 187.1s median time.
• Hermes Agent: 15/30 passed, $0.056 per success, 175.5s median time.
• Prime Agent: 15/24 valid runs passed, $0.131 per success, 242.1s median time.

The earlier Composio batch had a different winner per metric: the first batch chart put Oh My Pi first on tasks passed at 17/30, Claude Code first on speed at 123s, and OpenCode lowest on cost per successful task at $0.073.

Across all eight, Composio's conclusion summarized the same model at 47% to 67% task success, $0.019 to $0.104 per task, and 122.7s to 272.4s median time.

The Kimi K3 comparison is the cleanest cost spread. pauliusztin_'s post reported similar success rates across Kimi Code, Hermes, and Claude Code, but median token use split hard:

• Kimi Code: 22/28 passed, 61K median tokens, $0.22 average cost.
• Hermes: 21/28 passed, 67K median tokens, $0.28 average cost.
• Claude Code: 20/28 passed, 340K median tokens, $2.00 average cost.

One proposed mechanism was context replay. pauliusztin_'s follow-up listed the material Claude Code may keep feeding back across turns:

• previous messages
• tool outputs
• file contents
• command history

Composio's longer Kimi K3 harness writeup later clarifies a 25-valid-task setup across eight harnesses, with pass rates from 68% to 88% and cost estimates based on OpenRouter list prices.

The harness gap is mostly software architecture, not benchmark trivia. LangChain's Deep Agents post defines Deep Agents as an off-the-shelf agent harness whose job is getting the right context to the model at the right time.

The stack diagrams in the harness layer chart and hwchase17's LangChain diagram converge on the same moving parts:

• serving surfaces
• orchestration
• sandboxing
• context engineering
• memory
• tools
• agent loop
• evals
• MCP connectors

Composio's Pi vs Claude Code post makes the contrast concrete: Pi exposes read, write, edit, and bash, while Claude Code ships more built-ins, including sub-agents, plan mode, MCP, skills, checkpoints, and permissions.

Deep Agents landed in the middle of the cost table. Harrison Chase's Deep Agents note said LangChain had slimmed down the harness, and Composio's numbers put it second cheapest in the new batch.

Ante's pitch is harness minimalism at the binary level. Ante's README describes a roughly 15 MB Rust binary with zero runtime dependencies, embedded grep and git, and local inference through a pinned llama.cpp engine.

The same README says Ante's latest full Terminal-Bench 2.1 run scored 82.7% with DeepSeek V4 Flash 0731, covering 368 of 445 trials and about $68 of inference. Ante's methodology page says those runs use 89 tasks, five trials per task, official leaderboard constraints, pinned public binaries, and raw Harbor run links.

Ante also claims lower machine footprint than Claude Code on 20 parallel Docker tasks:

• about 7x less peak memory
• about 9x less average CPU
• about 5x less disk I/O

The interesting overlap is the 82.7 number. teortaxesTex noted that Ante's DeepSeek V4 Flash result matched the score DeepSeek cited for its own harness.

Together tested the cost side from a different angle: multiple cheap attempts against one stronger model attempt.

Together's post said two DeepSeek V4 Flash attempts solved more DeepSWE tasks than one GPT-5.6 Luna attempt at roughly one-third the cost. Together's linked blog put the single-shot numbers at 53.3% pass@1 for DeepSeek V4 Flash 0731 and 67.2% pass@1 for Luna, then reported DeepSeek pass@2 at 70.1%.

teortaxesTex's pass@ follow-up added the missing shape: V4 Flash pass@2 beat Luna pass@1, but V4 Flash pass@4 narrowly trailed Luna pass@2.

The hardest part of reading these harness numbers is knowing which parts reproduce outside the original setup.

Teknium's benchmark-detail complaint said there was no way to know how the result happened without an open benchmark and details. Composio later said in its prompt disclosure reply that tasks were described publicly, but exact prompts and grading criteria were withheld.

Prime Agent also complicated the clean table. Composio's Prime note said six Prime runs were excluded: two verifier timeouts against sessions up to 3.5M tokens and 33 tool calls, plus four runs that never recorded.

Composio's replies narrowed the scope in a few more places:

• The tests had not covered SOTA models yet, according to Composio's SOTA-model reply.
• The public runs used default configurations, with packaged configs like Oh My Pi, according to Composio's default-config reply.
• More content on fair eval methodology was still pending, according to Composio's eval-methodology reply.