AI21 reports 80.8% on SWE-Bench Pro with model-team agent

• AI21 turned SWE-Bench Pro into a model-team routing problem: AI21Labs' launch post reports 80.8% resolve on the full 731-task set at $5.99 per task.
• The pipeline splits work by cost and failure mode: AI21Labs' pipeline note puts open-model juniors on parallel exploration, a cheaper senior model on dependency extraction, and one frontier principal model on the final patch.
• The same team shape is showing up outside benchmarks: omarsar0's Raft setup uses scout, critic, and synthesizer agents, while ClaudeDevs' code-review launch turns review depth into selectable effort levels.
• Orchestration is becoming the product layer: SakanaAILabs' Fugu post adds NVIDIA Nemotron to a multi-model Fugu agent pool, and OpenRouter's MCP thread lets agents discover, rank, test, and report on models from inside the editor.
• Loop guardrails are getting explicit: _philschmid's Gemini update adds free-tier Managed Agents, token budgets, and cron triggers, while ClaudeCodeLog's 2.1.212 changelog adds default caps for web searches and subagent spawns.

AI21's full writeup has the useful bits: MiniMax-M3 starts at 57% as a ReAct baseline, the final pipeline reaches 80.8%, and the assembled context covers about 90% of removed gold-patch lines before the frontier model writes once. Sakana's Fugu post says Fugu is itself a model trained to call LLMs, including recursive calls to itself. Google's Managed Agents docs describe the API as a configurable harness that provisions a Linux sandbox in one call, and the Antigravity Agent docs now describe cron triggers for recurring runs.

AI21's result is useful because it names the pipeline, not just the score. The official writeup says the agent receives a natural-language issue, works inside a Docker container, acts through terminal commands, and outputs a git patch scored against hidden tests.

The team split is concrete:

1. Junior: MiniMax-M3 fans out across the repo in parallel, searching, trying fixes, and running tests.
2. Senior: GPT-5.2 reads those attempts and extracts the functions, callers, surrounding code, and tests the fix depends on.
3. Principal: Opus 4.8 or Fable 5 writes the patch once from the prepared brief.

In AI21Labs' pipeline note, the company framed most agent work as exploration that does not need frontier quality. In AI21Labs' cost note, AI21 said frontier compute is only 25% of the budget, putting the full pipeline at about one third the cost of a frontier agent run end to end.

The technical meat is the context measurement. AI21 compared the assembled context against gold patches for all 731 public SWE-Bench Pro tasks, then separated lines the gold patch removed or modified from lines it added.

The reported coverage:

• Removed or modified gold lines: about 90% coverage after parallel rollouts plus GPT-5.2 extraction.
• Added gold lines: about 71% coverage.
• Fable 5 pipeline cost: $5.99 per task.
• Solo Opus 4.8 reference cost: $18.28 per task, as AI21 attributes to Fireworks AI.

That is the cleanest claim in the release: cheap rollouts are not just votes, they become a repo map for the model that writes the final patch.

Omar Sar, who publishes agent research and tooling notes as omarsar0, used Raft as an LLM Council with three named roles: a scout agent for new papers, a critic agent that challenges them, and a synthesizer agent that writes the brief. The key detail in omarsar0's critic-agent note is that the critic runs on a different model than the scout.

His thread turns the pattern into a review loop:

• omarsar0's workflow note says the second agent disagrees before the result reaches him.
• omarsar0's demo result says the critic cut the two biggest overreaching results in the demo.
• omarsar0's report-loop note says an agent can write a Markdown report, receive inline comments in Raft, and revise.
• omarsar0's scheduled-run note says the same team can run daily and post updates back into the thread.

Claude Code made the same shape a product control. ClaudeDevs' launch post says /code-review now has effort levels, and ClaudeDevs' ultra note says /code-review ultra spawns a fleet of reviewer agents that independently reproduce every finding.

The Harness Handbook paper states the premise directly: agent capability depends on the harness that constructs prompts, manages state, invokes tools, and coordinates execution. Its problem statement is less flashy and more important: production harness behavior is spread across files, functions, execution stages, and state transitions, so changing a behavior requires finding all the code locations that implement it.

Open harnesses are becoming audit objects. rohanpaul_ai's Grok Build notes says xAI published Grok Build under Apache 2.0, with Rust code covering the agent loop, file tools, shell execution, web search, and terminal interface. Simon Willison, a developer and technical writer known for close reads of AI tooling releases, found the repo was 844,000 lines of Rust and even included a self-contained terminal renderer for Mermaid diagrams in his Grok Build dive.

That is Christmas come early for coding-agent nerds: the harness is now something people can diff, fork, inspect, and benchmark against their own workflows.

Sakana's Fugu announcement reads like the production version of AI21's benchmark pattern. The Sakana blog says Fugu sits behind a single API, dynamically selects and coordinates multiple models, and combines their strengths into one response.

Nemotron's role is specialist capacity. Sakana says NVIDIA's open model family will enter Fugu as specialized agents with strengths in coding, tool calling, and instruction following, alongside frontier and other open models already in the pool.

OpenRouter is exposing a different slice of the same layer. OpenRouter's task-insights note says its MCP now shows 30 task categories from real traffic, and OpenRouter's model-search note adds filters for output price, model age, intelligence and coding scores, tool-calling success rate, use case, modality, and context length.

Provider pinning is the eval gotcha. OpenRouter's provider-pinning note says the same model can now be pinned to the same provider on every call, which keeps model experiments from silently turning into provider experiments.

Google's Managed Agents update adds three operational controls to the Gemini API surface: free-tier usage for keys from free-tier projects, max_total_tokens to pause and resume tasks, and native cron triggers such as schedule: "0 9 * * *". The Managed Agents overview describes the underlying primitive as a single API call that provisions a Linux sandbox where an agent can reason, run code, manage files, and browse.

Claude Code's 2.1.212 changelog adds hard loop budgets. ClaudeCodeLog's full changelog lists a default 200-call session limit for WebSearch, a default 200-spawn per-session cap for subagents, and /clear as the reset point for the subagent budget.

The same changelog says /fork now copies a conversation into a new background session row in claude agents, MCP tool calls running longer than two minutes move to the background automatically, and inter-agent messaging no longer duplicates SendMessage bodies into replayed history and tool results.