Artificial Analysis replaces DeepSWE with FrontierCode in Coding Agent Index v2.0

• Coding Agent Index v2.0 replaces DeepSWE with FrontierCode and revises repository Q&A grading, as detailed in the launch announcement.
• Claude Code with Claude Opus 5.5 (max) leads at 61.7 in the published rankings, followed by Sonnet 5.5 at 59.3.
• Scores across versions measure different setups: the mean for 21 shared configurations fell from 54.8 to 47.4 in the version comparison.
• Repository Q&A keeps the same 124 tasks but changes the judge, voting and rubric wording, according to the grading update.

FrontierCode’s 100-task main split comes from a 150-task suite. The Q&A release also fixes two negative criteria whose wording inverted their intended meaning.

Artificial Analysis evaluates models together with their agent harnesses and execution settings. The v2.0 leaderboard covers:

• FrontierCode v1.1: 100 software engineering tasks from Cognition.
• Terminal-Bench 4.0: 66 terminal-use tasks, unchanged in this update.
• SWE-Atlas-QnA-AA: 124 repository Q&A tasks from Scale AI, with Artificial Analysis’s grading adjustments.

Each benchmark score averages three attempts per task. The index methodology gives each benchmark equal weight, regardless of its task count.

Claude Code occupies three of the top four positions in the announced rankings:

| Harness | Model and setting | Index score |
|---|---|---:|
| Claude Code | Claude Opus 5.5, max | 61.7 |
| Claude Code | Claude Sonnet 5.5, max | 59.3 |
| Muse Code | Muse Wonder Full, max | 58.9 |
| Claude Code | Claude Fable 5.1, max with fallback | 57.6 |

FrontierCode grades changes to real open-source repositories against whether a maintainer would merge them. Its task set is private to reduce contamination.

The verifier uses six check types:

1. Classical tests.
2. Command checks.
3. Reverse-classical tests.
4. Adaptive classical tests.
5. Scope checks.
6. LLM prompt review.

Failing any blocking check scores zero. Passing all blocking checks earns 70% to 100%, depending on the non-blocking checks satisfied.

Artificial Analysis also scans every trajectory with a modified version of Cognition’s rule-based scanner. Attempts flagged for accessing solution material score zero.

The questions, repositories, pinned commits, Docker images and reference answers remain unchanged. The published grading specification changes how answers are judged:

• Rubric wording: 127 of 1,306 criteria were adjusted, comprising 75 stability changes and 52 precision changes.
• Judge: GPT-6 Luna (xhigh) replaces Claude Opus 4.5.
• Voting: five votes per criterion, with majority vote determining the verdict and negative-criterion polarity applied afterward.
• Structured output: the judge returns reason, evidence and verdict fields.
• Execution evidence: criteria requiring observed or measured results accept reported execution results, rather than results inferred from reading code.

Task scoring remains all-or-nothing: every criterion must be met, with no partial credit.

Muse Code with Muse Spark Open lost 7.0 points under the revised grading, while Codex with GPT-6.1 Sol gained 2.1, according to the announcement. Muse Wonder Full leads this component, ahead of Fable and Opus.

For the 21 configurations evaluated on both versions, Artificial Analysis reports these means in its version comparison:

| Measure | v1.5 setup | v2.0 setup | Change |
|---|---:|---:|---:|
| Overall index | 54.8 | 47.4 | −7.4 points |
| Software engineering component | DeepSWE: 65.6% | FrontierCode: 45.1% | −20.5 points |
| Repository Q&A | 58.8% | 57.1% | −1.7 points |

Twenty of the 21 configurations received lower index scores. Terminal-Bench 4.0 is unchanged; the software engineering benchmark replacement accounts for most of the shift.

Four configurations sit on the reported cost frontier:

| Harness and model | Index score | API cost per task |
|---|---:|---:|
| Codex, DeepSeek V4.1 Flash (max) | 47.3 | $0.39 |
| Codex, GPT-6.1 Sol (max) | 54.1 | $1.50 |
| Codex, GPT-6 Astra (max) | 54.7 | $7.04 |
| Claude Code, Claude Opus 5.5 (max) | 61.7 | $11.42 |

These are pay-per-token API costs; subscription pricing is excluded. Moving from Sol to Opus adds 7.6 index points at approximately 7.6 times the API cost per task.

The Hugging Face release includes:

• data/test-00000-of-00001.parquet: the 124 tasks and adjusted rubrics.
• rubric_evaluation_config.yaml: the judge model, voting configuration, output schema and prompts.
• default_qa_config.yaml: Scale’s original SWE-Agent configuration, retained for reference. Artificial Analysis does not use it and runs each coding agent through its own harness.