Artificial Analysis launches AA-3D, a benchmark for interactive scene-building agents

• AA-3D tests complete scene-building agents on 55 briefs, using sandboxed three.js builds and blind human comparisons, according to the announcement.
• Four models are statistically tied at the top: Sonnet 5.5 leads at 1194 Elo, with confidence intervals overlapping Opus 5.5, GPT-6 Astra and Fable 5.1 in the launch results.
• Scene costs vary sharply: the cost comparison puts GPT-6 Luna at $0.20 per scene and top-ranked Sonnet 5.5 at $15.18.
• Domain winners mostly follow the overall rankings, but gaps below about 12 percentage points are noise, according to the domain breakdown.

Browser testing consumed 82% of tool time, compared with 6% for writing and editing code. MiniMax-M3 inspected its output 54 times per scene, more than any rival, and still ranked 19th.

Every model receives the same brief, delivery contract, tools and limits in Artificial Analysis’s open-source Stirrup harness, according to the generation specification.

• Budget: one attempt per brief, stopping after three hours or 250 turns.
• Compute: four CPU cores, 8 GB RAM and no GPU; WebGL renders in software.
• Tools: three.js 0.186.0, optional Rapier 0.20.0 and Blender 4.5, Python 3, Node.js 22 and headless Chromium for self-testing.
• Assets: optional CC0 lighting, materials, models, characters and animation clips.
• Delivery: an index.html page with assets, up to 200 MiB excluding the bundled three.js and Rapier libraries.
• Network: none. Each task runs in an isolated offline sandbox.

Artificial Analysis ranks 21 models using more than 10,000 counted votes from over 1,000 panelists, according to its announcement. The evaluation methodology separates scene eligibility from preference judging:

1. Browser check: a scene has 60 seconds to draw its first frame, then must keep rendering a non-blank image offline without errors.
2. Human review: scenes passing the automated check are reviewed before reaching panelists. Failures at either stage are excluded from voting.
3. Blind comparison: panelists interact with two scenes from the same brief for at least 30 seconds each. Model identities remain hidden even after voting.
4. Session validation: roughly one comparison in four has a strong consensus answer. Failed sessions lose all their votes; validation comparisons never enter the ratings.

Retained votes receive equal weight in a Bradley-Terry maximum-likelihood fit, rescaled to an Elo-like range. The reported 95% confidence intervals use a sandwich estimator.

The launch leaderboard reports these leading scores and 95% intervals:

| Model | Elo | 95% interval |
| --- | ---: | --- |
| Claude Sonnet 5.5 | 1194 | 1160 to 1228 |
| Claude Opus 5.5 | 1190 | 1155 to 1225 |
| GPT-6 Astra | 1165 | 1131 to 1199 |
| Claude Fable 5.1 | 1131 | 1098 to 1165 |

Artificial Analysis describes the four as statistically tied because each other model’s interval overlaps Sonnet’s.

Two other findings accompany the rankings:

• Open weights: Kimi K3 (982) and MiMo-V2.6-Pro (977) place 11th and 12th, more than 200 Elo behind first place.
• General intelligence: AA-3D Elo correlates with the Artificial Analysis Intelligence Index at Pearson 0.92, with about 18 Elo per index point.

The task catalogue covers seven domains:

• Landmarks and real places
• Machines
• Natural phenomena
• Living bodies
• Games, sports and piloting
• Space
• Stories and fantasy

A top-five model has the highest win rate in six domains. Living bodies is the exception: sixth-ranked GPT-6.1 Sol reaches 78.6%, alongside Sonnet 5.5 at 78.5%, in the domain results.

Each model-domain cell contains only two to ten scenes, with a median of seven, according to the launch analysis. Cells with fewer than five scenes are grayed out; differences below roughly 12 percentage points fall within noise.

Four models earn higher Elo than every cheaper model in the cost results:

| Model | Mean token cost per scene | Elo |
| --- | ---: | ---: |
| GPT-6 Luna | $0.20 | 1000 |
| Muse Spark 1.3 | $3.21 | 1035 |
| GPT-6 Sol | $3.46 | 1123 |
| Claude Sonnet 5.5 | $15.18 | 1194 |

GPT-6 Sol is the frontier’s interesting middle: Sonnet costs about 4.4 times as much for a 71-point Elo gain.

Input accounts for 69% of spend because agents resend their full context each turn, the launch article reports. Input represents 97% to 99% of tokens, and most models read nearly all of it from cache.

Inkling is omitted from cost plots because Artificial Analysis lacks a representative cache-hit rate for it.

Among the 19 image-capable models, inspecting their own output early correlates with Elo at Spearman 0.82, according to the agent-run analysis.

• First look: the top five inspect an image of their scene after a median of 10 to 13 turns, and by turn 20 in at least 89% of scenes.
• Later inspection: models ranked 14th to 20th first inspect their scenes after a median of 15 to 40 turns.
• Browser testing: the top five start at median turn 16, compared with 25 for the others.
• Longer runs: neither output-token volume nor turn count has a statistically significant relationship with Elo.
• Text-only models: GLM-5.3 and Nemotron 3 Ultra build without seeing their output.

The public AA-3D Voting interface reveals the two model names after visitors explore both scenes for 30 seconds and choose a favorite. Those votes never affect the leaderboard, which uses only the recruited panel, according to the methodology.