Accio open-sources 107-task CommerceAgentBench

• Accio has open-sourced CommerceAgentBench, a 107-task suite that grades the state an agent leaves in a mock commerce stack, as testingcatalog's launch post describes through saved drafts, labels, listings, and shipment IDs.
• The best observed run completed 66 of 107 workflows, a 61.7% pass rate, according to kimmonismus's leaderboard post.
• Harness configuration created an 11.3-point swing for GPT-5.6 Luna on the same task set, from 33.6% to 44.9%, in testingcatalog's comparison.
• The initial release includes 14 mock services, and testingcatalog's follow-up says Accio is accepting new replicas through pull requests.

The project README splits the suite into 53 CLI, 28 browser, 16 file, and 10 API/MCP tasks. One Gmail scenario hides a purchasing decision inside roughly 300 noisy emails, while another task uses a listing form that rejects bad input, as testingcatalog's follow-up noted.

CommerceAgentBench tests the resulting state in a mock stack, including a mailing label, saved draft, published listing, or shipment ID. The project README says each task starts in a fresh container and is checked by a deterministic or LLM-assisted verifier.

The 107 tasks also divide into 65 text-only, 20 browser-text-capable, and 22 vision-required cases.

A Gmail case in kimmonismus's benchmark post asks an agent to turn a chaotic supplier inbox into a recorded buying decision:

• Identify real suppliers and reconstruct the latest quotes.
• Compare landed costs across six Incoterms and four currencies.
• Detect a buried payment-redirection fraud attempt.
• Apply labels, save a reply draft, and create a kickoff event.

Accio evaluated 13 model families across three harnesses, according to testingcatalog's launch post, while its public leaderboard lists 39 model-harness pairs on the 107 workflows.

GPT-5.6 Luna scored 33.6% in one harness and 44.9% in another. That 11.3-point difference exceeded the gap between the second- and eighth-ranked entries on the board, testingcatalog's comparison said. Harness configuration is therefore an observed source of score variation in the benchmark.

The first release ships 14 mock services and accepts pull requests for more, as testingcatalog's follow-up reported. Each new replica creates a family of tasks outside the models' existing training exposure, according to that post.

The harness is Apache 2.0 licensed and the task suite is CC BY 4.0. Accio's benchmark case map catalogs the browser, API, CLI, and file surfaces.