AutomationBench-AA launches SaaS-agent leaderboard with 657 tasks

• AutomationBench-AA evaluates 657 Zapier workflow tasks across 40 simulated SaaS app environments, and Artificial Analysis says scoring is based on final system state rather than conversational judging.
• Claude Fable 5 led at 48.6%, barely ahead of Claude Opus 4.8 at 48.5%, with Artificial Analysis noting Fable fell back to Opus on about 18% of tasks.
• Business-rule safety is still the blocker: every evaluated model triggered guardrail violations, while Gemini 3.5 Flash had the best objectives-per-violation ratio in the guardrail results.
• Price and score split sharply: Gemini 3.5 Flash reached 42.6% at $0.49 per task, while the cost breakdown says task cost ranged from under five cents to nearly $1.50.
• Finance workflows were the hardest domain, with agents completing about one third of Finance objectives versus roughly 60% in Support and Operations, according to Artificial Analysis.

AutomationBench-AA comes with full Artificial Analysis results, Zapier's hosted leaderboard, an arXiv paper, and a GitHub repo. The benchmark is unusually practical for agent eval nerds: models discover REST APIs, work through irrelevant SaaS records, and get graded on whether the right data landed in the right simulated systems. The funniest scoreline is brutal for open weights: teortaxesTex called out GLM-5.2 falling into the mid-tier on a bench where SaaS workflow discipline matters more than code-bench swagger.

AutomationBench-AA is an independent Artificial Analysis leaderboard for Zapier's AutomationBench, run on a private benchmark subset developed from real Zapier workflow patterns.

The task shape is closer to office ops than chat QA:

• 657 workflow automation tasks.
• 40 simulated SaaS app environments.
• Domains: Finance, HR, Marketing, Operations, Sales, Support.
• Apps include Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira, HubSpot, QuickBooks, Xero, Asana, and Calendly.
• Models interact through REST APIs and discover endpoints through structured tool calls.
• Each task runs once with a 50-turn cap.
• Grading uses deterministic checks against final environment state.

Zapier built nearly 12,000 assertions for the benchmark, split into objectives the agent must satisfy and guardrails that should remain unbroken, according to Artificial Analysis.

The headline score is the share of task objectives completed without any guardrail violations. That makes the leaderboard harsher than pure objective completion, because a model can do useful work and still lose credit by breaking a business rule.

Top launch scores:

1. Claude Fable 5 with fallback: 48.6%.
2. Claude Opus 4.8 (max): 48.5%.
3. Gemini 3.5 Flash: 42.6%.
4. GPT-5.5 (xhigh): 42.1%.
5. GPT-5.5 (high): 40.2%.
6. Claude Sonnet 5 (max): 39.2%.
7. GPT-5.5 (medium): 37.8%.
8. Gemini 3.1 Pro Preview: 37.5%.

Fable's win is basically a photo finish: Artificial Analysis says the model completed 73% of task objectives, but fell back to Opus on about 18% of tasks.

Every launch model violated guardrails. The useful split is how much completed work each model got per business-rule break.

The best objectives-per-violation ratios were:

• Gemini 3.5 Flash: 15.0.
• Claude Opus 4.8 (max): 13.5.
• Claude Fable 5 with fallback: 13.0.
• GPT-5.5 (xhigh): 10.5.
• Claude Sonnet 5 (max): 9.5.
• Gemini 3.1 Pro Preview: 9.3.
• GPT-5.5 (high): 9.2.

Guardrail violations ranged from 0.46 per task for Gemini 3.5 Flash to 1.26 for Qwen3.7 Plus, according to Artificial Analysis. That ratio is the benchmark's sharp edge: most agent demos hide the cost of one bad write to the wrong system.

Cost per task spans more than an order of magnitude. Artificial Analysis put the low end below five cents for DeepSeek V4, Gemini 3.1 Flash-Lite, and Qwen3.7 Plus, with high-end runs near $1.50 for models such as Claude Opus 4.8 (max).

Gemini 3.5 Flash is the cost-performance outlier in the launch set: 42.6% at $0.49 per task. GPT-5.5 (xhigh) scored 42.1% at $1.32 per task, so Gemini effectively matched it at about 37% of the per-task cost, per Artificial Analysis.

Finance was the hardest business domain. Across evaluated models, agents completed around one third of Finance objectives, roughly half the Support and Operations rate of about 60%, according to Artificial Analysis.

That gap fits the benchmark design. Finance tasks tend to combine stricter invariants, multi-system state, and higher penalty for a plausible but wrong update.

The run traces show different agent styles behind similar headline scores.

• GPT-5.5 (xhigh): 49 tool calls across 25 turns per task.
• Claude Opus 4.8 (max): 35 tool calls across 14 turns, with 0.55 guardrail violations per task.
• GPT-5.5 (xhigh): 0.66 guardrail violations per task.
• Grok 4.3 (high): 13 turns, the fewest in the set, but only 8.1% on the headline score.

Artificial Analysis interpreted Grok's short runs as consistent with premature task completion rather than efficient finishing. Claude Opus 4.8 also produced the most output tokens per task in the chart at 32k, while GPT-5.5 (medium) used 4k.

GLM-5.2 (max) was the leading open-weights model at 27.8%, according to Artificial Analysis. That put it about 9.7 points behind Gemini 3.1 Pro Preview at 37.5%, with substantially higher guardrail violations per task.

The community read was less polite. teortaxesTex called GLM-5.2's result a crash to mid-tier models, which is a useful corrective to treating open-weight frontier strength as portable across every agent workload.

Artificial Analysis pointed readers to four primary artifacts:

• Full AutomationBench-AA results.
• Zapier leaderboard.
• arXiv paper.
• AutomationBench on GitHub.

Those links matter because the leaderboard is not just a model ranking. It is a benchmark recipe: simulated SaaS state, REST API discovery, objective assertions, guardrail assertions, one-run grading, and a 50-turn cap.