Ramp's 137-task Accounting Bench finds the best model fully solves 21% of tasks

• The best model fully solved only 21.2% of Ramp's accounting tasks across up to three attempts, according to RampLabs' result breakdown.
• Claude Fable 5.1 led average rubric score at 49.7%, while GPT-6 Astra reached 48.7% at roughly one-third the cost, per RampLabs' comparison.
• The suite uses 137 expert-grounded tasks spanning 22 simulated companies, RampLabs' announcement says, rather than isolated finance questions.
• Model strengths split by workflow: RampLabs' task results put Fable ahead on reconciliation and Gemini 3.8 Flash ahead on variance analysis.

One sample task requires reconciling contradictory ledger and workbook data before proposing a journal entry, in the official benchmark walkthrough. Ramp's engineering writeup describes the benchmark as a static environment for iterating on its Stack accounting agent before customer use.

Ramp froze the benchmark at 137 tasks across 22 synthetic company "Worlds," each stocked with company-specific QuickBooks ledgers, bank statements, invoices, contracts, internal memos, and reconciliation workbooks. Agents must find evidence in those systems, apply accounting treatment, then return exact answers or create and edit files.

The harness runs a basic-prompt ReAct agent in an isolated sandbox behind an MCP gateway. Progressive disclosure starts it with only meta-tools, requiring it to discover and add from a 92-tool catalog:

• 41 QuickBooks tools
• 32 spreadsheet tools
• 7 email tools
• 6 filesystem tools
• 5 PDF tools
• 1 code-execution tool

The benchmark page says that design is meant to represent broad production tool catalogs rather than a hand-picked toolset for each task.

Ramp grades each task against a human-created rubric using a frozen GPT-5.6 Luna judge. Mean reward retains criterion-level partial credit; solve rate counts a task when any of three independent attempts is perfect; pass³ requires a perfect result on all three attempts.

That metric split changes the leaderboard. Fable's 49.7% mean score edged Astra's 48.7%, but GLM-5.3 produced the most fully solved tasks at 21.2%. No evaluated model fully completed more than roughly one in five tasks even after retries.

The aggregate ranking does not carry cleanly across accounting workflows. Ramp found Fable 5.1 ahead on reconciliation among the four models in its comparison, while Gemini 3.8 Flash led variance analysis.

The result makes the benchmark useful as a workload map, not only as a single model score.

Ramp classified failures into its five most common error categories. Data interpretation and accounting judgment were the two most frequent, according to RampLabs' error analysis.

The sample walkthrough shows the kind of ambiguity behind those labels: an empty live ledger cannot automatically be treated as a zero balance when other exports show a recorded allowance, and a reserve workpaper can conflict with the governing aging-bucket policy.