Databricks benchmarks coding agents on internal codebase tasks

• Databricks built a coding-agent benchmark from real internal engineering tasks because public benchmarks like SWE-Bench are often over-tuned, according to the benchmark setup.
• OpenAI, Anthropic, and GLM-5.2 all landed on Databricks' cost-quality frontier, with Yuchenj_UW's summary framing the routing problem as picking the right model for each task.
• The simple Pi harness matched vendor harness success rates with Opus and GPT 5.5 at half the cost, according to the harness result.
• Per-token price broke as a proxy: Sonnet 5 cost less per token than Opus 4.8 but used more tokens and ended up worse on cost and quality in Databricks' example.
• Databricks tied the benchmark to OmniGent as a meta-harness and Unity AI Gateway as a central LLM usage gate in its follow-up.

The Databricks post is the useful artifact: it plots pass rate against mean dollars per task, not just leaderboard score. A small OpenBench follow-up in mattlam_'s screenshot put GLM-5.2 plus Pi at 11/11 solves on one completed segment, while ollama's rollout says GLM-5.2 now runs through Claude Code, VS Code, Codex, and Hermes on its cloud.

Databricks built the benchmark from tasks its engineers had already completed, then curated test suites to check whether coding agents could solve them end-to-end.

The codebase mix is the part public evals usually miss. Databricks' GLM-5.2 note says the internal tasks covered Scala, Go, Rust, Java, TypeScript, Protobuf, and Jsonnet, and were “VERY different” from SWE-Bench and TerminalBench.

The benchmark is also a product loop. Databricks said academic benchmarks remain useful, but companies care about their own task distributions, so “each company needs its own loop” in a later reply.

Databricks' chart turns the usual coding-model race into a cost frontier. The efficient line in the screenshot contains:

• GPT 5.4-mini with Codex, about $0.50 per task at 58% pass rate.
• GPT 5.4 with Codex at medium reasoning, about $0.60 at 74%.
• GPT 5.5 with Pi at medium reasoning, about $0.70 at 83%.
• Opus 4.8 with Pi at high reasoning, about $0.75 at 85%.
• GLM-5.2 with Pi, about $1.20 at 87%.
• Opus 4.8 with Claude Code at high reasoning, about $2.20 at 87%.
• Opus 4.8 with Pi at extra-high reasoning, about $2.30 at 90%.

The spicy dot is GLM-5.2: Yuchenj_UW's summary put OpenAI, Anthropic, and GLM-5.2 on the Pareto frontier and argued that the “real unlock” is an intelligent router.

Christmas came early for coding-agent harness nerds: the wrapper moved the curve.

Databricks said Pi reached the same success rate as vendor harnesses with Opus and GPT 5.5 at 2x lower cost, mainly because it sent smaller inputs to the model. In a reply to Thom_Wolf, matei_zaharia said Databricks looked closely at the result and that a simple hand-written agent loop also did well.

The community diagnosis was harness bloat. sh_reya pointed at system prompt text, skills, and tools accreting over time, while andykonwinski compared agent architectures to monolithic operating systems versus microkernels.

Databricks is still shrinking the kernel. In response to the microkernel analogy, matei_zaharia said Databricks would like to try even simpler harnesses.

The cleanest result was also the most expensive lesson: cheaper tokens did not guarantee cheaper tasks. Databricks said Sonnet 5 cost less per token than Opus 4.8, then used more tokens and landed at higher cost with lower quality in its example.

That matched a broader benchmark argument from fchollet, who wrote that scalar scores like “75% on XYZ” are meaningless without efficiency. His follow-up made the same point with latency and marginal cost: “how much does it cost for each one, and how long to get there?” fchollet's marginal-cost reply

The cost conversation got less abstract in zeeg's reaction: a “simple Fable coordinator session” costing hundreds of dollars was framed as burn, not a flex. zeeg's back-of-envelope reply translated $1,000 per week per person into roughly $50,000 per engineer per year.

Databricks called GLM-5.2 a major open-source coding-agent jump on its own codebase, not just on public coding evals.

The result landed in a live argument about what “open” buys. mervenoyann argued GLM-5.2 should be called an open model rather than a local model, and said the enterprise case gets stronger when a team can deploy open models on-prem so IP does not leave the organization.

The ceiling claim stayed contested in the community. theo called GLM-5.2 incredible but pushed back on claims that it is self-hostable in the same practical sense or comparable to Fable, while mbusigin said GLM-5.2 is the best open-weight model but that frontier commercial models remain better.

The benchmark's product read is routing: multiple models near the frontier, different costs, different task fit.

Databricks said the findings partly motivated OmniGent as a “meta-harness” for switching and composing agents, plus Unity AI Gateway for centralized LLM usage analysis and gating. Yuchenj_UW's token-efficiency reply reduced the operating principle to token efficiency.

The orchestration idea showed up outside Databricks too. Warp's interview clip summarized the pattern as SOTA models breaking down workflows and smaller models executing, while DynamicWebPaige phrased the rule as choosing the smallest model that can do the task reliably well.

Databricks did not present the chart as a universal leaderboard.

The company said the results came from its sample of its own codebase and were not meant to be comprehensive. matei_zaharia's rerun note added that the benchmark ran over a few days with multiple runs of each setup, and that Databricks plans to rerun it regularly.

The internal eval is paired with human feedback. Databricks has people testing the models day-to-day, and matei_zaharia described the benchmark as an attempt at a more objective comparison.

Small deltas also got a warning label. matei_zaharia's noise caveat said not to read much into close pass-rate differences, and floated context growth as one possible reason a model could get confused.

Enterprise pricing shaped the measurement. matei_zaharia's pricing reply said Databricks used pay-as-you-go pricing because that is now the enterprise norm, while individual developers still have monthly plans.

GLM-5.2's practical story now includes hosted tool integrations, not only benchmark dots.

ollama said GLM-5.2 on its cloud had more US and Europe capacity, delivered 80 to 120 output tokens per second during peak hours, and compared that with 30 to 40 tokens per second on other providers. The same post listed launch paths for Claude Code, VS Code, Codex, and Hermes, with a model page attached.

A separate OpenBench run was still in progress. mattlam_'s request asked Databricks to benchmark GLM-5.2 across reasoning levels and harnesses, matei_zaharia said Databricks would try other harnesses, and Yuchenj_UW said they might test that split too.