Skip to content
AI Primer
breaking

Together reports GLM-5.3 solves 87.6% of DeepSWE work for about $16

Together reports GLM-5.3 solved 87.6% of DeepSWE after four attempts for about $16, versus Fable 5 at 69.7% for $21.63. It estimates equal $100 budgets yield roughly 17 solved tasks for GLM-5.3 and three for Fable 5.

3 min read
Together reports GLM-5.3 solves 87.6% of DeepSWE work for about $16
Together reports GLM-5.3 solves 87.6% of DeepSWE work for about $16

TL;DR

DeepSWE v1.1 grades committed code in clean, isolated environments after removing dependency drift and flaky tests. Its repository covers original long-horizon tasks across TypeScript, Go, Python, JavaScript, and Rust. Together's full report says Fable recorded 16 routing infrastructure errors to GLM's one, while its headline scoring excludes those errors.

Pass@4 versus pass@1

The comparison's most easily missed detail is that the two top-line percentages use different attempt counts. In Together's full result table, Fable leads the first try, 69.7% to 69.0%, within the reported error bands.

Retries reverse the ordering:

  • Pass@1: GLM-5.3 69.0%, Fable 5 69.7%
  • Pass@2: GLM-5.3 81.1%, Fable 5 77.1%
  • Pass@4: GLM-5.3 87.6%, Fable 5 84.1%

DeepSWE's v1.1 update describes the suite as 113 tasks. Together ran 452 rollouts per model, so pass@4 here means at least one of four independent trials cleared a task's verifier.

Cost per rollout

Together prices the comparison per rollout: $3.99 for GLM-5.3 and $21.63 for Fable. Its $100 projection combines that cost with observed solves, yielding 17 GLM tasks versus three Fable tasks in togethercompute's equal-budget calculation.

The price difference did not correspond to a major wall-clock gap in the vendor's data. GLM averaged 35 minutes, 124 agent steps, and 80,000 output tokens per rollout; Fable averaged 34 minutes, 85 steps, and 114,000 output tokens, per Together's breakdown.

Routing errors

Official scoring excluded infrastructure errors, leaving Fable at 69.7% pass@1. Together's strict calculation, which counts those failures, puts Fable at 67.3% and GLM at 68.8% because Fable had 16 routing errors and GLM had one.

Together also notes a data limitation: Fable's published cost matched trajectory totals, while GLM's cost analysis was index-level because its per-turn trajectory JSON was unavailable on the public CDN. The same report defines a near miss as at least 80% of new tests passing without breaking the baseline suite.

Rust and serialization

Fable retained material leads in Rust, 85% to GLM's 70%, and data modeling and serialization, 88% to 79%, in Together's task taxonomy. GLM led JavaScript, 90% to 75%, plus concurrency and durability, 62% to 45%.

The taxonomy was generated from benchmark prompts by an LLM, according to Together's methodology note. Across the full task set, the models' per-task outcomes correlated at 0.65: each solved 88 tasks at least once, GLM alone solved 11, Fable alone solved seven, and seven beat both.

Further reading

Discussion across the web

Where this story is being discussed, in original context.

On X· 5 threads
TL;DR1 post
Pass@4 versus pass@11 post
Cost per rollout1 post
Routing errors1 post
Rust and serialization1 post
Share on X