Independent DeepSWE retest puts Ox Alpha at about 63%
A practitioner’s larger DeepSWE run reported roughly 63% for Ox Alpha, revising an earlier result near 80% from a 10-task subset. Testers report capable long-task work but note dead code and incomplete fixes.
TL;DR
- Ox Alpha’s viral 80% DeepSWE result came from ten tasks, and davis7’s initial table warned that the model’s real score could vary substantially.
- A larger DeepSWE subset landed at about 63%, 17 points below the first result, according to davis7’s revised run.
- The revised number sat two points below Fable 5 Medium and eight to 11 points below the Sol and Opus configurations that theo listed for reference.
- Long-horizon work produced usable output alongside cleanup failures: one review reported 14 significant bugs across 45 Ox-generated commits in stuhlmueller’s experiment.
DeepSWE’s public repository describes a benchmark of original long-horizon engineering tasks with isolated environments and program-based verifiers. The OpenRouter listing still calls Ox Alpha an anonymously operated preview, even as the community has treated its score as a proxy for a named model’s standing.
The 80% number was a 10-task sample
The first table recorded 80% for Ox Alpha, 65% for Fable 5, 62% for GLM-5.3 and Grok 4.6, and 52% for GPT-5.6 Sol. A contemporaneous warning from daniel_mac8 put the sample at 10 of DeepSWE’s 113 tasks.
Each pass or failure moves a ten-task mean by 10 points. The original result therefore represented eight passing tasks, while the benchmark’s methodology overview says its full task set spans 91 repositories and five languages.
The retest landed at 63%
The later run ended at roughly 63%. Davis said in davis7’s correction that winkey_h, whom he described as a DataCurve researcher and DeepSWE’s creator, ran it; he also called it a larger, more useful subset than his own first pass.
Theo’s reference comparison placed the new result beside these reported configurations:
- Fable 5 Medium: 65%, two points higher
- GPT-5.6 Sol xhigh and max: 71% and 73%, eight and 10 points higher
- Opus 5 high and max: 73% and 74%, 10 and 11 points higher
The 63% run remains a subset, not a published 113-task leaderboard entry. It replaces an apparent 15-point lead over Fable in the first sample with a two-point gap in the other direction.
Dead code and long context
Davis’s hands-on notes credited Ox with handling subagents and long, complex work, while flagging dead code, incomplete cleanup, and long waits at higher reasoning levels.
Stuhlmueller reported that Fable’s review of 45 commits produced by Ox found 14 significant bugs plus about a dozen smaller ones, including unmatched regexes, impossible conditionals, hangs, cache invalidation issues, and documentation corrections. In a separate long-context trial, rishdotblog reported in a test of an infinite loop that Ox could identify its own looping after an interruption, then resume the same loop.
The provider is still anonymous
OpenRouter says on the model page that a third-party provider operates Ox Alpha, retains prompts and completions, and has chosen to remain anonymous during the preview.
A forensic thread by chetaslua attributed the serving stack to Zhipu after malformed requests sent through OpenCode exposed a Java class name and API route matching Zhipu’s public interface. That is evidence about the server path, while OpenRouter’s listing still supplies no public developer name or release identity.