OpenAI audits SWE-Bench Pro and finds 30% of public tasks broken

• SWE-Bench Pro lost its preferred-eval status at OpenAI: the company says 30% of public tasks are broken and is retracting its earlier recommendation to use it as a leading coding benchmark OpenAI's audit thread.
• The failure modes are concrete: hidden requirements, contradictory instructions, overly strict tests, and incomplete grading criteria appear in OpenAI's issue breakdown.
• The audit was not just model vibes: OpenAI says it used model-based investigator agents plus independent reviews from five experienced software engineers OpenAI's QA pipeline.
• The timing hit a crowded leaderboard moment, with SWE-Bench Pro screenshots showing frontier systems clustered around the 60% to 80% range Kyle Jeong's leaderboard screenshot and one reaction calling the benchmark “saturated at 70%” teortaxesTex.
• The open question is how broken tasks skew model rankings: Nick Moran asked what share of flagged tasks frontier models pass versus fail Nick Moran's question, while matvelloso said the scores had not matched real-world usage for a while matvelloso.

OpenAI’s [official writeup]OpenAI audit link retracts its earlier SWE-Bench Pro recommendation, then shows a quality-assurance pipeline with automated screening, agent review, and five-reviewer human adjudication. The attached chart says “overly strict tests” were the largest issue category, with human annotations flagging 17.8% of the dataset. A separate leaderboard screenshot had Fable max at 80.4%, Opus 4.8 max at 69.2%, Grok 4.5 at 64.7%, Opus 4.7 max at 64.3%, GLM 5.2 at 62.1%, and GPT 5.5 xhigh at 58.6% Kyle Jeong's post.

OpenAI says SWE-Bench Pro “no longer reliably measures frontier coding capability” and found 30% of public tasks broken in its audit thread. That is the story: a widely cited coding benchmark now has an official warning label from the lab that previously recommended it.

The phrase “broken” covers tasks where a correct solution can fail for reasons outside the model’s coding ability. OpenAI’s listed causes were hidden requirements, contradictory instructions, overly strict tests, and incomplete grading criteria OpenAI's issue summary.

OpenAI’s chart splits the flagged dataset by issue category across human-supervised agent review and human annotations.

The human-annotation rates in the chart were:

1. Overly strict tests: 17.8%.
2. Low-coverage tests: 9.4%.
3. Misleading prompt: 7.5%.
4. Miscellaneous issues: 1.2%.
5. Underspecified prompt: 0.8%.

The agent-assisted review produced the same top category, with overly strict tests at 14.4% OpenAI's chart. That makes the benchmark problem less abstract than “contamination” or “saturation”: many tasks appear to have grader behavior that can reject valid work.

OpenAI described a three-part audit flow: automated screening, human-supervised agent review, and a human reviewer campaign OpenAI's pipeline screenshot. The human reviewer campaign used five independent experienced software engineers per task, according to OpenAI.

The pipeline structure was:

1. Initial data quality pipeline: model rollouts, patches, diffs, and task metadata fed into automated screening.
2. Human-supervised agent review: exhaustive agent checks ran on flagged tasks, then a researcher made the final judgment.
3. Human reviewer campaign: experienced software engineers reviewed flagged tasks, with five independent reviewers per task.

That methodology detail matters more than the headline percentage. OpenAI is effectively saying benchmark maintenance now needs agents to audit agent benchmarks, with humans still holding the final call.

SWE-Bench Pro was still being used as a leaderboard surface while the audit landed. Kyle Jeong pointed at the timing around a Grok 4.5 launch, and his screenshot showed multiple frontier coding systems packed into the same leaderboard range Kyle Jeong's post.

The visible SWE-Bench Pro scores in the screenshot were:

• Fable max: 80.4%.
• Opus 4.8 max: 69.2%.
• Grok 4.5: 64.7%.
• Opus 4.7 max: 64.3%.
• GLM 5.2: 62.1%.
• GPT 5.5 xhigh: 58.6%.

teortaxesTex reacted to a separate benchmark chart with “SWE-bench pro is saturated at 70%,” pointing at memorization-filtered pass-rate curves across SWE-bench variants teortaxesTex. The stronger version of the critique is not that high scores are fake; it is that a public benchmark with broken tasks and near-frontier clustering stops telling engineers which model will actually survive a repo.

matvelloso said the feeling had been building that model scores were not matching real-world usage, and OpenAI’s clarification surfaced that gap more clearly matvelloso. That reaction explains why a task-quality audit traveled outside benchmark circles.

A separate commenter, iScienceLuvr, framed the issue as broader than coding evals and said many medical AI evals are also broken or bad iScienceLuvr. The useful boundary here is narrow: OpenAI’s evidence is about SWE-Bench Pro, while the reaction shows how quickly benchmark distrust generalizes.

Nick Moran asked the obvious ranking question twice: what percentage of flagged tasks are passed by frontier models versus failed by frontier models Nick Moran's first question Nick Moran's follow-up. That split determines whether broken tasks mostly punish strong models, reward brittle benchmark-specific behavior, or inject noise across the board.

OpenAI’s public thread gives the 30% broken-task finding and issue categories, but the cited tweets do not include a pass-versus-fail breakdown for flagged tasks. Until that distribution is public, the audit weakens SWE-Bench Pro as a headline ranking tool more than it rewrites any single leaderboard row.

emollick called OpenAI’s metrics discussion confusing because the company had spent heavily on GDPval, a benchmark of autonomous model ability at hard tasks, but had not reported it for GPT-5.6 emollick. Matthew Berman split the eval world into public evals for labs and private evals for customers Matthew Berman.

That is the lasting tension from this audit. Public benchmarks are legible and gameable; private evals may be more relevant and less comparable. OpenAI’s SWE-Bench Pro retraction makes that tradeoff explicit without resolving it.