SWE-Bench Pro

Evaluating challenging long-horizon software engineering tasks in public open source repositories

SWE-Bench Pro is a Scale AI benchmark and leaderboard for evaluating AI coding agents on challenging long-horizon software engineering tasks across public, held-out, and commercial repositories.

Recent stories

1 linked story

newsPRIMARY2026-07-08

OpenAI audits SWE-Bench Pro and finds 30% of public tasks broken

OpenAI audited SWE-Bench Pro and found 30% of public tasks were broken. It retracted its earlier recommendation to use the benchmark as a leading coding eval.