OpenAI audit finds 30% of SWE-Bench Pro is broken
openai estimates that 30% of tasks in the widely used swe-bench pro coding benchmark are flawed due to strict tests and vague prompts, rendering recent performance gains potentially misleading.
openai audited the swe-bench pro coding benchmark—a dataset used to track agentic coding capabilities where frontier models recently jumped from 23.3% to 80.3% pass rates—using a pipeline of agent-assisted filters and senior engineer reviews. the investigation revealed that ~30% of the 731 tasks are broken due to misleading prompts, low-coverage tests, or overly strict requirements that invalidate functionally correct code. this high error rate implies that current benchmark results may be inflated, misrepresenting actual model progress and complicating deployment decisions under safety frameworks. openai advises developers to scrutinize results and highlights the increasing necessity of using agents to scale data quality checks.