OpenAI says 30% of AI code is broken
openai found 27% to 34% of tasks in swe-bench pro are flawed, forcing it to withdraw endorsement of the benchmark. this means recent ai coding gains may be overstated.
openai reviewed swe-bench pro, a standard for measuring ai programming skills, and found nearly a third of its tasks are flawed. using codex agents and human reviewers, the team identified issues where tests were too strict, vague, or incorrectly specified, often because they were pulled from human commit histories rather than designed for ai. the company is withdrawing its endorsement of the benchmark and calling for more reliable tests. this matters because benchmark results directly inform model release decisions and safety assessments under frameworks like preparedness.