OpenAI, AI 코드의 30%가 깨져 있다고 밝혀
openai는 swe-bench pro의 작업 중 27%~34%가 결함이 있음을 발견해 해당 벤치마크에 대한 지지를 철회해야 했습니다. 이는 최근 AI 코딩 성능 향상이 과장되었을 수 있음을 의미합니다.
openai reviewed swe-bench pro, a standard for measuring ai programming skills, and found nearly a third of its tasks are flawed. using codex agents and human reviewers, the team identified issues where tests were too strict, vague, or incorrectly specified, often because they were pulled from human commit histories rather than designed for ai. the company is withdrawing its endorsement of the benchmark and calling for more reliable tests. this matters because benchmark results directly inform model release decisions and safety assessments under frameworks like preparedness.