OpenAI、AIコードの30%が壊れていると発表
OpenAIは、swe-bench proのタスクの27%〜34%に欠陥があることを発見し、このベンチマークへの支持を撤回せざるを得なくなりました。つまり、最近のAIコーディングの進歩は過大評価されている可能性があります。
openai reviewed swe-bench pro, a standard for measuring ai programming skills, and found nearly a third of its tasks are flawed. using codex agents and human reviewers, the team identified issues where tests were too strict, vague, or incorrectly specified, often because they were pulled from human commit histories rather than designed for ai. the company is withdrawing its endorsement of the benchmark and calling for more reliable tests. this matters because benchmark results directly inform model release decisions and safety assessments under frameworks like preparedness.