OpenAI 稱 30% 的 AI 程式碼基準已損壞
openai 發現 swe-bench pro 中 27% 至 34% 的任務存在缺陷，因此撤回了对該基準的認可。這意味著近期 ai 編碼能力的提升可能被誇大。
openai reviewed swe-bench pro, a standard for measuring ai programming skills, and found nearly a third of its tasks are flawed. using codex agents and human reviewers, the team identified issues where tests were too strict, vague, or incorrectly specified, often because they were pulled from human commit histories rather than designed for ai. the company is withdrawing its endorsement of the benchmark and calling for more reliable tests. this matters because benchmark results directly inform model release decisions and safety assessments under frameworks like preparedness.