OpenAI 称 30% 的 AI 代码基准已损坏
openai 发现 swe-bench pro 中 27% 至 34% 的任务存在缺陷，因此撤回了对该基准的认可。这意味着近期 ai 编码能力的提升可能被夸大。
openai reviewed swe-bench pro, a standard for measuring ai programming skills, and found nearly a third of its tasks are flawed. using codex agents and human reviewers, the team identified issues where tests were too strict, vague, or incorrectly specified, often because they were pulled from human commit histories rather than designed for ai. the company is withdrawing its endorsement of the benchmark and calling for more reliable tests. this matters because benchmark results directly inform model release decisions and safety assessments under frameworks like preparedness.