OpenAI 審計發現 SWE-Bench Pro 中 30% 的任務存在問題
openai 估計廣泛使用的 swe-bench pro 編碼基準測試中有 30% 的任務存在缺陷，原因是測試過於嚴格且提示含糊，這使得最近的性能提升可能具有誤導性。
openai audited the swe-bench pro coding benchmark—a dataset used to track agentic coding capabilities where frontier models recently jumped from 23.3% to 80.3% pass rates—using a pipeline of agent-assisted filters and senior engineer reviews. the investigation revealed that ~30% of the 731 tasks are broken due to misleading prompts, low-coverage tests, or overly strict requirements that invalidate functionally correct code. this high error rate implies that current benchmark results may be inflated, misrepresenting actual model progress and complicating deployment decisions under safety frameworks. openai advises developers to scrutinize results and highlights the increasing necessity of using agents to scale data quality checks.