OpenAIの監査でSWE-Bench Proの30%に欠陥と判明
OpenAIの推定によれば、広く利用されているSWE-Bench Proコーディングベンチマークのタスクの30%が、厳格すぎるテストと曖昧なプロンプトにより欠陥を抱えており、近年の性能向上は誤解を招く可能性があります。
openai audited the swe-bench pro coding benchmark—a dataset used to track agentic coding capabilities where frontier models recently jumped from 23.3% to 80.3% pass rates—using a pipeline of agent-assisted filters and senior engineer reviews. the investigation revealed that ~30% of the 731 tasks are broken due to misleading prompts, low-coverage tests, or overly strict requirements that invalidate functionally correct code. this high error rate implies that current benchmark results may be inflated, misrepresenting actual model progress and complicating deployment decisions under safety frameworks. openai advises developers to scrutinize results and highlights the increasing necessity of using agents to scale data quality checks.