OpenAI 감사 결과, SWE-Bench Pro의 30%에 결함
OpenAI는 널리 사용되는 SWE-Bench Pro 코딩 벤치마크의 작업 중 30%가 엄격한 테스트와 모호한 프롬프트로 인해 결함이 있으며, 그 결과 최근의 성능 향상이 오해를 불러일으킬 수 있다고 추정합니다.
openai audited the swe-bench pro coding benchmark—a dataset used to track agentic coding capabilities where frontier models recently jumped from 23.3% to 80.3% pass rates—using a pipeline of agent-assisted filters and senior engineer reviews. the investigation revealed that ~30% of the 731 tasks are broken due to misleading prompts, low-coverage tests, or overly strict requirements that invalidate functionally correct code. this high error rate implies that current benchmark results may be inflated, misrepresenting actual model progress and complicating deployment decisions under safety frameworks. openai advises developers to scrutinize results and highlights the increasing necessity of using agents to scale data quality checks.