New SOOHAK benchmark exposes AI math failure
a new benchmark called soohak reveals frontier models score under 30 percent on research-level math and fail to recognize unsolvable problems.
developed by carnegie mellon, eleutherai, and seoul national university, soohak features 340 research-level problems and 99 unsolvable 'refusal' tasks to test calibration. while gemini 3 pro achieved 61 percent coverage on a subset compared to human mathematicians' 51 percent, the benchmark reveals that increased compute improves solving but not the ability to admit a problem has no answer.