AI evaluation costs hit $40k per run, becoming new compute bottleneck
huggingface reports ai evals are becoming the new compute bottleneck. the holistic agent leaderboard spent $40,000 on 21,730 rollouts. a single gaia run costs $2,829. static benchmarks compress well, but dynamic agent benchmarks remain expensive due to noise and scaffold sensitivity, with inference-time compute scaling costs.
Hugging Face reports that AI evaluation has become a major cost center. The Holistic Agent Leaderboard (HAL) spent $40,000 to run 21,730 agent rollouts, while a single GAIA run on a frontier model costs $2,829 before caching. Exgentic identified a 33× cost spread based on scaffold choice, highlighting that agent evaluations are inherently noisy and difficult to compress compared to static LLM benchmarks.
Historical data shows static benchmarks allow for significant compression—Flash-HELM achieved 100×–200× compute reductions with preserved rankings. However, these techniques fail for agentic systems. Training-in-the-loop benchmarks and the need for repeated reliable runs further multiply expenses, making evaluation the dominant compute line item for smaller models.