Reasoning stalls AI in cloud ops, new benchmark shows
new open-source benchmark cloudopsbench reveals ai agents struggle with reasoning in devops tasks, failing 40-47% of tasks on aws.
cloudopsbench, developed by the university of michigan and led by ang chen, is an open-source benchmark that evaluates ai agents on devops tasks in cloud operations. version 1.0 includes 45 tasks on aws, and despite testing top models like claude fable 5.1 and gpt-6 astra, none completed all tasks. the primary bottleneck identified is reasoning, with about 80% of failures caused by agents misjudging task scope, not checking all relevant services, or stopping prematurely. this benchmark runs tasks identically in real aws and an emulator, with plans for extensions to gcp, azure, and oci. the project is licensed under apache 2.0 and available on github.