新基准揭示推理成为云运维中AI的最大瓶颈
新的开源基准cloudopsbench显示，ai代理在devops任务中推理能力不足，在aws上失败40-47%的任务。
cloudopsbench, developed by the university of michigan and led by ang chen, is an open-source benchmark that evaluates ai agents on devops tasks in cloud operations. version 1.0 includes 45 tasks on aws, and despite testing top models like claude fable 5.1 and gpt-6 astra, none completed all tasks. the primary bottleneck identified is reasoning, with about 80% of failures caused by agents misjudging task scope, not checking all relevant services, or stopping prematurely. this benchmark runs tasks identically in real aws and an emulator, with plans for extensions to gcp, azure, and oci. the project is licensed under apache 2.0 and available on github.