新基準揭示推理成為雲運維中AI的最大瓶頸
新的開源基準cloudopsbench顯示，ai代理在devops任務中推理能力不足，在aws上失敗40-47%的任務。
cloudopsbench, developed by the university of michigan and led by ang chen, is an open-source benchmark that evaluates ai agents on devops tasks in cloud operations. version 1.0 includes 45 tasks on aws, and despite testing top models like claude fable 5.1 and gpt-6 astra, none completed all tasks. the primary bottleneck identified is reasoning, with about 80% of failures caused by agents misjudging task scope, not checking all relevant services, or stopping prematurely. this benchmark runs tasks identically in real aws and an emulator, with plans for extensions to gcp, azure, and oci. the project is licensed under apache 2.0 and available on github.