CloudOpsBench v1.0 shows reasoning is the main bottleneck for AI in cloud operations
CloudOpsBench is an open-source benchmark (v1.0) from a University of Michigan team led by Ang Chen that evaluates AI agents on 45 real-world DevOps tasks run on AWS. Top models performed poorly on many tasks (Claude Fable 5.1 ~60%, GPT-6 Astra ~53%); the authors say ~80% of failures stem from reasoning errors (wrong task scope, missed services, premature stopping) and ~20% from incorrect actions; the project (Apache 2.0) and a detailed failure analysis are on GitHub, with GCP/Azure/OCI extensions planned.