AI coding agents struggle in simulated workplace
ai coding agents complete less than 1 in 10 tasks in simulated software company. this reveals gap between benchmark performance and real-world application.
ai coding agents have shown impressive results on standard benchmarks, with anthropic's claude writing over 80% of the company's production code. however, in a simulated software company, these agents completed fewer than 1 in 10 tasks. claude opus 5.5 and gpt-6 astra were tested as digital employees with colleagues, project boards, and chat histories. the experiment reveals a significant gap between benchmark performance and real-world application. this challenges the deployment of ai in software development.