rotative
the decoder · November 22, 2026

AI coding agents struggle in simulated workplace

ai coding agents complete less than 1 in 10 tasks in simulated software company. this reveals gap between benchmark performance and real-world application.

read in 简体中文 · 繁體中文

ai coding agents have shown impressive results on standard benchmarks, with anthropic's claude writing over 80% of the company's production code. however, in a simulated software company, these agents completed fewer than 1 in 10 tasks. claude opus 5.5 and gpt-6 astra were tested as digital employees with colleagues, project boards, and chat histories. the experiment reveals a significant gap between benchmark performance and real-world application. this challenges the deployment of ai in software development.

read at source ↗

← rotative