AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test

Researchers developed a new benchmark test for AI coding agents that requires them to complete real-world web app modification tasks based on user requests, including those with images, audio, video, or documents as input. The best model tested fully completed only about one in four tasks correctly, though it met roughly 70 percent of requirements on average, meaning human cleanup is needed in nearly three out of four cases. Existing coding benchmarks have become too easy, with models scoring above 99 percent, so this new test draws from actual difficult coding sessions where users had to rephrase requests or roll back changes. The research also found that performance depends heavily on both the model and the working environment through which the agent operates, suggesting that leaderboards reporting only model names tell an incomplete story.