Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize

An agent harness is the code framework around a model that handles execution loops, tools, context, state management, and verification. Researchers tested whether large language models could engineer their own harnesses by starting with minimal seed code and building full systems for different task domains. The models' self-built harnesses performed comparably to human-engineered versions on writing and ML experimentation tasks but significantly underperformed on code and search tasks. When the models attempted to improve their harnesses through iterative changes based on feedback, only 34 of 64 changes showed the same direction of improvement on held-out test tasks, suggesting that harness optimization through model iteration remains unreliable and inconsistent.