Separating signal from noise in coding evaluations

OpenAI conducted an audit of SWE-Bench Pro, a widely used benchmark for measuring how well AI models can perform coding tasks, and found that approximately 30 percent of the tasks contain fundamental flaws. The issues identified fell into four categories: overly strict tests that enforce specific implementation details not mentioned in instructions, underspecified prompts that omit requirements, tests with low coverage that allow incomplete fixes to pass, and misleading prompts that point models toward wrong behavior. Accurate evaluation is important because flawed benchmarks can give false understandings of model capabilities and affect decisions about deployment and safety. OpenAI used a combination of automated analysis and human review by experienced software engineers to identify these broken tasks, recommending that other model developers carefully examine their results on this benchmark.