What We Learned by Reproducing 2,200 papers from ICML

A hackathon challenged community members to reproduce papers from a major AI conference using coding agents, with over 1,200 participants attempting to verify claims from about a third of the conference's accepted papers. The effort matters because the number of papers submitted to conferences has grown exponentially while reviewer capacity has not, making it difficult for human reviewers to thoroughly check research before publication. Coding agents can now attempt detailed verification work in hours rather than the days or weekends required by human reviewers, enabling large-scale checking of scientific claims. The results found that about half of examined papers had at least one claim independently verified, roughly a quarter had at least one claim falsified or contested, and the remainder had incomplete or inconclusive evidence.