How enabling two settings tripled our scores on the ARC-AGI-3 benchmark

A benchmark designed to measure how well AI agents learn and reason by exploring unfamiliar 2D games revealed surprisingly low scores, but researchers found that the poor performance resulted from how the test was set up rather than model limitations. Specifically, the benchmark's harness was discarding the model's private reasoning after each action and using a rolling truncation system that caused older actions to become invisible as history accumulated, forcing the model to relearn the game repeatedly. Enabling two settings called retained reasoning and compaction, which preserve the model's thinking process and use a summarization approach instead of discarding old information, tripled the scores while reducing output tokens by six times. This finding illustrates that benchmark results measure not just the capabilities of AI models themselves but also invisible choices about settings, harness design, and other technical configurations.