How enabling two settings tripled our scores on the ARC-AGI-3 benchmark

Source: OpenAI Date: 2026-07-29 URL: https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores

Summary

Direct fetch of the OpenAI post returned HTTP 403; this summary is grounded via corroborating secondary coverage (officechai.com, dotnetramblings.com, ARC Prize’s own response on X) rather than the primary source. OpenAI found that enabling two API settings — retained reasoning and compaction — raised GPT-5.6 Sol’s score on the ARC-AGI-3 public set from 13.3% to 38.3% while cutting output tokens by roughly 6x. The default evaluation harness had been discarding the model’s private reasoning after every move and dropping older actions once history exceeded the context window, effectively forcing it to relearn each game from scratch on every turn.

Implications

• Model capability clocks: a large apparent benchmark jump turned out to be a harness/memory-retention artifact rather than a new model or capability step — a caution against reading eval deltas as pure intelligence gains without checking scaffolding assumptions.
• Context/token efficiency: compaction simultaneously improved the score and cut token usage, reinforcing that better context management, not more context, is the lever for long-horizon agentic tasks.
• Agentic engineering patterns: ARC Prize’s public reply — that their verified scores intentionally use a “no-harness” approach to keep cross-provider comparisons fair — surfaces a real tension between benchmark fairness and realistic deployed-agent configuration.