Retained reasoning and compaction nearly tripled GPT-5.6 Sol’s ARC-AGI-3 score
On July 29, 2026, OpenAI reported that two Responses API settings—retained reasoning and compaction—raised GPT-5.6 Sol’s score on the public ARC-AGI-3 set from 13.3% to 38.3%. The revised harness also cut output-token use by roughly sixfold.
Why the harness changed the result
ARC-AGI-3 tests whether agents can infer the rules of unfamiliar 2D games without instructions. Performance is measured through Relative Human Action Efficiency, or RHAE. OpenAI estimated an average human score of 48% from official gameplay logs, while models receive only text representations of frames and cannot see their running score.
The official harness deliberately stays generic. After every action, however, it discarded private reasoning; once the history exceeded 175,000 characters, rolling truncation removed the oldest actions. GPT-5.6 Sol therefore had to reconstruct plans repeatedly and gradually lost observations needed to learn a game.
What the two settings changed
OpenAI rebuilt the harness with the Responses API. Passing the previous response ID retained reasoning across turns, reducing the need to reinterpret each state. Compaction replaced deletion with summaries, preserving useful discoveries while keeping context manageable.
The test implementation used a 175,000-token limit, which OpenAI said was comparable because action grids tokenize at close to a 1:1 ratio. With both settings active, the model learned more coherent strategies, acted faster and used fewer output tokens. In one illustrated game, it solved all six levels where no frontier model on that leaderboard passed the first.
Evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting.
OpenAI recommends the Responses API rather than the legacy Chat Completions API, with reasoning retention and compaction enabled. The result adds operational detail to GPT-5.6 gains in intelligence and token efficiency, showing that model quality and token efficiency depend partly on deployment architecture.
For businesses, the practical lesson is to benchmark the complete agent stack, not only the model name. Before rejecting a model or scaling spend, teams should test memory retention, context compression and tool-loop design under production-like workloads; otherwise an evaluation may measure an avoidable integration bottleneck.