arXiv:2607.15439: Verification Coding Agents Fully Solve ARC-AGI-3 with Around 99% RHAE

What is ARC-AGI-3, and which agent variants does Sergey Rodionov test?

ARC-AGI-3 is a general intelligence benchmark that frames tasks as interactive games — the agent must discover the rules of the environment and act within it, rather than solving static puzzles as in earlier ARC versions. Sergey Rodionov tests four nested variants of Codex-based agents (agents built on the foundation of OpenAI’s Codex approach to generating and executing code) to isolate the contribution of three capabilities: an executable world model — an internal simulator that predicts the consequences of actions by executing code, instead of textual guessing — task simplification, and explicit verification of the result before the final move.

The verification variant wins in all settings

The verification variant, which checks its own solution against an internal model of the environment before every move, proves to be the best across all tested settings — though at significantly higher computational resource cost than the other three variants. That trade-off is expected: the extra verification step means additional model calls and longer execution time per task, but the outcome justifies the cost.

The verification variant solves the entire public game set

With the gpt-5.6-sol model, the verification variant fully solves every public game in ARC-AGI-3 at both tested reasoning-effort levels (lower and higher degrees of computational “thinking” before a move), achieving approximately 99% RHAE — a metric measuring the share of successfully solved actions relative to the total number of attempts. Rodionov describes this as “saturation of the public set,” a state in which the benchmark no longer distinguishes more capable agents because it is nearly fully solved.

Public set saturation and the future of the benchmark

The author warns that held-out performance — results on the hidden, non-public set of games — remains untested with a newer model like gpt-5.6-sol, so the actual capability ceiling of agents cannot be determined from public results. Saturation of the ARC-AGI-3 public portion signals that future benchmarks will need to rely on hidden evaluations to retain discriminative value.