arXiv:2606.17930: Benchmark Results Are Protocol-Dependent — Inference Compute Changes Frontier Model Rankings

A new preprint warns that benchmark results depend heavily on the measurement protocol, calling into question standard frontier model leaderboards.

What did the authors test?

The paper tests 12 frontier models on 7 demanding benchmarks from software, mathematics, medicine, and cybersecurity. The key variable is inference compute — the amount of computation, i.e. the token budget, the model is allowed to spend on solving a task. Standard evaluations typically fix that budget, but the paper shows it is precisely what matters most.

What changes with a larger budget?

A larger token budget significantly improves results on FrontierMath, Humanity’s Last Exam, TerminalBench, and cybersecurity tests. More importantly, models rank differently depending on that budget — the model that is best with a small budget is not necessarily best with a large one. This means fixed-budget evaluations systematically underestimate real capabilities.

Why does this matter for model assessment?

The authors recommend that model capability be reported as a function of inference-time compute, not as a single number. The finding is also relevant for security and policy assessments: evaluating models without controlling for compute budget can produce unreliable and misleading rankings.