ArXiv Token Arena: continuous benchmark unifying energy and cognition reveals 6.2× difference in joules per correct answer across endpoints

The team of Yuxuan Gao, Megan Wang, and Yi Ling Yu published on May 1, 2026 on ArXiv Token Arena — a continuous benchmarking platform that evaluates AI inference at the endpoint level. The goal of the paper is to unify energy and cognition dimensions in a single measurement framework.

What does Token Arena measure that other benchmarks miss?

Standard AI benchmarks (MMLU, HumanEval, GSM8K) measure model quality under ideal laboratory conditions — without energy, cost, or latency dimensions. Token Arena takes a different approach: it measures the specific combination of provider, model, and configuration as the fundamental unit of measurement.

The reason: in real production, an application does not consume “model X” — it consumes an endpoint at a specific provider with a specific quantization, specific batch settings, and specific hardware backend. The same GPT-4 model through Open Router may be an order of magnitude faster or five times cheaper than directly through the OpenAI API, depending on routing.

The platform evaluates five dimensions simultaneously:

1. Output throughput (tokens/sec)
2. Time to first token (TTFT, critical for interactive applications)
3. Blended price (combined input and output cost)
4. Effective context (how much long-context capability the model actually uses, not the nominal limit)
5. Quality (math, code, reasoning, not just an MMLU average)

Synthesized into three composite indicators: energy efficiency, cost per correct answer, and endpoint fidelity.

What surprising differences did Token Arena discover?

Measurement across 78 endpoints in 12 model families revealed differences larger than the industry narrative suggests:

• Up to 12.5 points difference in quality for the same model on different endpoints on math/code benchmarks
• An order of magnitude difference in tail latency (p99) — some endpoints are 10× slower in worst-case scenarios
• A factor of 6.2 difference in joules per correct answer

The last figure may be the most significant. If two endpoints of the same model differ 6.2× in the energy required to generate a correct answer, provider choice becomes a sustainability strategy question, not just a cost question. The carbon footprint of AI inference operations in 2026 is no longer trivial; differences between endpoints mean that some AI deployments emit nearly seven times more CO₂ than others for the same result.

What does this mean for enterprise provider decisions?

The main takeaway: endpoint matters more than model name. A team that selects a provider based solely on price per token may end up with 12.5 points worse quality or 6× greater energy cost — without knowing it without benchmarking that covers all five dimensions.

Token Arena is published under the CC BY 4.0 license, meaning other organizations can reuse results and methodology. This is rare for AI benchmarks — most commercial benchmark suites remain under restrictive licenses. The open license supports an ecosystem of independent reproducibility studies.

The paper is available on ArXiv under ID 2605.00300.