arXiv:2607.18084: WorldCupArena tests language models on predicting football World Cup results

How does WorldCupArena test language model predictions?

Zhaokai Wang and colleagues, in the paper “WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting” (arXiv:2607.18084), present WorldCupArena, a dynamic benchmark — a test framework whose set of tasks updates in parallel with a real, ongoing event, rather than remaining fixed from the moment of publication. It uses FIFA World Cup 2026 matches as its task source, and tests both classic language models and deep-research agents, AI systems that autonomously search additional data sources such as player statistics or past results before answering.

Three levels of prediction: outcome, score, lineup

Models in the benchmark provide predictions at three levels of detail: the basic match outcome (win, loss, or draw), the exact final score, and the predicted player lineup for each team. The authors show that models with similar accuracy at the basic outcome level can diverge significantly from one another when compared at the finer levels — exact score and lineup — indicating that the coarse outcome metric hides real differences in prediction quality.

The best system versus bookmaker and human baselines

The best tested system achieves only modest improvements over bookmaker and human baselines when compared solely on a match’s exact score. The difference becomes clearer in the specialized scoring metric the authors introduce for a finer assessment of prediction quality, where the best system achieves a more noticeable advantage than in the coarser exact-score comparison.