arXiv:2606.09380: Trace tournaments solve the zero-gradient problem in RLVR training of language models

The zero-gradient problem in RLVR training

Reinforcement Learning with Verifiable Rewards (RLVR) is becoming the dominant paradigm for improving the reasoning capabilities of language models. The method rewards traces depending on the correctness of the final answer, and the advantage estimate is computed relative to the group of simultaneously generated samples.

The problem arises when the entire group of traces receives an identical reward — all correct or all incorrect. The standard group-relative approach then returns a zero gradient: the model receives no useful learning signal and training stagnates. This phenomenon is most pronounced on easier tasks (where all are correct) and on hard tasks (where all are incorrect) — precisely where precise gradient guidance matters most.

How do trace tournaments solve the dead signal?

Instead of a group comparison with equal rewards, Reasoning Arena organizes head-to-head tournaments between traces and estimates the relative strength of each via the Bradley-Terry probabilistic model on a dynamic set of anchors. Han Zhou, Adam X. Yang, Laurence Aitchison, Anna Korhonen, and Albert Q. Jiang presented this approach in the paper arXiv:2606.09380 (published June 8, 2026).

The key advantage is that a new trace does not need to be compared with all others — the expensive O(n²) approach — but only with a selected anchor. Even when traces share the same reward, differences in reasoning quality become visible through tournament outcomes. In this way, samples with zero advantage are converted into useful gradient steps, and information about intra-group differences is not fabricated — it is extracted from the reasoning process itself.

Results: faster, cheaper, and more accurate

Evaluation on competitive math and coding benchmarks records an accuracy improvement of +7.6% compared to the standard group-relative approach. In addition, training accelerates by 27 to 41 percent, and sample generation compute decreases by approximately 50 percent.

The authors conclude that Reasoning Arena is not only more efficient but also a cheaper alternative to existing RLVR training methods — particularly valuable for research groups with limited compute budgets that want to advance reasoning capabilities without a proportional increase in costs.