arXiv:2606.28166: Tandem RL — Verifiable Rewards With More Readable Chain of Thought and Better Handoff to Smaller Models

Researchers from EPFL published a preprint on arXiv addressing a practical problem in modern language model training techniques for mathematical reasoning.

RLVR and the Readability Problem

RLVR (Reinforcement Learning with Verifiable Rewards) is a method that improves the reasoning capabilities of language models by rewarding correct, objectively verifiable answers — most commonly in competitive mathematics. Models in this process generate a “chain of thought”: an explicit step-by-step reasoning process. However, standard RLVR develops idiosyncratic patterns: language mixing, illogical structure, and poor readability — preventing the learned patterns from being used by weaker models or humans.

How Does Tandem RL Work?

Tandem RL (TRL) introduces a different approach: a stronger, trained model alternately generates the chain of thought together with a frozen weaker model. Both models share a common reward signal. In this way, the stronger model implicitly learns to write in a way that the weaker model can follow. The authors (Jiao, Singhal, West, Anderson — EPFL) trained TRL on the Qwen3-4B-Instruct model using competitive mathematics task sets.

TRL Outperforms Standard RLVR in Readability and Handoff Robustness

Results show that TRL achieves comparable solo performance to standard RLVR — without any drop in accuracy on its own tasks. The key difference lies in transfer quality: standard RLVR mixes languages and develops non-transferable patterns, while TRL generates significantly more readable chain-of-thought sequences. The paper identifies three emergent properties from the same training: better handoff to smaller models, lower distributional divergence, and more readable chain-of-thought. The paper is available as a preprint on arXiv (cs.AI, 21 pages).