arXiv:2605.03871: EvoLM — language models that improve themselves without external supervision

A new paper from arXiv introduces a post-training method that entirely eliminates dependence on external supervision. EvoLM allows a language model to improve itself using so-called “discriminative rubrics” — explicit criterion scales that co-evolve alongside the policy model through iterative training.

What makes this approach different?

Classical RLHF (Reinforcement Learning from Human Feedback — a fine-tuning method using human ratings of model outputs) requires either human labels or a separate reward model trained on them. EvoLM instead uses temporal contrast: it compares older model outputs to newer ones and extracts signals for improving the rubrics from that difference.

The system structures the model’s inherent evaluation ability into explicit rubrics that are alternately trained with the policy. This closes a loop in which the generator and evaluator share the same foundation but advance asynchronously.

Numbers that move industry benchmarks

The Qwen3-8B rubric generator outperforms GPT-4.1 on RewardBench-2 by 25.7 percentage points, and SkyWork-RM (8B) — the previous state-of-the-art reward model — by 16 percentage points. The policy trained with this method reaches 69.3% on the OLMo3-Adapt evaluation suite.

This is a rare case where an open, relatively small model outperforms a frontier model in the evaluator role — which until now has been the domain of giant closed systems.

Why does this matter for the RLHF ecosystem?

If the results are confirmed through independent replication, EvoLM opens the path to cheaper and more transparent training. A reward model is a model that scores the output quality of another model, and a rubric is an explicit criterion scale. Open alternatives to GPT-4.1 and Claude judge systems are especially important for research teams and companies that do not want external API dependence during a critical training phase.

The question of how robust the method is against “reward hacking” when the model evaluates itself remains open — but results on public benchmarks suggest that temporal contrast provides sufficient protection against quality collapse.