Direct-OPD: How a Weaker Model Can Train a Stronger One Without Expensive RL

Training language models with reinforcement learning on verifiable rewards — the so-called RLVR approach — has produced some of the most impressive improvements in the reasoning capabilities of modern AI models. But this approach carries a practical cost: every new, more powerful model requires its own expensive RL campaign, in which it must generate thousands of rollouts while learning from rewards. As models grow in size, post-training itself becomes the scaling bottleneck.

Researchers from Tsinghua University and Zhipu AI proposed an elegant solution to this problem in a paper published July 6, 2026 — a method called Direct On-Policy Distillation (Direct-OPD).

Why Is RL Expensive for Every New Model?

The previous answer to this problem was an approach known as “weak-to-strong” distillation: run RL on a smaller, cheaper model and use that model as a teacher for the stronger target model. But naive implementation of this approach does not work well. The final policy of the weaker RL-trained model mixes the useful gains that RL brought with the inherent limitations of the smaller model. When the stronger model imitates such a policy, it takes on the limitations along with the gains.

The problem is conceptual: direct distillation of the final policy does not transfer what RL actually learned — it also transfers everything that the stronger model already knows better than the weaker one. Instead of a clean RL signal, the stronger model receives a package that includes the weaknesses of the smaller model.

Direct-OPD: Transferring the Shift, Not the Policy

The key insight behind Direct-OPD is methodologically precise: rather than imitating what the weaker model says, Direct-OPD extracts what RL changed in the weaker model.

The method compares the post-RL checkpoint of the weaker model with its pre-RL reference checkpoint. The log-ratio of those two distributions reveals which actions RL made more probable and which less probable — without that signal being mixed with the inherent weaknesses of the smaller model. This log-ratio is then used as a dense implicit reward for the stronger student, which generates rollouts on its own on-policy states.

The result is a methodology that directly transfers the RL supervisory signal across scale levels, without training an explicit reward model and without running sparse-reward RL on the large target model. The computational cost falls on the weaker model, and the signal benefits the stronger one.

Results: +14.1 Percentage Points on an Elite Benchmark

The empirical results are unambiguous. On the AIME 2024 benchmark — a set of competitive math problems that is the standard measure of reasoning in the strongest models — Direct-OPD improved Qwen3-1.7B from 48.3% to 62.4%, a jump of 14.1 percentage points.

That result was achieved in just 4 hours of training on 8 A100 GPUs — a duration that demonstrates the practical feasibility of the approach outside a laboratory context. Moreover, Direct-OPD outperforms direct RL on the stronger model with the same number of steps (step-matched direct RL): for the same compute expenditure, it achieves a better result.

The method also exhibits another valuable characteristic: it enables sequential stacking of multiple RL improvements. Shifts from several different RL experiments can be applied successively — as modular layers that combine without mutually canceling each other. This opens up the possibility of building more complex improvement chains that leverage the results of multiple specialized RL campaigns.

Implications for Superalignment and Scalable Oversight

The Direct-OPD research touches on one of the fundamental open problems in AI safety: superalignment, or scalable oversight. The question is how we can supervise AI models that are more capable than their supervisors.

Direct-OPD offers a concrete contribution: it demonstrates that RL gains can be transferred upward in scale without losing their essence. A weaker model that has undergone RL training can function as a calibrator for a stronger one without transferring its own weaknesses. The authors explicitly articulate this lesson — RL outcomes can be used as implicit reward signals across model scale levels, not just as final models for imitation.

This opens research directions that include stacking successive RL improvements from different domains and building supervision hierarchies — systems where each level learns from a weaker level but surpasses its inherent limitations.

The paper is available on arXiv (2607.05394), and the project page at bytedtsinghua-sia.github.io/Direct-OPD/ announces the forthcoming public release of code and checkpoints.