arXiv: DiRL — a direction that distinguishes reasoning from memorization in RL training of language models

A multi-university research team published on June 9, 2026, on arXiv a paper proposing a new reinforcement learning (RL) method for language models called DiRL (Direction-Aware Reinforcement Learning). Authors Jiangnan Xia, Yucheng Shi, Yu Yang, Kishan Panaganti, Zhenwen Liang, and Ninghao Liu identified a fundamental problem that had not been adequately addressed: standard exploration methods reward trajectory diversity, but do not distinguish whether that diversity stems from genuine reasoning or memorized shortcuts.

The core problem: every new path is not a new idea

When a model generates a new token sequence that differs from previous attempts, existing methods register that as successful exploration and assign a reward. But the authors show that a trajectory can look new for two completely different reasons:

• The model applied a novel logical procedure — genuine reasoning
• The model varied the surface form but relied on the same memorized patterns and shortcuts

Treating both cases identically means the reward system unwittingly encourages the model to optimize through memorization rather than by developing general reasoning strategies. This is especially dangerous because models that memorize shortcuts can appear competent on test sets close to training data — but fail easily as soon as they encounter a problem formulated differently or requiring a different logical step.

What is DiRL and how does it distinguish reasoning from memorization?

DiRL solves the problem by introducing a reasoning-memorization direction extracted directly from the model’s internal representations. This is a latent vector that describes which pole — reasoning or memorization — the model moves toward when generating a given trajectory.

Based on this direction, direction-weighted gradient features are constructed: a mathematical description that assigns to each model parameter update a measure of how close it is to the reasoning pole versus the memorization pole.

Rewards are then shaped according to this measure:

• Exploration aligned with the reasoning direction is reinforced
• Variations aligned with the memorization direction are suppressed

The key practical advantage is that DiRL integrates into standard GRPO (Group Relative Policy Optimization) without any architectural changes to the model. It requires no special layers, external classifiers, or costly manual data labeling — everything needed is contained in the internal representations the model already builds during training.

Evaluation on mathematical and general benchmarks

The authors evaluated DiRL on a set of mathematical and general reasoning benchmarks, where the method shows significant improvements compared with existing diversity-exploration approaches. The paper is structured across 12 pages with 6 figures that visualize the difference in model behavior with and without DiRL.

Consistent improvements across different task types suggest that the reasoning-memorization direction is a generic model property — not an artifact of a specific dataset or domain. The authors emphasize that this generality is the method’s core advantage over approaches optimized for narrow task categories.

Why the reasoning-memorization distinction goes beyond mathematics

The implications of this work extend beyond mathematical tasks. Whenever a language model is trained with reinforcement learning — for instruction following, code solving, step planning in agentic systems, or multi-step logical reasoning — there is a latent risk that the model optimizes the reward through pattern memorization rather than by developing general strategies.

Previous exploration methods tried to address the problem by increasing diversity at the token or gradient level, but without insight into what that diversity actually means for the model’s internal state. DiRL brings that insight directly from the model’s representations: instead of measuring diversity at the surface level, it measures it where the difference between reasoning and memorization structurally manifests.

Open methodology under a CC BY 4.0 license

The methodology is published under a CC BY 4.0 (Creative Commons Attribution 4.0) license, making it freely available for use, adaptation, and further development in academic and industrial research.

For research teams working on RL post-training of language models, DiRL offers a principled and architecturally lightweight tool for improving exploration quality — without needing to change the foundations of the pipeline. The preprint with code and method details is available on arXiv (ID: 2606.10346).