arXiv:2606.03937: Entropy is not enough for visual reasoning, claims new RL framework

A new paper titled Entropy Is Not Enough questions an established practice in reinforcement learning for multimodal models. The authors, led by a team around Senjie Jin and Fudan University, argue that methods that work excellently for text fail as soon as the model has to reason over an image.

What is the problem with entropy?

Reinforcement learning (RL) trains a model by rewarding good decisions. To assign credit to individual tokens, entropy — a measure of the model’s uncertainty about the next word — is often used. The logic is that uncertain tokens are the most informative. But the authors warn that visually sensitive tokens — those that rely directly on image content — naturally have low entropy. The classic entropy approach therefore ignores them, even though they are key to a correct conclusion.

How does VEPO solve the problem?

The proposed framework is called VEPO (Vision-Entropy token-selection for Policy Optimization). It introduces multiplicative fusion: it directs gradient credit toward tokens that are at once visually grounded and highly informative. The product of the two measures means a token must satisfy both criteria to receive weight, avoiding the overestimation of purely textual but uncertain tokens.

How large are the improvements?

The results show consistent gains over baselines that use entropy alone: +2.28 points on models of 7 billion parameters and +3.15 points on smaller models of 3 billion parameters. The smaller scale yields a larger gain, suggesting that smart token selection is especially valuable when the model’s capacity is limited.