VRRL: Reinforcement Learning Forces Visual Models to Actually Use the Image During Self-Correction

Self-correction capability is one of the key characteristics demanded of vision-language models (VLMs) in agentic applications. When a model makes an error, it should recognize it and fix it — ideally by using the original visual input as a source of truth.

The problem, documented by Liyan Tang, Fangcong Yin, and Greg Durrett, is that existing VLMs do not do this correctly. When entering a self-reflection phase, models tend to rely on prior linguistic context rather than genuinely looking at the image again. The result is corrections not grounded in the visual input — the model changes its answer, but not because it visually verified its mistake, but because it shifted linguistic patterns.

Why Do Standard Approaches Fail to Solve Visually Ungrounded Self-Reflection?

Standard fine-tuning improves overall accuracy but does not target the specific problem of error correction conditioned on visual input. Reflection-oriented fine-tuning teaches the model the format of self-reflection, but without any guarantee that the correction will be genuinely grounded in the image. A model can produce a correctly structured reflection that entirely ignores the visual evidence.

Reinforcement learning (RL) offers a better starting point because the reward can signal final answer accuracy. But standard RL does not force the path to the correct answer to pass through visual verification — the model can learn correct answers through shortcuts in linguistic space. VRRL (Visually Grounded Self-Reflection via Reinforcement Learning) addresses precisely this gap.

Two Technical Innovations Within the VRRL Framework

VRRL builds on an RL framework with two specific modifications designed to force visually grounded correction.

The first is trajectory prefix masking. During training, the initial steps of the trajectory — including the initial error — are masked from the RL signal. The model receives reward or penalty exclusively based on what it does during the correction phase. In this way, the optimization pressure is directed toward how to correct an error, not just how to avoid it from the very start. The correction must be grounded in something — and the only thing the model has available during the reflection phase, besides prior text, is the original image.

The second technique is buffered roll-ins — an experience replay mechanism that builds a diverse pool of failure trajectories from past training epochs. Instead of the model always starting from the same or similar errors, the roll-in buffer exposes it to a wide range of failure modes. This prevents overfitting to a specific error type and improves generalization on out-of-distribution examples — critical for agentic systems that encounter unpredictable visual inputs.

Results: Significantly Better OOD Performance

VRRL was evaluated on visual grounding — tasks requiring localization and interpretation of objects in tables and charts — and on spatial navigation tasks that test the ability to track visual relationships across sequences of images.

Across all tested configurations, VRRL achieves significantly better performance on out-of-distribution examples (OOD) compared to standard RL baselines and reflection-oriented fine-tuning. OOD evaluation is particularly relevant for agentic applications because models in production regularly receive visual inputs that differ from the training distribution — and this is precisely where standard approaches fail.

Broader Context for Agentic VLM Architectures

VRRL targets a specific and practically important failure mode: visual verification that does not actually change behavior. In agentic loops where a VLM iteratively executes actions, observes a visual feedback signal, and adjusts its plan, this gap has direct operational consequences — a model that reflects without visual grounding merely propagates the same errors in new formulations.

The methodological contribution of the paper lies not only in better benchmark numbers. VRRL demonstrates that the choice of what is masked and what is replayed in RL training can deliberately force a specific cognitive mode in the model. For researchers building visual agents, this opens space for designing RL algorithms that explicitly target capabilities such as causal visual reasoning or spatial tracking, rather than relying solely on a global reward for answer accuracy.