arXiv: TruDi — diffusion policies for massively parallel on-policy RL

Diffusion policies hit a wall in parallel training

Reinforcement learning based on massive GPU parallelization has become the dominant paradigm for training agents that control robots and virtual characters. Yet while Gaussian policies operate stably in this setting, diffusion models — which have shown exceptional results in planning and generating complex movements — have consistently failed at this hurdle. The reason is precise: in an on-policy setting the training data distribution changes rapidly between update steps. Diffusion models, whose sampling process passes through many denoising steps, are particularly sensitive to this shift and easily become unstable.

What is TruDi and how does it solve this problem?

A team of researchers — Huy Le, Onur Celik, Denis Blessing, and collaborators — has proposed TruDi (Trust-Region Diffusion Policies): the first method that successfully enables diffusion policies to train in a massively parallel, on-policy RL setting. The central contribution of the paper is the application of trust-region optimization with a KL-divergence constraint that, unlike earlier approaches, is enforced across the entire diffusion trajectory rather than only on the final sampled result. This choice is not arbitrary: because the intermediate steps of the diffusion process are mutually dependent, constraining only the final sample is insufficient to prevent distribution shift along the trajectory. TruDi resolves this by explicitly tracking and constraining divergence at every step of the process, keeping training stable even under aggressive parallel updates.

Evaluation across 73 tasks and 4 benchmarks

To confirm the method’s generalization, the authors evaluated TruDi on 73 tasks distributed across 4 benchmarks for massively parallel RL. Results show that TruDi consistently outperforms or keeps pace with strong baselines on standard tasks.

The clearest advantages appear in the more demanding category: humanoid robot control. Humanoid control has traditionally posed a particular challenge because it requires coordinating a large number of degrees of freedom over long planning horizons — precisely the setting where the expressiveness of diffusion policies comes to the fore, but only if training is stable. TruDi delivers that stability.

Diffusion policies as a new strong baseline

TruDi’s practical contribution goes beyond a benchmark winner: the paper establishes diffusion policies as a new, legitimate baseline for massively parallel on-policy RL — a regime that had until now been almost exclusively the territory of Gaussian policies.

Gaussian policies have predictable and analytically tractable behavior, which makes them the natural choice when training stability is the priority. But their expressiveness is limited by the distributional shape they assume. Diffusion models impose no such constraints and are capable of modeling complex, multimodal distributions over actions — which is especially valuable for tasks such as humanoid locomotion or fine manipulation, where there may be multiple valid solutions.

TruDi makes that expressiveness available for the first time without sacrificing stability. This opens the possibility that future research on massively parallel RL can proceed with diffusion policies as a standard component rather than an exotic alternative.

Methodological precision as the key to success

It is worth underscoring why the level at which the trust-region constraint is applied matters. In classical trust-region algorithms the constraint is applied to the full policy viewed as a black box. When the policy is Gaussian, this is trivial — the distribution has a closed form. For diffusion policies the action distribution emerges as a product of a multi-step process with no closed form, making it considerably harder to estimate or constrain divergence.

TruDi resolves this problem by extending the constraint to the trajectory level of the diffusion process: KL-divergence is measured and enforced at every denoising step, ensuring that the policy update stays within a safe trust region at the level of intermediate steps as well, not just at the level of the visible action. This distinction makes the difference between a method that works and one that collapses into instability.

Implications for robotics and control

Massively parallel on-policy RL is currently one of the dominant approaches for training policies in physical robotics, particularly in simulation environments that run thousands of simulated instances simultaneously on a GPU. Integrating diffusion policies into that ecosystem — which TruDi now enables — potentially opens a path toward policies that are both fast to train and highly expressive in modeling actions.

The paper, submitted on June 13, 2026, arrives at a moment when the robotics community is actively seeking ways to exploit generative models within the training loop. TruDi offers one concrete, evaluated answer to a question that had until now remained without a satisfactory solution.