vLLM: vime — new RL framework that unifies Megatron training and vLLM inference in a single pipeline

The team behind vLLM (the fast LLM inference framework) published on June 9, 2026, a new framework for post-training language models called vime. The full name reveals the purpose: integration of Megatron distributed training and vLLM inference into a unified pipeline for reinforcement learning (RL). vime is built on top of the slime training stack and designed to bridge the gap between two established parts of deep-learning infrastructure — distributed training and distributed inference — which have until now mostly been used separately.

Architecture: three coordinated stages

vime uses an architecture with three distinct but tightly coordinated stages:

1. Training (Megatron): Megatron-LM handles parameter updates and model weight synchronization across a large number of GPUs. It is responsible for the actual gradient steps that improve the model.

2. Rollout (vLLM + router): vLLM, known for high-throughput LLM inference, generates samples (rollouts) — token sequences the model produces as responses or solutions to given problems. The router coordinates traffic distribution between the inference layer and the rest of the system.

3. Decoupled data buffer: An intermediary layer connecting the training and rollout stages, enabling injection of custom logic (reward filters, custom message formats) and bridging the speed difference between both stages.

This design is built on top of the slime training stack, ensuring compatibility with already-developed implementations of GRPO (Group Relative Policy Optimization) and PPO (Proximal Policy Optimization) algorithms for Qwen3 and GLM-4.5 models.

How much faster is vime than existing solutions?

A benchmark comparison on the Qwen3-30B-A3B model (MoE architecture, mixture-of-experts) shows a stark difference between NVIDIA GB200 and H200 hardware, measured as mean step time in the full RL pipeline:

• GB200: ~147 seconds per step
• H200: ~252 seconds per step
• Difference: GB200 is 1.72× faster end-to-end

It is important to note that these results measure the entire pipeline — combining Megatron training and vLLM rollout — not an isolated measurement of inference or training alone. Such holistic benchmarks are relevant for practitioners planning RL post-training on specific hardware.

R3 routing replay: addressing training-inference distribution drift

One of the key technical challenges in RL post-training is distribution drift between the rollout model that generates samples and the model being trained on those samples. As training progresses, the model changes, but rollout data may have been generated by an older version — resulting in a gap measured as the log-probability difference for the same tokens.

vime introduces R3 routing replay, which specifically addresses this problem for MoE (Mixture-of-Experts) architectures. Results for Qwen3-30B-A3B MoE on A100 GPUs:

• Without R3 replay: log-probability difference ~0.019
• With R3 replay: log-probability difference ~0.013

For comparison, on Qwen3-4B (a dense model) on A100, the log-probability difference remains stable at ~0.011 throughout training, while a baseline comparison without drift management reaches ~0.77 — which in practice would seriously undermine the stability of the RL process.

Results with GLM-4.5-Air on GB200

An end-to-end pipeline example is also shown for the GLM-4.5-Air model trained with the GRPO algorithm on GB200 hardware. Key indicators across 100 training steps:

• Mean raw reward: ~0.56
• Training-rollout log-probability difference: stays in the 0.02–0.03 range (mean ~0.028)

The stability of the distribution gap shows that vime successfully keeps the training and rollout models in sufficient alignment for reliable RL learning, even on the latest GB200 hardware, which has different throughput characteristics from A100 and H200.

Benefits for the research community

vime was designed from day one as a simple, stable, and efficient framework — as stated in the official announcement title. Specifically:

• Supports GRPO and PPO algorithms with validated end-to-end examples for Qwen3 and GLM-4.5
• Runs on multiple hardware types: A100, H200, and GB200
• Specially optimized for MoE models thanks to the R3 replay mechanism
• Built on the vLLM ecosystem, which is already widely adopted in the community

For researchers and engineers who want to implement RL post-training without writing infrastructure code from scratch, vime offers a reference architecture with verified performance on specific hardware and clear numbers that facilitate cost and capacity planning.