DiffusionGemma in vLLM: the first 26-billion-parameter discrete diffusion model integrated into a production inference framework

The vLLM project, Google DeepMind, and NVIDIA teams today announced the integration of DiffusionGemma into vLLM — giving this production inference framework its first natively supported discrete diffusion language model (dLLM). The model has 26 billion parameters and is built on the Gemma4 architecture.

Why is diffusion-based text generation different?

Classical large language models (LLMs) generate text autoregressively — token by token, from left to right. DiffusionGemma works entirely differently: it starts from a canvas of 256 noise-filled tokens and iteratively refines them through multiple steps until coherent text emerges. This approach opens up possibilities that autoregression structurally cannot permit, such as revising earlier parts of a response during later denoising steps.

The architecture has a dual-mode design. In encoder mode, the model uses causal (unidirectional) attention to process the input prompt — identical to classical transformers. In decoder mode, used during iterative denoising, it switches to bidirectional attention with sliding windows (±W tokens), allowing the model to look in both directions for each token on the canvas and coordinate the whole.

How the model decides when tokens are finalized

DiffusionGemma uses entropy-based sampling. At each denoising step, the model computes a confidence score for every token and accepts them in order from highest to lowest confidence — until the accumulated entropy exceeds a predefined threshold. In this way, the most certain tokens are “frozen” early, while less certain ones receive more attention iterations.

Between iterations the model also receives a self-conditioning signal: the full softmax distribution from the previous step is converted into a probability-weighted embedding average and fed back to the model as additional input. This stabilizes convergence and reduces oscillation in later steps.

Integration into the vLLM infrastructure

Rather than writing entirely new execution code, the team adapted the existing speculative decoding infrastructure within vLLM. Two key additions:

• ModelState abstraction — generalizes model state for non-autoregressive architectures; the concrete DiffusionGemmaModelState implementation tracks diffusion loop state per request.
• DiffusionSampler — a sampling handler that distinguishes three phases: prefill (loading the input), denoise (iterative refinement), and commit (token finalization).

Per-sequence dynamic causal attention support enables mixed batches — requests in encoder and decoder mode can coexist in the same batch without conflicts.

Measured performance

The FP8-quantized version of DiffusionGemma achieves 1,288 tokens per second on an NVIDIA H200 GPU at batch size 1, roughly six times faster than autoregressive baseline models. On the H100, 1,008 tokens per second were measured — about five times faster than autoregression and 2.6× faster than certain baseline model variants.

Two quantized checkpoints are available:

• FP8 with dynamic activations
• NVFP4 with quantized weights and activations

Both are published on the RedHatAI Hub and are immediately available for use within vLLM.

What this means for the ecosystem

The integration of DiffusionGemma demonstrates that vLLM is not limited to classical autoregressive models. By generalizing the speculative decoding infrastructure, the project opens the door to future diffusion models without the need to write separate inference stacks. 11 contributors from Google DeepMind, the vLLM team, and NVIDIA worked on the project — indicating serious long-term support for this class of models.