Helion in vLLM: PyTorch-native DSL replaces CUDA kernels and accelerates FP8 inference for Qwen3 models

The PyTorch team published today a blog post in collaboration with Red Hat describing the integration of nine Helion kernels into vLLM for FP8 inference of Qwen3 models. The authors are Sean Chen from Red Hat and Yanan Cao from Meta Platforms (PyTorch team).

What is Helion and why should developers care?

Helion is a domain-specific language (DSL) described as “PyTorch with tiles” — it enables writing high-performance GPU kernels using PyTorch syntax extended with tile-programming concepts, without writing direct CUDA code. The goal is hardware portability: the same kernel should work on NVIDIA H100, B200, and future architectures without manual porting.

Helion provides advanced infrastructure for ahead-of-time (AOT) autotuning — automatically finding optimal tile sizes and memory layouts for each target GPU architecture and input dimension. Kernel development is fast: the authors report that most of the nine kernels were implemented and validated within a single working day.

Which kernels have been integrated into vLLM

The nine kernels cover nearly the entire forward pass of FP8 inference:

1. dynamic_per_token_scaled_fp8_quant
2. rms_norm_dynamic_per_token_quant
3. silu_and_mul_dynamic_per_token_quant — new fusion kernel
4. fused_qk_norm_rope
5. per_token_group_fp8_quant
6. rms_norm_per_block_quant
7. silu_and_mul_per_block_quant
8. scaled_mm
9. scaled_mm_blockwise

The integration reduces the number of kernel calls in the forward pass from 11+ to 10 fused operations for per-token quantization.

Performance at the individual kernel level

Non-GEMM kernels show consistent and significant speedups on the H100 compared to previous implementations (torch.ops._C):

• rms_norm_per_block_quant: +67% (2.055× vs torch.ops._C)
• silu_and_mul_per_block_quant: +73% (2.269× vs torch.ops._C)
• rms_norm_dynamic_per_token_quant: 1.80× vs torch.ops._C
• fused_qk_norm_rope: 1.38× vs torch.compile

GEMM kernels (scaled_mm) show mixed results: 1.08× on the H100, but 0.739× on the B200. The authors openly acknowledge that the cause is a Triton code-generation limitation for the Blackwell architecture, and that a CuteDSL backend is under development to resolve this issue.

Speedup at the full model level

Testing was conducted on Qwen3 models (1.7B, 8B, and 32B parameters) on the H100 and B200:

• Per-token quantization (H100): ~1.05× overall throughput for Qwen3-1.7B
• Speculative decoding, Qwen3-8B, H100: up to 1.09× total throughput (1.15× TTFT speedup at batch size 16)
• Per-group quantization (H100 and B200): ~1.05× across all workloads

For Qwen3-8B with speculative decoding and 3 speculative tokens, a 1.33× TTFT speedup (time to first token) was measured.

Practical limitations

The authors are transparent about the trade-offs. Autotuning large kernels such as scaled_mm requires roughly one day to cover all input shapes (168 distinct dimensions for the three Qwen3 variants). Helion’s runtime dispatch introduces a few tens of microseconds of latency per kernel call, so CUDA graph capture is mandatory for achieving optimal performance.

Helion per-group quantization kernels are not yet compatible with the UE8M0 format required by DeepGEMM on the B200, which is another open item.

Looking ahead

The project was published as GitHub issue vllm-project/vllm#32962 with implementations of all nine kernels. The goal is to expand Helion support to additional architectures and resolve GEMM performance on Blackwell through the CuteDSL backend. The collaboration between Red Hat and the Meta PyTorch team on this problem signals that the PyTorch ecosystem is seriously addressing the challenge of hardware-portable high-performance inference kernels.