vLLM and Tencent Hunyuan upstream two HPC backends for NVIDIA Hopper GPUs

The Tencent Hunyuan AI Infra team and the vLLM team have jointly upstreamed two high-performance backends into the mainstream vLLM project. These are the HPC_ATTN attention backend and the hpc MoE backend, available as first-class components without needing to fork or modify the vLLM core. This contribution marks one of the rare instances in which an industrial team upstreams production-grade kernel optimizations directly into the open project they rely on, rather than holding them as an internal advantage.

Two new HPC backends in the vLLM ecosystem

The collaboration between Tencent’s Network Platform division, the Hunyuan AI Infra team, and vLLM/Inferact maintainers delivers solutions for two chronic bottlenecks in modern LLM inference: dynamic load balancing of the decode phase and FP8 MoE routing overhead. Both backends are integrated through standard vLLM registration mechanisms — PR #46020 for the attention backend and PR #45924 for the MoE backend — meaning users select the new backend through configuration parameters without any core modifications.

The reference model on which the backends were validated is Hy3, a Tencent Hunyuan model with 295 billion total parameters and 21 billion activated. The model features 192 experts with top-8 routing, 64 query and 8 KV heads, a head dimension of 128, and a 256K-token context window. A speculative decoding layer (MTP) adds another 3.8 billion parameters.

Why dynamic scheduling is critical for attention

Static split-KV schedules assume uniform sequence length within a batch. In a production environment that rarely holds — a mix of short and long requests creates idle waiting on compute units that have finished their work while others are still processing long sequences. The result is lost GPU cycles proportional to the length dispersion within the batch.

The HPC_ATTN backend solves this with a per-step load-balanced decode scheduler that distributes 64-token tiles across compute units proportional to the actual length of each sequence in the current generation step. A fused prologue combines RoPE rotation, QK-Norm normalization, and the KV-write operation into a single kernel pass, eliminating redundant memory reads and writes between those operations.

On a mixed workload of one 128K-token sequence and 31 4K-token sequences, dynamic scheduling achieves a 2.95x speedup over a static equivalent. The average advantage over FlashInfer and FlashAttention implementations is 2.25x on mixed-length decode batches. The backend supports BF16 and FP8 precision.

FP8 MoE pipeline as a single execution path

Standard MoE inference passes through a series of separate kernel calls: routing, Gate-Up GEMM, activation function, quantization, Down GEMM, and top-k reduction. Each transition between kernels means a round-trip to High Bandwidth Memory (HBM), which in decode-dominant workloads — where the batch is small and latency is critical — becomes the dominant overhead that masks the hardware’s actual computational speed.

The hpc backend fuses the entire pipeline into a single execution path using Programmatic Dependent Launch (PDL) chaining. Routing tables are held in shared memory rather than HBM, eliminating costly retrieval of expert indices between phases. The architecture is FP8-only, consistent with the hardware capabilities of the Hopper generation.

Measured results on NVIDIA H20 GPUs

End-to-end measurements were conducted on an 8x H20 configuration with the Hy3 model. Time to First Token (TTFT) was reduced by an average of 24%. Time Per Output Token (TPOT) improved by an average of 17%, and at the largest batch size of 64 the improvement rises to 30%.

The MoE backend achieves 1.59x lower latency than the best baseline (Triton/CUTLASS) at TP8/EP1 configuration, and 1.21x at TP1/EP8. The advantage is most pronounced at small and medium batch sizes, corresponding to typical production decode scenarios in which a small number of concurrent requests must be served with minimal latency.

Integration without forks and Hopper-specificity

Both backends are intended exclusively for GPUs with Hopper architecture — H20 is cited as the optimal platform. Reliance on PDL mechanisms and Hopper-specific instructions means that extension to older architectures is non-trivial and has not been announced.

By upstreaming through the formal PR process, the Tencent Hunyuan team follows a contribution model that gives all vLLM users access to the optimizations without the burden of managing forks against every new upstream release. For operators scaling MoE inference on Hopper clusters, this is a directly applicable improvement available through the standard vLLM upgrade cycle.