PyTorch: TokenSpeed achieves record 580 tokens/s on Qwen3.5-397B-A17B with NVIDIA Blackwell GPUs

LightSeek Foundation published results from the TokenSpeed inference engine that set a new throughput record for the Qwen3.5-397B-A17B model: 580 tokens per second in a configuration with eight NVIDIA Blackwell B200 GPUs (TP8, Tensor Parallelism factor 8).

What is TokenSpeed and how does it differ from vLLM?

TokenSpeed is an open-source LLM inference engine available under the MIT license, developed by LightSeek Foundation. It is designed with a “speed-of-light” performance philosophy — the goal is to achieve performance comparable to NVIDIA TensorRT-LLM, with the developer ergonomics characteristic of vLLM. It is based on a native SPMD (Single Program, Multiple Data) approach and static computation graph compilation.

The Qwen3.5-397B-A17B model on which the record was achieved uses a hybrid attention architecture that combines standard full-attention layers with linear attention layers based on the Gated Delta Network (GDN) mechanism.

What are the key optimizations behind the record?

TokenSpeed achieves high throughput through several technical innovations:

Hybrid prefix caching for GDN/Mamba: Context caching is split between a C++ layer (logical cache) and a Python layer (physical tensors). Radix-tree matching with MambaSlot extensions enables state reuse with copy-on-write semantics. Result: KV cache hit rate exceeding 90% in agentic workloads with multiple conversation turns.

Mamba State Update optimization: Inter-step state copying is eliminated using index indirection instead of tensor scatter — the update operation drops from O(L·D) to O(1) for pointer updates.

Prefill-Decode disaggregation: The context filling and token generation phases are separated with unified state transfer, a three-phase handshake, and overlapping network communication with early execution layers.

CUDA graph recording: The entire decode loop is recorded as a CUDA graph, eliminating the overhead of launching each kernel separately.

How does TokenSpeed handle long contexts?

TokenSpeed shows exceptional throughput stability at long contexts on NVIDIA Blackwell B200 GPUs with NVFP4 quantization:

Context length	Throughput
128K tokens	~530 tok/s
256K tokens	~495 tok/s
1M tokens	~445 tok/s

A drop of only ~16% when moving from 128K to 1M tokens means that TokenSpeed remains practical even for extremely long agentic sessions that are otherwise demanding for inference systems.

Agentic workload results

In a benchmark simulating real agentic workloads (50K tokens in the first turn + 800 tokens per subsequent turn, 10-15 conversation turns), TokenSpeed achieves:

• TP8: ~580 tok/s (single user, peak)
• TP4: 500+ tok/s (single user)
• 16 parallel users (TP4): ~2K tok/min per GPU

TokenSpeed also supports Flash Attention 4 (FA4) with head dim=256 on the Blackwell architecture, currently in active development. The project is available on GitHub and the community is open for contributions.