vLLM: Day-0 support for TML Inkling — trillion-parameter model up to 380 tokens/s on GB200
vLLM provides Day-0 support for TML Inkling, a multimodal model with 1 trillion parameters. On four NVIDIA GB200 GPUs the system achieves up to 380 tokens/s per user with MTP and long-context serving.
This article was generated using artificial intelligence from primary sources.
What is vLLM and what is TML Inkling?
vLLM (open-source engine for serving large language models) has announced Day-0 support for TML Inkling — meaning support for the new model was available on the day of its release, without waiting for a subsequent system update. TML Inkling is a multimodal model with 1 trillion parameters (10¹²), placing it in the heaviest class of publicly available models. For comparison, GPT-4 is estimated at around 1.8 trillion parameters, while most open-source models today have 7 to 70 billion.
What does the performance look like?
On four NVIDIA GB200 GPUs, vLLM with TML Inkling achieves up to 380 tokens per second per user — which is exceptional for a model of this size. The system uses MTP (multi-token prediction — predicting multiple tokens at once instead of one by one), which further accelerates text and image generation. The implementation also supports long-context serving and various forms of parallelism (tensor, pipeline), enabling scaling across multiple GPUs.
Why is this significant for AI infrastructure development?
The integration requires no modifications to the vLLM core — TML Inkling works directly with the existing system. This means organizations already using vLLM can immediately start serving a trillion-parameter multimodal model without additional stack adaptations. Compared to the previous state, where models of this size required specialized inference frameworks or significant modifications, this advance significantly lowers the barrier to production deployment of frontier models.
Source: vLLM Blog
Frequently Asked Questions
- What does 'Day-0 support' mean for an AI model?
- Day-0 support means that support for a new model is available on the day of the model's release — without waiting for a subsequent update to the inference engine.
- What is vLLM?
- vLLM is an open-source engine for serving large language models, optimized for high throughput and low latency in production environments.
📬 AI news in your inbox
A daily digest built your way — pick topics, sources and cadence. One-click unsubscribe.