vLLM Semantic Router v0.3 'Themis': Production-Grade Stateful Query Routing

The vLLM team released version v0.3 “Themis” of its Semantic Router on June 5, 2026, presenting the first production-ready version of the tool for routing queries between models. The release is aimed at teams that, in real enterprise environments, need reliable and predictable management of traffic to language models.

What is the Semantic Router and what is it for?

The Semantic Router is a component that decides where to send each incoming query. Instead of sending all queries to one and the same model, the router analyzes their meaning and content and routes them to the model most suitable for that type of query. This achieves a better ratio of response quality to processing cost.

Until now, this kind of approach was often reserved for experimental setups. With the Themis release, the vLLM team signals that the technology has matured to a level at which it can be reliably used in production.

What makes v0.3 “Themis” production-ready?

The key to production readiness lies in several innovations. Themis brings canonical configuration, meaning a clearly defined and standardized way of setting up the router. Along with it comes an inspectable flow that traces the path from signal, through decision, to the applied policy (routing rules).

This transparency allows operations teams to understand why a particular query was routed to a particular model. It is a prerequisite for reliably maintaining the system in production, where incorrect routing can affect response quality and costs.

What does stateful routing bring compared to stateless?

One of the most prominent changes is the move to stateful routing. In the stateless approach, each routing decision is made in isolation, regardless of context. The stateful approach, in contrast, takes state into account when making decisions, achieving more consistent behavior.

In addition, Themis brings reproducible routing behavior for Kubernetes deployments. This means the router will make the same decisions under the same conditions, which is important for testing, debugging, and reviewing the system’s operation.

What benefits does it bring to enterprise inference stacks?

Themis is explicitly aimed at enterprise inference stacks in which deterministic and auditable traffic routing is needed. The release’s emphasis is on safer operations, which includes alignment of the CLI and dashboard so that behavior through the command line and through the graphical interface matches.

For organizations that process large volumes of queries to multiple models, this Semantic Router release offers a tool with which they can transparently manage traffic with the ability to subsequently verify every decision made. This turns the router’s operation from a black box into a process that can be tracked and verified.

Why is production readiness a turning point?

The transition from an experimental to a production-ready tool is important because only then can the technology reliably carry real traffic. In query routing, a wrong or unpredictable decision can mean a more expensive response, poorer quality, or hampered debugging.

Themis addresses these requirements with a combination of canonical configuration, reproducibility, and an inspectable flow from signal to decision. For teams building enterprise inference stacks, this means they can introduce smart query routing without losing control over the system. Since it is an open-source project from the vLLM team, organizations can adapt the router to their own needs and audit its behavior, which in a business environment is often a prerequisite for adopting a new technology.