Why engine benchmarks disagree
None of the three engines wins every benchmark, because the result moves with the model, the GPU, the engine version and the traffic. Pick vLLM when you swap models often, SGLang when requests repeat the same opening text, and TensorRT-LLM when one model runs unchanged on NVIDIA GPUs.
The number that decides most is the share of each prompt that repeats. The engine does not have to read repeated text again, and on our API it is billed at a lower rate.
Which engine should you run in production, and do you need to run one at all?
We are Lyceum, a sovereign European AI inference provider for open-source models. Sovereign means built and run in Europe: our serverless models run in Paris and Finland. Prompts and outputs are processed, not stored, and never used for training. Open-weight models are models whose trained weights anyone can download and run. Since January 2026, the strongest of them have trailed the best closed models by an average of four months, according to Epoch AI. So a better open model arrives every few months, and your engine has to run each new one.
Published numbers measure different things. The SGLang paper reports up to 6.4 times higher throughput (requests answered per second) than the systems it compared. It measured against vLLM 0.2.5, mostly on smaller 24 GB NVIDIA A10G GPUs. Nine months later, vLLM 0.6.0 reported 2.7 times the throughput of its 0.5.3 release, on one H100, NVIDIA's larger data-centre GPU. So the 6.4 times says little about today's vLLM.
A speed-up quoted without its model, GPU, version and traffic is not a number you can plan against.
What vLLM, SGLang and TensorRT-LLM are each built for
A model stores a record of every token it has read, called the KV cache (key-value cache). How an engine stores and reuses that record shapes its speed.
vLLM keeps the KV cache in fixed-size blocks, like pages of memory in an operating system. Its authors call this method PagedAttention: memory is taken in small pages as the answer grows. Nothing is reserved for the longest possible answer, so almost no memory sits unused. The vLLM engine serves a model without a separate build step, which suits teams that change models often.
SGLang stores cached text in a tree and matches each new request against the longest shared start, a technique it calls RadixAttention. It gains most when many requests open with the same text. Examples are agent loops, where a model calls tools and reads their results turn after turn, and long chats. The vLLM engine reuses a shared start too: its automatic prefix caching keeps a prompt's start in memory. SGLang is built around that reuse.
TensorRT-LLM is NVIDIA's library for tuning a model to NVIDIA GPUs. For years it needed an engine build: a step that compiles the model into a file for one GPU type before it serves a token. The build fixed in advance how many requests run at once (the batch size) and how long prompts may be. NVIDIA is removing it. Run the stable release, 1.2.1, today: it still has the engine build. The 1.3 line drops it after release candidate (test version) 1.3.0rc20 of 30 June 2026.
| Engine | Built for | Watch out for |
|---|---|---|
| vLLM | Many models, frequent model changes | No tuning step per GPU saves effort; a test shows if a tuned TensorRT-LLM is faster |
| SGLang | Requests that share a long opening text | Little gain when prompts are short and answers long |
| TensorRT-LLM | One model, unchanged, on NVIDIA GPUs | Tuning per model, GPU and number format (such as FP16 or FP8, the bits per weight); engine build removed in the 1.3 line |
A repeated prompt start changes the cost more than the engine
Every request has two phases. Prefill is the phase when the model reads your prompt and builds its KV cache. Decode is the phase when it writes the answer, one token at a time.
Prefix caching means the engine keeps the KV cache of a prompt's start. When the next request starts with exactly the same text, the engine skips prefill for that part. The vLLM documentation states the limit plainly: prefix caching shortens prefill and does not speed up decode.
Take a hypothetical coding-agent turn. The agent sends 20,000 input tokens. 18,000 of them repeat the previous turn's start: the system prompt, the tool definitions and the history. Only 2,000 are new, and the model writes 500 tokens.
On our serverless inference, GLM-5.3 costs $1.40 per million input tokens and $4.40 per million output tokens. Cached input costs $0.26 per million tokens (prices read 30 October 2026).
cost per turn = new input x input price
+ cached input x cached input price
+ output x output price
(prices in USD per million tokens)| Case (hypothetical turn) | Input | Cached input | Output | Total |
|---|---|---|---|---|
| Nothing reused | 20,000 x $1.40 = $0.0280 | $0 | 500 x $4.40 = $0.0022 | $0.0302 |
| Start reused | 2,000 x $1.40 = $0.0028 | 18,000 x $0.26 = $0.00468 | $0.0022 | $0.00968 |
The reused turn costs about a third. Over 1,000 turns, that is $9.68 instead of $30.20. The engine also reads 2,000 new tokens instead of 20,000, so the first token arrives sooner.
The example favours reuse, because it assumes a stable prompt start and a quick next turn. It does not show a measured hit rate. A cache hit (finding the start still in memory) is not guaranteed. We keep recent prompt starts in GPU memory only until newer requests replace them.
NVIDIA Dynamo runs above the engines
NVIDIA Dynamo is often compared with the three engines, but it is not a fourth one. It coordinates engines across many GPUs and servers. NVIDIA's documentation says Dynamo works with vLLM, SGLang and TensorRT-LLM. It runs on a cluster or on a single machine.
It can route a request to the GPU that already holds its KV cache, so the shared start is not read twice. Our NVIDIA Dynamo guide goes further.
We use that routing ourselves. On 5 October 2026, our CTO announced a new serving engine for GLM-5.3, built with our own changes on top of NVIDIA Dynamo. Each turn of a session goes back to the GPU that holds its KV cache. That cut the GLM-5.3 cached input price from $0.44 to $0.26 per million tokens, and input from $1.75 to $1.40.
An open engine does not mean you have to run it
vLLM, SGLang, TensorRT-LLM and Dynamo are open source. You can read the code, pin a version and profile a slow request. A closed engine shows you only what its provider exposes.
Open code also makes a move to another provider possible. It stays a migration, not a switch: weights, drivers, networking and storage still need testing on the new hardware.
Running an engine yourself is the costly part. Someone has to update drivers, tune each model and keep spare GPUs for traffic peaks. Without a platform team, that lands on product engineers.
You have three ways to use open models with us, from least to most work:
- Serverless inference. Call open models through one OpenAI-compatible API, billed per token with no base fee. We choose and run the engine.
- Dedicated endpoints. GPUs reserved for your model, with the model configuration agreed before launch. Our endpoint guide explains when steady traffic pays for one.
- GPU virtual machines. Root access, billed per second, for teams that run their own engine. See our compute options.
How to choose: test on your own traffic
Pick on two questions. How often do you change models? And how much repeated text does a typical request carry? Frequent model changes point to vLLM. A high share of repeated text makes SGLang one strong option, though vLLM reuses shared starts too. One frozen model on NVIDIA GPUs points to TensorRT-LLM.
Then test:
- Record a sample of real production prompts, so the share of repeated text is yours.
- Replay them against each candidate on the same GPUs and settings.
- Raise the number of parallel requests from one to your real peak.
- Log time to first token (TTFT), the wait before the first word, and tokens per second.
- Count the work around it: start-up time, driver updates and recovery after a failure.
If your share of repeated text is high and you have no platform team, start on our API. If the numbers later justify your own stack, move to a GPU virtual machine with the same prompts.
Frequently asked questions
Does SGLang generate tokens faster than vLLM?
Not in general. SGLang saves time when requests share a long opening text. Writing the answer takes about as long, so short prompts with long answers gain little.
Do I still need to build a TensorRT engine for TensorRT-LLM?
On the stable 1.2.1 release, yes. The 1.3 line drops the engine build.
Can I run my own engine on your GPUs?
Yes. Our GPU virtual machines give you root access and are billed per second, so you can install the engine and version you choose. Our serverless API runs the engine for you.