You start vLLM, the weights load, and the server stops with CUDA out of memory before the first request arrives. Or it runs for hours, then slows down and logs warnings about cache space. Which vLLM setting fixes which out-of-memory error, and when is the GPU too small?
Why does vLLM run out of memory before it serves a single request?
vLLM runs out of memory before the first request because it reserves almost all of its memory at startup. On boot, it measures how much memory the KV cache can use. Then it allocates that memory up front, in three steps:
- Load the model weights onto the GPU.
- Run a test pass that measures the working memory. That covers activations, the numbers each layer passes to the next while it computes. It also covers CUDA graphs: vLLM records the GPU's work for common batch sizes once, so later steps replay it faster.
- Give the rest of its memory share to the KV cache.
So an error at boot is not a leak: the three parts do not fit the share you allowed. An error under load is different: the cache exists, but the requests running at once need more of it than it holds.
What does gpu_memory_utilization actually reserve, and what is left for everything else?
--gpu-memory-utilization sets the share of GPU memory one vLLM instance may use for all three steps, not for the KV cache alone. The default is 0.92. The remaining 8% stays outside vLLM's share, for the CUDA driver and PyTorch, the Python library vLLM runs on.
The running example is Llama 3.1 70B Instruct, Meta's open model with 70.6 billion parameters. It runs on NVIDIA H100 GPUs with 80 GB each.
share at 0.92: 0.92 × 80 GB = 73.6 GB per GPU
weights, 16-bit: 70.554 billion × 2 bytes = 141.11 GB
weights, 8-bit: 70.554 billion × 1 byte = 70.55 GB
FP8 is an 8-bit number format, so each weight takes one byte instead of two. This is a calculated example: we count 80 GB as 80 billion bytes, 1 GB as a billion bytes, and ignore small extras.
How does max-model-len set the size of the KV cache?
--max-model-len sets the longest request vLLM accepts, prompt and answer together. Unset, vLLM takes it from the model config. For Llama 3.1 70B that is 131,072 tokens. The setting does not free any memory. It lowers the minimum the cache must hold: vLLM refuses to start unless one request of that length fits.
The cache per token follows from the model config. For every token, each layer stores two short lists of numbers, a key and a value, that later tokens look back at. A layer splits its attention into 64 heads, but 8 heads share each set of keys and values. So Llama 3.1 70B keeps 8 key-value heads (KV heads) per layer. Each list is 128 numbers long, at 2 bytes per number. Our KV cache memory guide explains the formula.
KV cache per token = 80 layers × 2 (key, value) × 8 KV heads × 128 × 2 bytes
= 327,680 bytes (320 KiB; 1 KiB = 1,024 bytes)
128 = the model's width (hidden size, 8,192 numbers per token)
split across its 64 attention heads (8,192 / 64)
one request of 131,072 tokens = 131,072 × 327,680 bytes = 42.95 GB
(40 GiB; 1 GiB = 1,073,741,824 bytes)
one request of 8,192 tokens = 8,192 × 327,680 bytes = 2.68 GB
Lowering the setting to the length your application really sends helps twice. The engine can start with less cache. Because each request can then grow less, more of them fit at once.
"No available memory for the cache blocks" - what is vLLM telling you?
The engine shows this message when nothing is left for the KV cache after the weights and the test pass. It appears when the remaining memory is zero or less. A second message appears when some memory is left, but not enough for one full-length request:
To serve at least one request with the model's max seq len (131072),
(40.00 GiB KV cache is needed, which is larger than the available KV cache memory (...).
Based on the available memory, the estimated maximum model length is ...
The example on one H100 produces the first message. The 8-bit weights take 70.55 of the 73.6 GB share, which leaves 73.6 − 70.55 = 3.05 GB. Assume the test pass needs 4 GB per GPU (hypothetical; your boot log prints the real figure). The remainder is below zero, so vLLM stops.
vLLM's memory guide lists four ways to make room before the cache:
- Split the model across GPUs with tensor parallelism (
--tensor-parallel-size), so each GPU holds part of the weights (setup guide). - Load a quantized checkpoint, a copy of the model saved with fewer bytes per weight, such as FP8. Our comparison of quantization formats covers the choices.
- Lower
--max-model-lento what your application sends. - Record fewer CUDA graphs.
--enforce-eagerrecords none and runs each step directly, which saves memory but runs slower.
On one H100 only the first two help, because no memory is left for any request length.
Startup OOM or runtime OOM: which knob do you turn first?
Turn the setting that matches when the error appears. At startup, make the reservation fit. Under load, vLLM does not crash at first. It pauses some requests, frees their cache and computes them again later, which vLLM calls preemption. The log then says the request "is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space". RECOMPUTE means the paused request rebuilds its cache from the prompt when it resumes, which costs time.
| When it fails | What vLLM says | First setting | Next |
|---|---|---|---|
| At startup | No available memory for the cache blocks | More GPUs (--tensor-parallel-size) or 8-bit weights | Fewer CUDA graphs |
| At startup | KV cache needed is larger than the available KV cache memory | Lower --max-model-len | More GPUs |
| Under load | Preempted because there is not enough KV cache space | 8-bit KV cache (--kv-cache-dtype fp8) | Fewer requests at once (--max-num-seqs, the cap on requests processed together), more GPUs |
The two-GPU example shows the load case. With tensor parallelism across 2 GPUs, each H100 holds half the weights. It also holds 4 of the 8 KV heads per layer, so it stores only half of each token's cache.
weights per GPU: 70.55 GB / 2 = 35.28 GB
cache per GPU: 73.6 − 35.28 − 4 (test pass) = 34.32 GB
cache per token, per GPU: 327,680 bytes / 2 = 163,840 bytes
tokens per GPU: 34.32 GB / 163,840 bytes ≈ 209,000
Now assume a hypothetical peak of thirty requests of 16,384 tokens each. They need 491,520 tokens, so vLLM preempts. Raising the share to 0.95 adds only 2.4 GB per GPU (about 14,600 tokens), so change it last. An 8-bit KV cache stores each number in one byte instead of two. That doubles the cache to about 419,000 tokens. Check answer quality on your own prompts before you keep it.
When is the error a sizing problem rather than a configuration problem?
The error is a sizing problem when the weights alone fill the share, because no setting shrinks them except quantization. Llama 3.1 70B in 16-bit needs 141.11 GB, almost twice the memory of one H100. In 8-bit it fits on paper but leaves 3.05 GB for everything else.
| Setup (8-bit weights, 4 GB test pass) | Cache per GPU | Upper bound: cache ÷ 16,384-token request |
|---|---|---|
| 1 × H100, 80 GB | none | 0 |
| 1 × H200, 141 GB | 55.2 GB | 10.3 |
| 1 × B200, 192 GB | 102.1 GB | 19.0 |
| 2 × H100, tensor parallel | 34.3 GB | 12.8 |
The last column divides the cache by the request length. It is an upper bound for requests at full length, not a measured concurrency. Two GPUs must exchange partial results on every step, and that traffic needs buffers of its own. The example counts no memory for them, so it favours the two-GPU row and makes that figure optimistic. Our GPU virtual machines come with H100 (80 GB), H200 (141 GB) or B200 (192 GB). Each machine has 1, 2, 4 or 8 GPUs. A sizing tool on our GPU compute page shows the memory a Hugging Face model needs and which GPU fits.
How do you verify the fix without waiting for it to fail in production again?
Read the boot log before you send traffic. After the test pass, vLLM logs the cache size and the concurrency it allows. The line reads like "GPU KV cache size: N tokens, Maximum concurrency for 131,072 tokens per request: K x". The "x" is cache size divided by the maximum request length: how many full-length requests fit at once. In the two-GPU example, expect about 209,000 tokens and 1.6 x, depending on the real test pass.
--kv-cache-memory-bytes sets the cache size per GPU directly, and vLLM then ignores the share setting. Set it to the size your boot log reported, so a library update cannot change the cache silently. Finally, replay your peak traffic with your longest real prompts. If no preemption warning appears at peak, the cache fits.
When does handing the engine to a managed endpoint cost less than tuning it?
A managed endpoint costs less when a model we serve fits your workload and nobody on your team wants to own vLLM settings. Lyceum is a sovereign European AI inference provider for open-source models: built and run in Europe, with our serverless models in Paris and Finland. You call them through one OpenAI-compatible API, billed per token with no base fee. Prompts and outputs are processed, not stored.
export OPENAI_BASE_URL="https://api.lyceum.technology/openai/v1"
export OPENAI_API_KEY="$LYCEUM_API_KEY"
Keep the engine when you need a model we do not serve or a pinned vLLM version. A GPU virtual machine gives you root access in our European data centres, billed per second. Our endpoint guide compares the three routes with break-even traffic.
Frequently asked questions
Should I set gpu_memory_utilization to 1.0?
No. The CUDA driver and PyTorch still need memory outside vLLM's share. Raise it in small steps and watch the boot log.