How many simultaneous users can one GPU node serve before it runs out of memory? The weights are fixed once you pick a model. The cache grows with every request. The next sections compute both for three models you can download from Hugging Face, then start them with vLLM, an open-source program that serves models.
How much VRAM the weights of Qwen 2.5 72B, Gemma 3 27B and Phi-4 need
VRAM is the memory on the GPU. For weights in BF16 (a 16-bit number format), each parameter takes two bytes, so the file size on Hugging Face is the number to plan with.
| Model | Parameters | Weights in BF16 | Context length | Licence |
|---|---|---|---|---|
| Qwen 2.5 72B Instruct | 72.7 billion | 145.4 GB | 32,768 (131,072 with YaRN) | Qwen licence |
| Gemma 3 27B | 27.4 billion | 54.9 GB | 131,072 | Gemma terms, accepted on Hugging Face |
| Phi-4 | 14.7 billion | 29.3 GB | 16,384 | MIT |
The context length is the most tokens one request can hold, prompt plus answer. YaRN is a method that stretches the range of positions Qwen can tell apart. The model card warns that it can hurt shorter texts, so enable it only when you need the length.
The KV cache decides how many users fit
Each token a request holds leaves a key and a value in every layer of the model. Together they form the KV cache (key-value cache). Each layer has query heads, which read the cache, and KV heads, which store it, so only the KV heads add memory. Head size is the number of values in one head.
cache per token = 2 (key and value) x layers x KV heads x head size x 2 bytes (16-bit)
Qwen 2.5 72B = 2 x 80 x 8 x 128 x 2 = 327,680 bytesThe same formula gives the other two models. We read these `config.json` files on 20 October 2026: Qwen 2.5 72B, Gemma 3 27B (an Unsloth copy of Google's file) and Phi-4.
| Model | Layers | KV heads | Head size | Cache per token |
|---|---|---|---|---|
| Qwen 2.5 72B | 80 | 8 | 128 | 327,680 bytes (0.33 MB) |
| Gemma 3 27B | 62 | 16 | 128 | 507,904 bytes (0.51 MB) |
| Phi-4 | 40 | 10 | 128 | 204,800 bytes (0.20 MB) |
One example: ten requests of 16,000 tokens
The example is hypothetical: ten requests at once, each holding 16,000 tokens, which fits all three context limits. For Qwen 2.5 72B the cache is 10 x 16,000 x 327,680 bytes = 52.4 GB, on top of 145.4 GB of weights. Double the prompt length and the cache doubles: at 32,000 tokens it is 104.9 GB.
| Model | Weights | Cache for 10 x 16,000 tokens | Total |
|---|---|---|---|
| Qwen 2.5 72B | 145.4 GB | 52.4 GB | 197.8 GB |
| Gemma 3 27B | 54.9 GB | 81.3 GB | 136.1 GB |
| Phi-4 | 29.3 GB | 32.8 GB | 62.1 GB |
The table counts the cache of all layers at the full 16,000 tokens, with BF16 weights. It assumes ten requests running at once and says nothing about speed or answer quality. Activations (temporary values the GPU computes while answering) are not counted, so keep a few GB spare.
Which GPU profile fits each model
Our GPU virtual machines come as NVIDIA L40S (48 GB), A100 and H100 (80 GB each), H200 (141 GB), B200 (192 GB) and B300 (288 GB), with 1, 2, 4 or 8 GPUs each. As of 20 October 2026, vLLM uses 92% of each GPU's memory by default. Usable memory is GPUs x GB x 0.92, and spare is usable memory minus the Total column of the example table above.
| Model | Setup | Usable memory | Spare |
|---|---|---|---|
| Qwen 2.5 72B | 2 x H100 or A100 | 147 GB | does not fit (197.8 GB needed) |
| Qwen 2.5 72B | 2 x H200 | 259 GB | 62 GB |
| Qwen 2.5 72B | 4 x H100 or A100 | 294 GB | 97 GB |
| Qwen 2.5 72B | 1 x B300 | 265 GB | 67 GB |
| Gemma 3 27B | 1 x H200 | 130 GB | does not fit (136.1 GB needed) |
| Gemma 3 27B | 1 x B200 | 177 GB | 40 GB |
| Phi-4 | 1 x H100 or A100 | 74 GB | 12 GB |
| Phi-4 | 2 x L40S | 88 GB | 26 GB |
When a model needs several GPUs, tensor parallelism splits every layer's weights across them, and `--tensor-parallel-size` sets how many.
Quantization shrinks the weights, not the cache
Quantization stores each weight in fewer bits. The table counts parameters times bytes only, so check the real file size of the format you download.
| Model | BF16 (2 bytes) | 8-bit (1 byte) | 4-bit (0.5 byte) |
|---|---|---|---|
| Qwen 2.5 72B | 145.4 GB | 72.7 GB | 36.4 GB |
| Gemma 3 27B | 54.9 GB | 27.4 GB | 13.7 GB |
| Phi-4 | 29.3 GB | 14.7 GB | 7.3 GB |
Weight quantization leaves the cache in the example unchanged. vLLM can also store the cache itself in an 8-bit floating-point format with `--kv-cache-dtype fp8`, which halves the bytes per token. Test answer quality on your own prompts before you rely on it.
vLLM settings that matter for all three
vLLM stores the cache in small blocks that it hands out as requests grow. Its authors report near-zero waste in cache memory, because blocks are handed out only as a request grows. Four flags set the rest of the memory split:
| Flag | What it controls | Value for the example |
|---|---|---|
| `--tensor-parallel-size` | GPUs that share each layer, default 1 | 2 for Qwen, 1 for Gemma 3 and Phi-4 |
| `--max-model-len` | Longest prompt plus answer per request, taken from the model config unless you set it; vLLM must fit one request of this length | 16384 |
| `--gpu-memory-utilization` | Share of each GPU vLLM may use, default 0.92 | 0.92 |
| `--max-num-seqs` | Requests that run together at one time; more requests at once need more cache | 10 |
The flags come from the vLLM engine arguments page. Start Qwen 2.5 72B on two H200 GPUs like this:
vllm serve Qwen/Qwen2.5-72B-Instruct \
--tensor-parallel-size 2 \
--max-model-len 16384 \
--gpu-memory-utilization 0.92 \
--max-num-seqs 10Deploy on a dedicated endpoint or a GPU VM
Lyceum is a sovereign European AI inference provider for open-source models. Our GPUs run in European data centres, and dedicated and VM locations are agreed per contract.
A GPU VM gives you root access over SSH, so you install vLLM and run the command above. It bills per second while it runs. A dedicated endpoint takes the Hugging Face model ID and a hardware profile, and exposes an OpenAI-compatible endpoint. Phi-4 fits one GPU:
lyceum infer deploy microsoft/phi-4 -g gpu.h100 --min-replicas 1 --max-replicas 2 --waitThe create-deployment reference lists every setting, including the vLLM flags above.
If you would rather not run GPUs, our serverless inference also serves other models, such as Qwen3.8 27B, per token through one OpenAI-compatible API.
Choosing between the three models
Pick the smallest model that passes your own tests, because memory grows with size and with every user.
- Phi-4 has the lowest memory need, a 16,384-token limit and an MIT licence. It suits prompts up to 16,384 tokens on one GPU.
- Gemma 3 27B supports 131,072 tokens, but its cache per token is the largest of the three. You accept Google's terms on Hugging Face first.
- Qwen 2.5 72B is the largest and needs more than one 80 GB GPU for its weights alone. Read its licence before commercial use.
Check your own numbers with the model sizing tool on our compute page, which shows the memory a model needs and the GPU it fits on, then launch a VM or a dedicated endpoint.