How many simultaneous users can one GPU node serve before it runs out of memory? The weights are fixed once you pick a model. The cache grows with every request. The next sections compute both for three models you can download from Hugging Face, then start them with vLLM, an open-source program that serves models.

How much VRAM the weights of Qwen 2.5 72B, Gemma 3 27B and Phi-4 need

VRAM is the memory on the GPU. For weights in BF16 (a 16-bit number format), each parameter takes two bytes, so the file size on Hugging Face is the number to plan with.

ModelParametersWeights in BF16Context lengthLicence
Qwen 2.5 72B Instruct72.7 billion145.4 GB32,768 (131,072 with YaRN)Qwen licence
Gemma 3 27B27.4 billion54.9 GB131,072Gemma terms, accepted on Hugging Face
Phi-414.7 billion29.3 GB16,384MIT

The context length is the most tokens one request can hold, prompt plus answer. YaRN is a method that stretches the range of positions Qwen can tell apart. The model card warns that it can hurt shorter texts, so enable it only when you need the length.

The KV cache decides how many users fit

Each token a request holds leaves a key and a value in every layer of the model. Together they form the KV cache (key-value cache). Each layer has query heads, which read the cache, and KV heads, which store it, so only the KV heads add memory. Head size is the number of values in one head.

cache per token = 2 (key and value) x layers x KV heads x head size x 2 bytes (16-bit)
Qwen 2.5 72B    = 2 x 80 x 8 x 128 x 2 = 327,680 bytes

The same formula gives the other two models. We read these `config.json` files on 20 October 2026: Qwen 2.5 72B, Gemma 3 27B (an Unsloth copy of Google's file) and Phi-4.

ModelLayersKV headsHead sizeCache per token
Qwen 2.5 72B808128327,680 bytes (0.33 MB)
Gemma 3 27B6216128507,904 bytes (0.51 MB)
Phi-44010128204,800 bytes (0.20 MB)

One example: ten requests of 16,000 tokens

The example is hypothetical: ten requests at once, each holding 16,000 tokens, which fits all three context limits. For Qwen 2.5 72B the cache is 10 x 16,000 x 327,680 bytes = 52.4 GB, on top of 145.4 GB of weights. Double the prompt length and the cache doubles: at 32,000 tokens it is 104.9 GB.

Four boxes joined by arrows show the worked example: one token takes 327,680 bytes of cache, ten requests of 16,000 tokens hold 160,000 tokens, the cache is 52.4 GB, and with 145.4 GB of weights the total is 197.8 GB, which fits on two H200 GPUs with 62 GB spare. Calculated example.
For ten requests of 16,000 tokens, the cache of Qwen 2.5 72B is 52.4 GB and the total with weights is 197.8 GB (calculated example, BF16 weights, 16-bit cache).
ModelWeightsCache for 10 x 16,000 tokensTotal
Qwen 2.5 72B145.4 GB52.4 GB197.8 GB
Gemma 3 27B54.9 GB81.3 GB136.1 GB
Phi-429.3 GB32.8 GB62.1 GB

The table counts the cache of all layers at the full 16,000 tokens, with BF16 weights. It assumes ten requests running at once and says nothing about speed or answer quality. Activations (temporary values the GPU computes while answering) are not counted, so keep a few GB spare.

Stacked horizontal bars of weights and KV cache in GB for ten requests of 16,000 tokens each. Qwen 2.5 72B: 145.4 GB weights plus 52.4 GB cache, 197.8 GB. Gemma 3 27B: 54.9 plus 81.3, 136.1 GB. Phi-4: 29.3 plus 32.8, 62.1 GB. A dashed line marks the 130 GB usable on one H200; only Phi-4 stays below it. Calculated example.
In the same example the KV cache is 26% of Qwen 2.5 72B's memory, 60% of Gemma 3 27B's and 53% of Phi-4's, and only Phi-4 stays under the 130 GB one H200 offers (calculated).

Which GPU profile fits each model

Our GPU virtual machines come as NVIDIA L40S (48 GB), A100 and H100 (80 GB each), H200 (141 GB), B200 (192 GB) and B300 (288 GB), with 1, 2, 4 or 8 GPUs each. As of 20 October 2026, vLLM uses 92% of each GPU's memory by default. Usable memory is GPUs x GB x 0.92, and spare is usable memory minus the Total column of the example table above.

ModelSetupUsable memorySpare
Qwen 2.5 72B2 x H100 or A100147 GBdoes not fit (197.8 GB needed)
Qwen 2.5 72B2 x H200259 GB62 GB
Qwen 2.5 72B4 x H100 or A100294 GB97 GB
Qwen 2.5 72B1 x B300265 GB67 GB
Gemma 3 27B1 x H200130 GBdoes not fit (136.1 GB needed)
Gemma 3 27B1 x B200177 GB40 GB
Phi-41 x H100 or A10074 GB12 GB
Phi-42 x L40S88 GB26 GB

When a model needs several GPUs, tensor parallelism splits every layer's weights across them, and `--tensor-parallel-size` sets how many.

Quantization shrinks the weights, not the cache

Quantization stores each weight in fewer bits. The table counts parameters times bytes only, so check the real file size of the format you download.

ModelBF16 (2 bytes)8-bit (1 byte)4-bit (0.5 byte)
Qwen 2.5 72B145.4 GB72.7 GB36.4 GB
Gemma 3 27B54.9 GB27.4 GB13.7 GB
Phi-429.3 GB14.7 GB7.3 GB

Weight quantization leaves the cache in the example unchanged. vLLM can also store the cache itself in an 8-bit floating-point format with `--kv-cache-dtype fp8`, which halves the bytes per token. Test answer quality on your own prompts before you rely on it.

vLLM settings that matter for all three

vLLM stores the cache in small blocks that it hands out as requests grow. Its authors report near-zero waste in cache memory, because blocks are handed out only as a request grows. Four flags set the rest of the memory split:

FlagWhat it controlsValue for the example
`--tensor-parallel-size`GPUs that share each layer, default 12 for Qwen, 1 for Gemma 3 and Phi-4
`--max-model-len`Longest prompt plus answer per request, taken from the model config unless you set it; vLLM must fit one request of this length16384
`--gpu-memory-utilization`Share of each GPU vLLM may use, default 0.920.92
`--max-num-seqs`Requests that run together at one time; more requests at once need more cache10

The flags come from the vLLM engine arguments page. Start Qwen 2.5 72B on two H200 GPUs like this:

vllm serve Qwen/Qwen2.5-72B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 10

Deploy on a dedicated endpoint or a GPU VM

Lyceum is a sovereign European AI inference provider for open-source models. Our GPUs run in European data centres, and dedicated and VM locations are agreed per contract.

A GPU VM gives you root access over SSH, so you install vLLM and run the command above. It bills per second while it runs. A dedicated endpoint takes the Hugging Face model ID and a hardware profile, and exposes an OpenAI-compatible endpoint. Phi-4 fits one GPU:

lyceum infer deploy microsoft/phi-4 -g gpu.h100 --min-replicas 1 --max-replicas 2 --wait

The create-deployment reference lists every setting, including the vLLM flags above.

If you would rather not run GPUs, our serverless inference also serves other models, such as Qwen3.8 27B, per token through one OpenAI-compatible API.

Choosing between the three models

Pick the smallest model that passes your own tests, because memory grows with size and with every user.

  • Phi-4 has the lowest memory need, a 16,384-token limit and an MIT licence. It suits prompts up to 16,384 tokens on one GPU.
  • Gemma 3 27B supports 131,072 tokens, but its cache per token is the largest of the three. You accept Google's terms on Hugging Face first.
  • Qwen 2.5 72B is the largest and needs more than one 80 GB GPU for its weights alone. Read its licence before commercial use.

Check your own numbers with the model sizing tool on our compute page, which shows the memory a model needs and the GPU it fits on, then launch a VM or a dedicated endpoint.

Sources

  1. huggingface.co
  2. huggingface.co
  3. huggingface.co
  4. huggingface.co
  5. huggingface.co
  6. huggingface.co
  7. docs.vllm.ai
  8. arxiv.org
  9. arxiv.org
  10. docs.vllm.ai
  11. docs.vllm.ai