You run a support bot, a summariser and a classifier, all fine-tuned from the same open model. Each one sits on its own GPU and is busy a few hours a day. How do you serve several models on one GPU without running out of memory? Sharing the base model is what frees the memory: adapters reuse one copy of the model's weights, while separate servers each load their own copy.

The VRAM fragmentation problem in production inference

A model on a GPU uses memory for two things. The weights are the model's learned numbers, and they are fixed in size. The KV cache (key-value cache) is the record the model keeps of every token it has read, so it grows with each request.

Older serving systems reserved cache memory for the longest answer a request might produce. Most of that space stayed empty. The authors of vLLM measured that existing systems used only 20.4% to 38.2% of their cache memory for actual tokens.

vLLM fixes this with PagedAttention. It stores the cache in small fixed-size blocks. A request gets a new block only when it needs one, the way an operating system hands out memory pages. That leaves almost no cache memory unused. The paper compares vLLM with two earlier serving systems, FasterTransformer and Orca. At the same level of latency, vLLM served 2 to 4 times as many requests per second.

PagedAttention makes room inside one model's cache. A second model on the same GPU needs one of two routes, covered next.

Decision flow for putting several models on one GPU with vLLM. Start: do the models share a base model? Yes: run one vLLM server with one adapter per fine-tune, enable LoRA and set max-loras to the number of adapters in use. No: different model families need their own weights, so run separate servers whose memory shares add up to less than the GPU, or split the GPU into MIG slices. Both end boxes are highlighted.
Whether your models share a base model decides the route: adapters on one server, or separate servers with their own memory.

Multi-LoRA serving: one base model, many fine-tunes

LoRA (low-rank adaptation) fine-tunes a model by training two small matrices next to each weight matrix. The original weights stay untouched. The result is an adapter: a file of a few megabytes to a few hundred megabytes. The server loads the base model once and applies the matching adapter to each request. Its docs say adapters "can be efficiently served on a per-request basis with minimal overhead".

The running example is Llama 3.1 8B Instruct, an open model with 8,030,261,248 parameters. Its config lists 32 layers and a hidden size of 4,096 numbers per token. The intermediate size of 14,336 is the inner width of each feed-forward part. An attention head is one of the parallel parts of a layer that compare tokens. The model has 32 of them per layer, and every 4 share one KV head, the part that writes to the cache. Sharing makes the cache 4 times smaller: it stores 8 KV heads per layer. The head size is the width of one head. The GPU is one NVIDIA H100 with 80 GB.

weights (16-bit)       = 8,030,261,248 × 2 bytes           = 16.06 GB
head size              = 4,096 hidden size / 32 heads       = 128
KV cache per token     = 32 layers × 2 (key, value) × 8 KV heads × 128 × 2 bytes
                       = 131,072 bytes (128 KiB)
one 8,192-token request = 8,192 × 131,072 bytes            = 1.07 GB
(1 KiB = 1,024 bytes)

Each layer has 7 weight matrices, called projections. Query, key, value and output belong to attention, where tokens look at each other. Gate, up and down belong to the feed-forward part, which transforms each token on its own. The rank is the inner width of the adapter. A rank-16 adapter adds a matrix A (input width × 16) and a matrix B (16 × output width) to each. So each matrix gets 16 × (input + output) new parameters. The key and value matrices are 1,024 wide (8 KV heads × 128), and 14,336 is the config's intermediate size.

input + output widths per layer:
  query, output projections   2 × (4,096 + 4,096)  = 16,384
  key, value projections      2 × (4,096 + 1,024)  = 10,240
  gate, up projections        2 × (4,096 + 14,336) = 36,864
  down projection             14,336 + 4,096       = 18,432
                                             total = 81,920
adapter parameters = 16 × 81,920 = 1,310,720 per layer
                   × 32 layers   = 41.9 million parameters × 2 bytes = 83.9 MB

So a second fine-tune costs 84 MB as an adapter, against 16.06 GB as a second copy of the model. On a server you start yourself, it takes two flags:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --lora-modules support=./adapters/support summary=./adapters/summary \
  --max-loras 2

Each request then names the adapter as its model, for example "model": "support". Our guide to hosting fine-tuned models covers latency and when to merge an adapter into the model's weights instead.

Serving two different models on one GPU

Adapters only work when the models share a base. A chat model and a code model from different families need two vLLM servers on the same GPU. The setting that makes this work is --gpu-memory-utilization, the share of GPU memory one server may use for weights, working memory and cache.

The vLLM docs call it "a per-instance limit". Their example: two servers on one GPU can each take 0.5. Each server reserves its share without seeing the other. If both start with the default of 0.92, the second finds no memory left.

Here is the worked example on one H100, as a calculation. Option A runs two fine-tunes as two servers, each started with --gpu-memory-utilization 0.46. Option B runs one base model with both adapters. Both reserve a hypothetical 2 GB per server for working memory. That covers activations, the numbers each layer passes to the next, and CUDA graphs, recorded GPU steps that vLLM replays. Requests are a hypothetical 8,192 tokens.

Setup on one H100 (80 GB)Memory shareLeft for the cacheRequests of 8,192 tokens
A: two servers, one per fine-tune2 × 36.8 GB2 × 18.74 GB2 × 17.5 = 35
B: one server, base + 2 adapters73.6 GB55.37 GB51.6
A, per server:  0.46 × 80 GB = 36.8 GB − 16.06 GB weights − 2 GB = 18.74 GB → 18.74 / 1.07 = 17.5 requests
B, one server:  0.92 × 80 GB = 73.6 GB − 16.06 − 0.17 (two adapters) − 2 = 55.37 GB → 55.37 / 1.07 = 51.6 requests

Option B fits 1.48 times as many requests, because it loads the 16.06 GB of weights once. It counts memory only and leaves out the same things for both, so the comparison holds. It ignores the compute an adapter adds and the fact that two servers compete for the same GPU cores. It does not measure speed.

Calculated example on one 80 GB H100 with two Llama 3.1 8B fine-tunes. Setup A, two vLLM servers at a 0.46 memory share each: each loads 16.06 GB of weights and a hypothetical 2 GB of working memory, leaving 18.74 GB of KV cache each, about 35 requests of 8,192 tokens in total. Setup B, one server with the base model and two adapters of 84 MB each at a 0.92 share: 55.37 GB of KV cache, about 51.6 requests.
In our calculated example with an assumed 2 GB of working memory, two Llama 3.1 8B fine-tunes fit about 52 requests as adapters and 35 as two servers on one H100.

A third route is MIG (Multi-Instance GPU), which splits an H100 into isolated slices with their own memory and compute. An H100 80 GB offers, for example, seven slices of 10 GB or two slices of 40 GB. Slices protect each model from the other's traffic. In exchange, memory one slice leaves unused cannot go to the other.

Hardware selection: H100 vs B200 for multi-model workloads

More models on one GPU means more weights and more cache in the same memory. Capacity sets how many models and requests fit. Memory bandwidth, how fast the GPU reads that memory, sets how fast each answer is written. The H100 SXM, the version built onto the server board, has 80 GB at 3.35 TB/s.

We are Lyceum, a sovereign European AI inference provider for open-source models: our GPUs run in European data centres. Prompts and outputs on our API are processed, not stored, and never used for training. Our GPU virtual machines come with an H100 80 GB, an H200 141 GB or a B200 192 GB. Each machine has 1, 2, 4 or 8 GPUs, as listed in our machine types. In the same calculation, option B on one B200 leaves 158.4 GB for the cache, room for about 148 requests of 8,192 tokens.

B200, option B: 0.92 × 192 GB = 176.6 GB − 16.06 − 0.17 − 2 = 158.4 GB → 158.4 / 1.07 = 148 requests

A bigger GPU only pays off when the extra cache gets used. Start with the number of requests your models run at once, and pick the smallest GPU that holds them.

Where NVIDIA Dynamo fits in multi-model serving

NVIDIA Dynamo works one level above a single GPU. It coordinates many copies of a model across GPUs and machines, with routing that knows which copy already holds a request's cache. It also separates reading the prompt from writing the answer, and it scales the number of copies with traffic. It runs with vLLM and with two other serving engines, SGLang and TensorRT-LLM.

Dynamo does not put more models on one GPU. It decides where requests go once you run many copies.

Common mistakes in multi-model deployments

Shares that add up past the GPU

Every vLLM server takes 92% unless you say otherwise. Set the shares so their sum stays below the GPU's memory, as in option A.

One adapter per batch

A batch is the group of requests the GPU processes in one step. The flag --max-loras sets how many adapters vLLM mixes into one batch, and the default is 1. Requests for a second adapter then wait for the next batch. Set it to the number of adapters that get traffic at the same time.

A rank limit set too high

--max-lora-rank reserves memory for the largest rank you allow. The vLLM docs advise setting it to the highest rank you actually use.

Watching memory, not latency

Models on one GPU share its compute. When all of them get busy at once, each answer slows down. Track your slowest requests per model, and check our status page when latency changes on our side.

To run this setup, rent a GPU virtual machine and start vLLM with the adapters you need. A dedicated endpoint, our managed API on GPUs reserved for you, runs one Hugging Face model per deployment. So on dedicated endpoints, different models are separate deployments, and a server with several adapters belongs on a GPU virtual machine.

Frequently asked questions

Can I load and unload LoRA adapters without restarting vLLM?

Yes. With VLLM_ALLOW_RUNTIME_LORA_UPDATING set to True, vLLM accepts requests on /v1/load_lora_adapter and /v1/unload_lora_adapter. Its docs warn against this in production unless the environment is isolated and fully trusted.

Can I train the adapter on Lyceum?

Our fine-tuning is in early access. You upload a few hundred examples, choose GLM-5.3 Flash or Qwen3.8 27B, and we train a LoRA adapter you can download.

Sources

  1. arxiv.org
  2. docs.vllm.ai
  3. huggingface.co
  4. huggingface.co
  5. docs.vllm.ai
  6. docs.nvidia.com
  7. nvidia.com
  8. docs.nvidia.com
  9. docs.vllm.ai