Multi-Model Serving on Single GPUs with vLLM and PagedAttention

Traditional inference servers allocate a static block of VRAM for the Key-Value (KV) cache of every active model. When you serve a single Llama 3 70B model on an H100 (80GB), the weights alone consume roughly 35GB to 40GB in FP16. The remaining memory is reserved for the KV cache to handle long context windows. If that model is only receiving intermittent traffic, the reserved memory sits locked, preventing other workloads from utilizing the available compute cycles.

This inefficiency stems from memory fragmentation. Standard memory allocators cannot predict the length of incoming sequences, leading to 'internal fragmentation' where reserved space goes unused. vLLM addresses this through PagedAttention, which partitions the KV cache into non-contiguous blocks, similar to virtual memory in operating systems. This allows multiple models to share the same physical memory pool more fluidly.

• Static Allocation

  Reserves maximum context length per request, wasting up to 60-80% of VRAM.

• Dynamic Paging

  Allocates memory only as tokens are generated, enabling higher concurrency.

• Multi-Tenancy

  Permits different model architectures to reside on the same chip if the combined weight and cache footprint fits within the VRAM ceiling.

For teams running specialized versions of the same base model, such as different language translations or industry-specific fine-tunes - serving separate full-parameter models is unnecessary. vLLM's Multi-LoRA support allows you to load one set of base model weights into VRAM and dynamically swap small adapter layers (LoRA) during the forward pass.

In this scenario, the base model (e.g., Mistral 7B) stays resident in memory. When a request arrives for 'Adapter A', vLLM applies the specific weights for that request without reloading the entire 7B parameters. This reduces the memory overhead of each additional model from gigabytes to mere megabytes. According to a 2025 performance report, Multi-LoRA setups can support up to 200 concurrent adapters on a single A100 with less than a 5% latency overhead compared to a single-model deployment.

This strategy is particularly effective for SaaS platforms providing personalized AI features. Instead of provisioning a new GPU for every customer's fine-tuned model, you serve them all from a single cluster, scaling to zero when no requests are active for specific adapters. Lyceum Technology provides the underlying infrastructure to support these high-density deployments, ensuring that European data residency requirements are met while maintaining high performance.

Choosing the right silicon is critical when planning for multi-model density. The introduction of the Blackwell (B200) architecture has shifted the economics of multi-tenancy. The B200 offers significantly higher memory bandwidth and larger VRAM capacities, which are the primary bottlenecks for concurrent inference.

When serving multiple models, the GPU must constantly fetch weights and KV cache data from HBM (High Bandwidth Memory). The H100's 2TB/s to 3.3TB/s bandwidth is sufficient for 2-3 concurrent medium-sized models. However, the B200's increased throughput allows for even tighter packing of models without hitting the 'memory wall' where compute cores wait for data to arrive. For European startups, Lyceum offers B200 nodes that provide a structural cost advantage without egress fees, reducing total costs compared to older architectures.

1. VRAM Capacity

   More memory allows for larger KV caches, supporting longer context windows across all served models.

2. Compute Preemption

   Modern NVIDIA drivers allow for better context switching between kernels, reducing the 'jitter' in time-to-first-token (TTFT) when multiple models are active.

3. Power Efficiency

   Serving three models on one B200 is more power-efficient than serving them across three T4 or A10 GPUs, leading to lower operational costs.

The release of NVIDIA Dynamo 1.0 has fundamentally changed how engineers manage multi-model stacks. Dynamo acts as an intelligent orchestration layer that sits between the hardware and the inference engine (like vLLM). It provides real-time VRAM prediction and runtime estimation, allowing the system to decide which models should be co-located on the same GPU based on current traffic patterns.

By integrating Dynamo with vLLM, infrastructure leads can automate the 'packing' of models. If Model A is seeing a spike in traffic, Dynamo can automatically migrate Model B to a different node to prevent resource contention. This level of automation closes the gap between DIY infrastructure and proprietary black-box platforms. Open-stack technologies offer transparent, portable infrastructure that avoids vendor lock-in while delivering high performance.

For teams transitioning off hyperscaler credits, this transparency is vital. Users utilize a stack built on vLLM and NVIDIA Dynamo that remains portable across providers. This portability is a core design principle at Lyceum, ensuring that your scaling strategy remains flexible as your company grows.

The most frequent error in multi-model serving is over-provisioning the KV cache. Many engineers set the gpu_memory_utilization parameter in vLLM to 0.90 (90%) by default. While this works for a single model, it leaves no headroom for the overhead of managing multiple model states or the system-level memory required for context switching. In a multi-model environment, a more conservative 0.70 to 0.80 is recommended to maintain stability.

Another mistake is ignoring the impact of request concurrency on latency. While you can fit three models on one GPU, if all three receive simultaneous bursts of traffic, the compute cores will be shared. This results in a linear increase in time-per-output-token.

Request-Level Latency

Finally, failing to account for GDPR and data residency is a common oversight for European startups. Serving models on US-hosted infrastructure may be acceptable during the prototyping phase, but production workloads involving sensitive customer data often require EU-sovereign hosting. Dedicated inference endpoints in European data centers ensure that multi-model stacks are both efficient and compliant with local regulations.