GPU efficiency
5 articles in GPU efficiencySearch all articles
Articles
10 November 2026
Quantization on dedicated inference: NVFP4 vs FP8 vs full precision
Storing a model's numbers in 8 bits instead of 16 halves its GPU memory and usually keeps its answers. Going down to 4 bits saves more memory, costs a little accuracy and needs NVIDIA's Blackwell GPUs. On our dedicated endpoints, we agree this setting with you before launch.
4 June 2026
Long context inference: GPU requirements & VRAM guide
Teams that want a model to read whole contracts or codebases are often surprised by the GPU bill. Every token a model reads takes GPU memory, so long requests need far more memory than the model alone: one 128K-token request on Llama 3.1 70B adds 42.9 GB. Size the GPUs for how many long requests run at once, store that memory in 8-bit, and cap request length at what users really send. Past 128K tokens, call a 1M-token model per token on Lyceum's EU-hosted API instead of buying GPUs.
18 May 2026
GGUF vs GPTQ vs AWQ: the definitive LLM quantization framework
When you run an open model yourself, you have to pick a file: the original, or a smaller GGUF, GPTQ, AWQ or FP8 version. The wrong pick means the model won't load, or leaves too little GPU memory for your users. The rule is simple: GGUF for laptops, CPUs and Macs, AWQ or GPTQ for serving users on GPUs, and FP8 on newer GPUs such as the H100. On Lyceum you can deploy AWQ and GPTQ files on dedicated endpoints in Europe, or skip the choice and call the model per token on our API.
19 April 2026
Multi-model serving on single GPUs with vLLM and PagedAttention
Teams with several fine-tuned versions of one open model often give each its own GPU, and most of those GPUs sit idle most of the day. They can share one GPU instead. If all versions come from the same base model, load them as LoRA adapters, small add-on files, on one vLLM server, so the base model sits in memory only once. If they are different models, run one vLLM server per model and give each a fixed share of the GPU's memory. On Lyceum you can run this on your own GPU machine in European data centres, with an H100, H200 or B200.
1 September 2026
Fixing vLLM CUDA out of memory: KV cache tuning guide
If you run an open model on your own GPUs with vLLM, the server can stop with "CUDA out of memory". It hits teams that host large models such as Llama 3.1 70B themselves. It is rarely a leak: at startup vLLM reserves GPU memory for the model and for the requests it will handle, and the two don't fit. If it fails at startup, give the model more room with a second GPU or an 8-bit version. If it fails under load, too many long requests share the memory, so shorten them or run fewer at once.
No articles match.
Try a different word or topic, or clear the search.