Serverless inference cold start latency: shorten it or hide it

What a GPU cold start is made of

A cold start is the wait a request pays when no copy of your model is running. In that time, a GPU is attached, the software and model files load, and the program that answers requests starts. You shorten it by moving fewer bytes and skipping setup work, or avoid it by keeping one copy running.

Stopping every copy when traffic stops is called scale-to-zero, and it saves money for every idle hour. Count how often your traffic pauses for an hour before you decide whether copies stay running.

An engineer on a sales call asked us about cold start times. How long do cold starts take, and how do we keep them away from our users? The answer depends on which of four steps takes the longest.

Each running copy of a model is called a replica. The four steps are attaching a GPU, pulling the software image, loading the weights and starting the serving engine. The software image packages the drivers and libraries. The weights are the model's learned numbers, often many gigabytes. The serving engine is the program that answers requests, such as vLLM, a widely used open-source engine for language models.

A June 2026 study of vLLM measured the engine start step by step. Most of it is CPU-bound, meaning the processor that runs the setup code is the bottleneck, not the GPU. On the authors' machine, a 3-billion-parameter model needed 20.32 seconds to start the engine. Moving from an NVIDIA L40S to the faster NVIDIA H100 GPU gave almost no speedup. Reading the weights, even from disk, was only about 7 to 10% of that engine start.

Log each step as its own timer before you optimise anything. Otherwise you may speed up the image download while the time goes into engine setup.

When scale-to-zero pays off: one week of an internal assistant

Scale-to-zero pays off when idle hours are many and slow first requests are rare. Our dedicated deployments pause on their own. A deployment with no requests for one hour is scaled to zero and marked paused, and billing for its replicas stops. The next request resumes it, and that first request is slower while the replicas start.

Take a hypothetical team that runs an internal document assistant on a 27-billion-parameter open model. Staff use it on weekdays from 09:00 to 18:00. The weights take 54 GB at 16 bits per number, so they fit on one 80 GB GPU.

The pause leaves 118 of 168 hours unbilled, or 70%, for the price of five slow mornings. For an internal tool, that trade favours the pause.

The example assumes one replica, no idle gap over an hour during the day and no traffic spike. It does not show how long a resume takes. We publish no resume time, so measure it on your own deployment.

You deploy a Hugging Face model with our CLI. Replace the placeholder with your model's Hugging Face ID. In the example, `--min-replicas 1` keeps one replica running from the first request until the pause, so daytime requests never wait. It does not switch off the one-hour pause, which applies to every deployment.

lyceum infer deploy your-org/your-27b-model -g gpu.h100 --min-replicas 1 --max-replicas 3

How to shorten each stage of a cold start

Each stage has its own lever, and your step timers show which one to pull first.

Move fewer bytes for the weights

Quantisation (keeping each weight in fewer bits) shrinks the download. The example's 54 GB model needs 13.5 GB at 4 bits per weight, a quarter of the bytes. Test answer quality after you quantise.

Store weights as Safetensors, a file format Hugging Face describes as safe and fast to load because it avoids extra copies. The vLLM engine can also stream weights to the GPU while it is still reading them.

Make the image arrive sooner

Keep the image small and cached on the machine that runs it, so a restart skips the download. Tools that start a container before its image has fully downloaded exist too. Compress image layers with zstd: on its project's benchmark, it decompresses about four times faster than zlib. Kubernetes image volumes mount files from an image registry into a pod, so the weights can ship apart from the code image.

Skip or reuse engine setup

Part of vLLM's start is compiling: it turns the model's Python code into fused GPU programs that run faster. The engine saves that result in a compile cache on disk, so the next start can reuse it. In the study, this step took 11 to 21 seconds without the cache and 3 to 6 seconds with it. Keep the cache directory on storage that survives restarts.

The engine also records CUDA graphs at startup. A CUDA graph is a recorded series of GPU operations that the engine replays instead of launching each one again. The `--enforce-eager` flag in vLLM skips the compiling and the recording for the fastest start, at the cost of slower answers. Our vLLM production deployment guide covers the other startup flags.

Copy a running state instead of rebuilding it

Some tools skip setup by copying a state that already exists. NVIDIA's open-source Dynamo streams model weights GPU to GPU, so a new replica starts faster. NVIDIA's cuda-checkpoint saves and restores the GPU state of a running process. Lightweight virtual machines such as Firecracker boot in under 125 milliseconds and resume from snapshots. A fast boot still leaves the weights to load into GPU memory, unless a GPU-side tool restores them. Our NVIDIA Dynamo guide walks through a deployment.

How to hide the cold starts you cannot remove

A warm-up request hides a cold start by paying it before a user arrives. In the example, a scheduled request at 08:45 resumes the paused deployment. The first employee at 09:00 then finds it running. Our dedicated endpoint takes the deployment ID as the model name:

curl https://api.lyceum.technology/api/v2/external/v1/chat/completions \
  -H "Authorization: Bearer $LYCEUM_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "<deployment_id>", "messages": [{"role": "user", "content": "ping"}]}'

Voice or code completion can never wait. That traffic needs a keep-alive request more often than the one-hour idle pause, or a different product. That product is serverless inference: you call an open model from our catalogue per token, with no base fee. There is no image to pull and no weights to load on your side.

Lyceum is a sovereign European AI inference provider for open-source models, built and run in Europe. Our serverless models run in Paris and Finland, and prompts and outputs are processed, not stored, and never used for training.

Point any OpenAI client at `https://api.lyceum.technology/openai/v1`. The serverless inference docs list the models and parameters. Time to first token (TTFT) is how long a request waits for its first generated token. Our status page charts the TTFT we measure on each serverless model.

For GLM-5.3, an open model from Z.ai, the median of that chart from 14 to 21 October 2026 was about 0.4 seconds. GLM-5.3 reasons before it answers, and its first token can be part of that reasoning. So the first visible word of the answer arrives later than 0.4 seconds.

How to compare serverless GPU platforms on cold starts

Compare platforms on your own model after it has been idle, not on a demo endpoint that is always warm.

Distance adds delay to every request, warm or cold. Our EU vs US latency comparison measures it per provider.

If your traffic pauses for hours and a slow morning is acceptable, deploy a dedicated endpoint and let it pause. If every request must be fast, start with per-token serverless inference.