What serverless GPU inference is

A large language model needs a GPU (graphics processing unit) with enough memory to hold its weights, the numbers it learned in training. Running that GPU yourself means renting it, loading the model and paying for every hour it sits idle.

Serverless GPU inference moves that work to the provider. The model is already loaded on GPUs that many customers share. You send a request to an API and pay per token. A token is the chunk of text a model reads or writes, roughly three quarters of an English word. A quiet night costs nothing.

You pay for the tokens you use, not for the hours a GPU is booked. When is that cheaper for you, and what happens to your data on the way?

Serverless GPU inference architecture: what happens to one request

We are Lyceum, a sovereign European AI inference provider for open-source models: built and run in Europe, with our serverless models in Paris and Finland. Our serverless API is OpenAI-compatible. It accepts the request format of OpenAI's API, so the official OpenAI SDK works with a new base URL and key. On our serverless inference, the call looks like this:

from openai import OpenAI
client = OpenAI(
    api_key="lk_...",
    base_url="https://api.lyceum.technology/openai/v1",
)
resp = client.chat.completions.create(
    model="qwen/qwen3.8-27b",
    messages=[{"role": "user", "content": "Summarise this ticket in two sentences: ..."}],
    stream=True,
)
for chunk in resp:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

The request then passes three stages.

  1. The gateway, the server that receives your call, checks your API key. Our legal page shows what it keeps: a usage record with your organisation, the model, token counts and the time. Your prompt passes through in memory and is never logged.
  2. A router picks a running copy of the model, called a replica, that has room for your request.
  3. The serving engine on that replica reads your prompt and writes the answer token by token. Streaming, switched on in the call above, sends each token as it is written.
Flow of one serverless request in four boxes: your app calls with the OpenAI SDK and a new base URL; the gateway checks the API key and keeps a usage record with token counts, while the prompt passes through in memory; a router picks a replica with room; the highlighted replica runs the model on its GPUs and holds the KV cache that prompt caching reuses.
One request on our serverless inference: the gateway keeps token counts, not your prompt, and the KV cache stays on one replica.

Why the first token comes fast or slow

While the engine reads a prompt, it stores a compact record of every token in GPU memory: the KV cache (key-value cache). The open-source engine vLLM stores that record in small fixed-size blocks, a method its authors call PagedAttention. Small blocks waste little memory, so more requests fit on one GPU.

If your next request starts with exactly the same text, a replica that still holds that cache only reads the new part. This reuse is called prompt caching. vLLM's documentation describes the same mechanism as automatic prefix caching. The cache lives on one replica only. So routing decides whether prompt caching works: a request that lands on another replica reads everything again.

NVIDIA's open-source Dynamo framework, in production since 16 March 2026, can route requests to GPUs that already hold the KV cache of earlier turns. Its documentation calls this KV-aware routing. It works with vLLM and with two other serving engines, SGLang and TensorRT-LLM. On 5 October 2026 we moved GLM-5.3, one of the open models on our API, to a new engine built on Dynamo. Each turn of a session now goes back to the GPU that holds its KV cache. More requests find their cache, so each cached token takes less GPU time to serve. That let us cut GLM-5.3's cached input price by 41%.

A cold start, the wait while a model loads onto a GPU, does not apply here: the shared model is already running. Time to first token (TTFT), the wait before the first word of the answer, still varies with load. For Qwen3.8 27B, our status checks measured a median TTFT of 291 ms and 45 output tokens per second. These are short requests over the last 7 days, read on the model page on 30 October 2026.

Serverless GPU inference economics: a worked example

Take a hypothetical support assistant on Qwen3.8 27B, an open model with 27 billion parameters. It handles 40,000 requests per working day. Each request starts with the same 1,800 tokens of instructions and help-centre text. It adds 400 tokens of new text and gets 350 tokens back. Prices are from the model page, read on 30 October 2026, with no base fee. Input costs $0.40, cached input $0.10 and output $2.40 per million tokens.

Without cache hits
  input   (1,800 + 400) × 40,000 = 88.0M tokens × $0.40 = $35.20
  output          350 × 40,000 = 14.0M tokens × $2.40 = $33.60
  per day                                               $68.80
  per month (22 working days)                        $1,513.60
With every shared start found in the cache
  new input        400 × 40,000 = 16.0M tokens × $0.40 = $6.40
  cached input   1,800 × 40,000 = 72.0M tokens × $0.10 = $7.20
  output           350 × 40,000 = 14.0M tokens × $2.40 = $33.60
  per day                                               $47.20
  per month (22 working days)                        $1,038.40

Reused prompt starts take $21.60 off each day, 31% of the bill. Caching is best-effort, so the real bill sits between the two totals. The usage numbers in each response tell you where you are. They count the cached tokens, and the field is absent when nothing was cached:

Calculated example for a hypothetical support assistant on Qwen3.8 27B with 40,000 requests a day: without cache hits the day costs $68.80 ($35.20 input, $33.60 output); with every shared 1,800-token start cached it costs $47.20 ($6.40 new input, $7.20 cached input, $33.60 output). Prices from the model page, read 30 October 2026.
Calculated example with hypothetical traffic: the same day costs $68.80 with no cache hits and $47.20 when every shared start is cached.
details = resp.usage.prompt_tokens_details
cached = details.cached_tokens if details else 0

The example favours caching, because it assumes requests arrive close enough together to find a warm cache. It also leaves out retries and reasoning tokens, the text some models write while thinking before the answer, billed as output.

To get closer to the lower total, put the instructions and fixed context first and the user's message last. Keep changing values such as timestamps out of the start.

Per token or a dedicated endpoint?

A dedicated endpoint reserves replicas for one customer and bills the time they run, not the tokens. Our dedicated endpoints pause after an hour without requests, and the first request after a pause waits while the replicas start. Say the example's traffic runs from 08:00 to 20:00 and one replica can handle it, an assumption you would test. That replica would then run 13 hours per working day: 12 hours of traffic plus the idle hour before the pause. Over 22 working days that is 286 replica-hours, the number to multiply by the rate in a quote.

Dedicated endpoints come with commitment pricing: a price for reserved capacity, agreed per contract. Compare your monthly token bill, here $1,038.40 to $1,513.60, with a quote for those replica-hours. Our guide to pay-per-token vs dedicated GPU inference works through that decision in depth.

We recommend starting on per-token billing and pricing a dedicated endpoint only once a month of real usage shows a steady bill. Spiky traffic leaves reserved replicas idle, and idle replicas cost money that tokens do not.

Serverless GPU inference compliance: what to check before you send data

Serverless means your prompts leave your network and run on someone else's GPUs. Your customers will ask where, who can see them and what is kept. Settle these questions first.

QuestionWhat to checkOur answer
Where does the model run?The processing region per model, not per providerOur website says our serverless models run in Paris and Finland; each model page states its region. Qwen3.8 27B's page lists "Processing region: EU-hosted"
What is stored?Prompts, outputs and logsPrompts and outputs are processed, not stored
Is my data used for training?The provider's termsNever used for training
Who processes data for us?A data processing agreement (DPA), the contract under GDPR, and the list of subprocessors, the other companies that handle the dataThe DPA is part of our terms; the subprocessor list is on our legal page
Which certifications apply?Certificates actually held, not "in progress"; ISO 27001 is the common security-management standardOur own ISO 27001 audit is in progress

Separately from our own audit, the data centre operators hold ISO certifications for their facilities. The legal page has the DPA, the TOMs (technical and organisational measures) and the subprocessor list, one click each.

Check each model's processing region on its model page before you send personal data.

Limits to plan for on a serverless API

A shared API has limits that a GPU of your own does not.

  • Generation time. One request may run for up to 300 seconds. For long answers, stream the response, so your client's timeout does not cut it off.
  • Output length. One request may ask for at most 131,072 output tokens.
  • Rate limits. An HTTP 429 error (Too Many Requests) means you sent more than your limit allows, or the model is under heavy load. Retry with exponential backoff, which waits longer after each failed try.
  • Availability. An SLA (service level agreement) is a contract promise on uptime. We don't publish a standard uptime SLA for serverless inference yet, and it's on our roadmap. Our status page shows incidents live.
import time
from openai import OpenAI, RateLimitError
client = OpenAI(api_key="lk_...", base_url="https://api.lyceum.technology/openai/v1")
def ask(messages, tries=5):
    for attempt in range(tries):
        try:
            return client.chat.completions.create(model="qwen/qwen3.8-27b", messages=messages)
        except RateLimitError:
            time.sleep(2 ** attempt)
    raise RuntimeError("still rate-limited after retries")

Need predictable latency for one model at all hours? Then read the cluster guide to choosing an endpoint and our inference page.

Frequently asked questions

Do I pay for a cold start on serverless inference?

No. The model already runs on shared GPUs, so no GPU starts for your request. You pay for the tokens of each request only.

Which GPUs run serverless inference?

You do not choose one. The provider runs each model on GPUs sized for it and you call the model by its ID. Pick the model, then check its speed and price on its model page.

Can I keep the same code when I move to a dedicated endpoint?

Yes, both accept the OpenAI request format. You change the base URL and set the model field to your deployment ID, the name the platform gives your endpoint.

Sources

  1. arxiv.org
  2. docs.vllm.ai
  3. nvidianews.nvidia.com
  4. docs.nvidia.com