What a private LLM endpoint is, and two ways to run one

A private LLM endpoint gives you a model of your choice behind your own address and your own API key. Its GPUs are reserved for that endpoint.

Pick the route by how much of the stack you want to run. A dedicated endpoint runs the copies of the model, the scaling and the pausing for you. A GPU virtual machine gives you the whole machine and every setting, and the work that comes with it.

How do you put an open model behind an endpoint of your own, and what does it take to keep it running?

Route 1: a dedicated endpoint from a Hugging Face model ID

We are Lyceum, a sovereign European AI inference provider for open-source models. Our GPUs run in European data centres. On our inference products, prompts and outputs are processed, not stored, and never used for training. A dedicated endpoint gets GPUs reserved for it alone. We agree the data centre with you, and you can ask for a specific EU location.

A dedicated deployment starts with a Hugging Face model ID, the name a model has on Hugging Face. You choose a hardware profile, which is the GPU type. The platform then runs one or more replicas. A replica is one running copy of the model on its GPUs.

lyceum infer deploy meta-llama/Llama-3.1-8B-Instruct \
  -g gpu.h100 \
  --min-replicas 1 \
  --max-replicas 3

The command returns a deployment ID. The endpoint is OpenAI-compatible, so you call it with that ID as the model name:

from openai import OpenAI
client = OpenAI(
    api_key="lk_...",
    base_url="https://api.lyceum.technology/api/v2/external",
)
resp = client.chat.completions.create(
    model="<deployment_id>",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

The autoscaler adds and removes replicas between your minimum and maximum. It aims at a target of requests per second per replica, 10 by default. It also watches a latency target: by default, 95 out of 100 requests should finish within 5,000 ms. Before it removes a replica, it waits 300 seconds, the stabilisation window. That stops it from scaling down and up again in quick succession.

Two product behaviours decide your bill. Replicas are billed for the time they run, not per request. The minimum applies while traffic flows. After an idle hour, the platform pauses the deployment at zero anyway, and billing for its replicas stops. The next request restarts it at your minimum, so that request is slower.

Before you deploy, check how much GPU memory your model needs. The Hugging Face deployment guide covers the vLLM settings for one model.

A worked example: replica hours for one day of traffic

Take a hypothetical internal assistant on a weekday. From 09:00 to 18:00 it receives 25 requests per second. From 18:00 to 23:00 it receives 6. From 23:00 to 09:00 nobody uses it. The deployment keeps the default target of 10 requests per second per replica, with a maximum of 3 replicas.

replicas needed = requests per second / 10, rounded up
replica hours   = replicas x hours in that period
Period (hypothetical traffic)Requests per secondReplicasReplica hours
09:00 to 18:0025327
18:00 to 23:00615
23:00 to 24:00, idle (minimum of 1 until the idle-hour pause)011
00:00 to 09:00, paused000
Whole day33

Three replicas kept on all day would run for 72 replica hours. Following the traffic needs 33, which is 54 percent fewer hours of GPU time to pay for.

Calculated example. Step chart of running replicas over one hypothetical weekday. Zero replicas until 09:00, three until 18:00, one until 24:00 including one idle hour before the pause. The shaded area adds up to 33 replica hours. A dashed line at three replicas all day stands for 72 replica hours.
Our calculation for a hypothetical weekday: following the traffic, replicas run 33 hours instead of 72 for three replicas kept on all day.

The calculation favours the autoscaler. It assumes the replica count changes the moment traffic changes. It ignores the stabilisation window and the minutes a replica needs to start. It also shows hours, not money: the commercial terms for a dedicated endpoint are agreed per contract. With steady peak traffic around the clock, fixed 3 replicas and autoscaling would both come to 72 replica hours.

Route 2: your own Docker image with vLLM on a GPU VM

Some teams need more control than a dedicated endpoint gives, such as a pinned engine version or custom code in the container. For them, the second route is a GPU virtual machine. It is a machine with root access in our European data centres, and you pay per second while it runs.

You launch it with your SSH public key, the key your machine uses to log in:

lyceum vm start -h h100 -k "$(cat ~/.ssh/id_ed25519.pub)" -g 1

The launch command prints the machine's IP address. You log in over SSH and run the serving engine as a container. The open-source vLLM project publishes an official Docker image that starts an OpenAI-compatible server. A minimal start looks like this. The --ipc=host flag lets the container use the host's shared memory, which vLLM needs to pass data between its processes.

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8000:8000 --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-0.6B \
  --api-key "$VLLM_API_KEY"

The --api-key flag makes the server check every request for that key. You can pass several keys, which lets you rotate them without downtime. Pin the image to a version tag instead of latest, so a restart does not change the engine under you.

Everything around the container is now yours: TLS (the encryption behind https), a public address and restarts. Scaling means more machines behind a self-hosted API gateway.

Two rows. Dedicated endpoint: your app calls an OpenAI-compatible endpoint, an autoscaler and replicas on reserved GPUs; everything after your app sits inside a box labelled We run it, which pauses at zero after an idle hour. GPU virtual machine: your app calls your TLS and gateway, your vLLM container on port 8000, on one GPU VM; only the VM sits inside We run it.
On a dedicated endpoint we run everything after your app; on a GPU virtual machine you run the gateway and the container, and we run the machine.

What the open serving stack under your endpoint does

While a model answers, it keeps a record of every token it has read: the KV cache (key-value cache). The vLLM engine stores that record in small fixed-size blocks instead of one large reserved area per request. Less memory sits unused, so more requests fit on one GPU. Throughput is how many tokens a server produces per second. The vLLM paper reports near-zero wasted cache memory and 2 to 4 times the throughput of earlier serving systems at the same latency.

NVIDIA Dynamo works one level up, across many GPUs. It is open source and can split reading the prompt and writing the answer onto different GPUs. Reading the prompt is compute-heavy in one burst, while writing is limited by memory, token by token. Split, each step runs on GPUs sized for its work. Its router sends a request to the GPU that already holds the matching KV cache. It works with vLLM and two other serving engines, TensorRT-LLM and SGLang.

We use it ourselves, in our serverless inference: shared open models you call through one API and pay for per token. On a shared API, the next turn of a conversation can land on a machine that does not hold its KV cache. That machine reads the whole conversation again, which is slower and costs more GPU time. On 5 October 2026 we moved GLM-5.3 to a new engine with our own modifications on top of NVIDIA Dynamo. GLM-5.3 is one of the open models on our API. A cache hit means a turn finds its earlier record and skips that re-reading. For GLM-5.3, our serverless API now hits the cache about as often as a dedicated machine would. For a chat or agent workload on that model, keeping the cache is no longer a reason to move to a dedicated endpoint.

Which route fits your traffic and your team

Your situationRoute
Steady production traffic, a Hugging Face model, no platform teamDedicated endpoint
Custom engine settings, your own code in the containerGPU virtual machine with your Docker image
Traffic that comes and goes, a model from our catalogueServerless inference, billed per token

Per-token billing follows your traffic. A dedicated endpoint costs less per token the busier you keep it. The break-even between the two depends on how many hours a day your traffic stays high.

Start with your own traffic log: count requests per second by hour for one week. If the busy hours fill most of the day, deploy your model with lyceum infer deploy and let the replicas follow the curve. Our guide Run open models without a platform team: pick your endpoint compares all three routes in more depth.

Frequently asked questions

Can I deploy a gated model such as Llama?

Yes. A gated model is one whose weights you must request access to on Hugging Face. Pass your Hugging Face token when you create the deployment.

Where can I see whether the platform has an incident?

Our public status page at status.lyceum.technology lists service incidents. For a dedicated endpoint, service levels are agreed in your contract.

Sources

  1. docs.vllm.ai
  2. docs.vllm.ai
  3. arxiv.org
  4. NVIDIA Developer Blog, Introducing NVIDIA Dynamo, 18 March 2025
  5. NVIDIA, ai-dynamo/dynamo repository and v1.0.0 release tag (read 3 August 2026)