We take the servers off your plate: provisioning, patching, scaling and monitoring stay with us. What stays with you is application work, the kind you would own on any stack: evaluation, tool permissions, retries and spend limits. Budget time for those first, and choosing an endpoint stays a small decision.
How do you run open models when you have no platform team? You have a feature to ship and nobody to keep GPUs healthy. The endpoint you pick decides your bill and your on-call load. This article explains how to do it. It follows one example workload from a per-token API call to the traffic level where a dedicated deployment pays off. Then it covers the work your team still owns.
Start with the model API your application needs
Serverless inference means you send a request and we run it on shared GPUs. You pay per request, with no idle compute cost, which suits low-volume or spiky traffic. Our inference page states the terms: one OpenAI-compatible API, billed per token, no base fee. We are Lyceum, a sovereign European AI inference provider for open-source models. Our serverless models run in Paris and Finland, and prompts and outputs are processed, not stored.
Open models, whose weights you can download and run yourself, are close behind the best closed ones. Epoch AI measured the gap (29 May 2026). On average since January 2026, the best open models reach a capability level about four months after the best closed ones. Test on your own documents before you trust that.
You can run that test with your existing SDK. Set the base URL, your key and a model ID:
from openai import OpenAI
client = OpenAI(
api_key="lk_...",
base_url="https://api.lyceum.technology/openai/v1",
)
resp = client.chat.completions.create(
model="z-ai/glm-5.3",
messages=[{"role": "user", "content": "Summarise this contract clause."}],
)
print(resp.choices[0].message.content)To price this, take a hypothetical document assistant. The numbers are invented, not a customer. Each request sends 3,000 input tokens and gets 500 output tokens back from GLM-5.3. The model page lists $1.40 per 1M input tokens and $4.40 per 1M output tokens (read 20 October 2026):
input: 3,000 x $1.40 / 1,000,000 = $0.0042
output: 500 x $4.40 / 1,000,000 = $0.0022
one request = $0.0064| Phase | Requests per day | Requests per 30 days | Monthly cost |
|---|---|---|---|
| Pilot | 300 | 9,000 | $57.60 |
| Steady | 20,000 | 600,000 | $3,840.00 |
Hypothetical traffic, prices read 20 October 2026. The table leaves out the lower rate for repeated prompt starts. It also leaves out the text a model writes while it thinks, which is billed as output, so real output can be higher.
Recognise when a dedicated deployment is justified
A dedicated deployment runs a model on reserved GPUs behind its own OpenAI-compatible endpoint. You are billed for the hours those GPUs run, not per request. You deploy from a Hugging Face model ID. If the model's author restricts access, you also pass your Hugging Face token.
A deployment with no requests for 1 hour is paused, and billing stops. The first request after a pause is slower while the model starts again.
Our inference page says the busier you keep a dedicated endpoint, the less each token costs. So the question is how many requests a month keep it busy enough. Call the hourly price of your deployment P, a placeholder for the price we quote you. Using the cost per request from above:
deployment cost per month = 720 hours x P
break-even requests = (720 x P) / $0.0064
= P x 112,500 requests a monthA month of running is 720 hours, so every dollar of hourly price adds 112,500 requests a month, or 3,750 a day, to the break-even. A model this size runs on a full multi-GPU machine, which puts the break-even for this workload in the millions of requests a month. At the steady 20,000 requests a day from the table, per-token billing stays cheaper. Dedicated prices are agreed per contract and we do not publish them, so we cannot fill in P for you.
The example counts every hour of the deployment as paid. That holds at the traffic where dedicated competes: at millions of requests a month, the deployment never goes the hour without requests that would pause it. The example also prices the same model on both sides and says nothing about speed. What it cannot price is why teams reserve capacity anyway: predictable latency, full control of the model, your own endpoint. If those matter to you, we agree commitment pricing per contract, so you know the cost before launch.
Keep agent and retrieval responsibilities explicit
An agent is a program that lets the model call tools in a loop. Retrieval-augmented generation (RAG) fetches your documents and puts them into the prompt. Both need orchestration, retrieval, tool permissions, evaluation and failure handling. Model serving supplies none of these. Our guides on agent latency and cost budgets and RAG sizing cover the sizing.
The security risk sits on your side. OWASP calls it excessive agency: the model takes damaging actions because it has too many tools, permissions or freedom. Give each tool the least it needs.
Tool calls are one of those responsibilities. You can set the tool_choice request field to "required" to ask for a tool call. Several serverless models, GLM-5.3 among them, accept the field but do not enforce it, so a reply can arrive as plain text. Our docs name every affected model, checked 17 September 2026. Check message.tool_calls and fall back when it is empty:
msg = resp.choices[0].message
if not msg.tool_calls:
# the model answered in text: retry, or route to a fallback path
handle_plain_answer(msg.content)The split of work looks like this:
| Responsibility | What we run | What your team runs |
|---|---|---|
| Model serving | GPUs, the endpoint, model versions | Which model ID you call |
| Application state | Nothing is kept for you beyond a usage record | Conversation and document storage |
| Tools | Nothing: we never call your tools | Which functions the model may call, and what each may change |
| Evaluation | Nothing | A test set from your documents, run on every model change |
| Incident response | A public status page | Retries, fallbacks and the message your user sees |
Plan for errors and changing traffic
Errors arrive with an HTTP status. Retry the temporary ones and fix the rest:
| Status | Meaning | What to do |
|---|---|---|
| 429 | No free capacity, or the model backend is rate limited | Retry with backoff |
| 502, 503 | The backend failed or is briefly unavailable | Retry with backoff |
| 504 | Generation passed the 300-second limit | Stream the response or shorten the request |
| 400 and other 4xx | The request was rejected | Fix the request; do not retry unchanged |
Make the backoff randomised. Google's SRE book says to always use randomised exponential backoff when scheduling retries, so many clients do not retry in the same instant. Stream long outputs, because most "cut off" errors are a client timeout firing before the response ends.
Two controls help with traffic you cannot predict. An owner or admin can put a spend limit on an API key. The limit is not exact: a key can pass it by a few minutes of usage before new requests are refused. And every response carries a request ID in its headers, called the trace ID. Quote it with a UTC timestamp when you contact support.
We publish no standard uptime SLA (a contractual availability guarantee) for serverless inference yet. For dedicated endpoints we agree service levels in the contract. Watch the status page, and design your app to survive a failed call.
Choose the smallest setup that meets the requirement
Start on serverless. Stay there while your traffic comes and goes and while you test models. Move to dedicated when one of three things is true. Your traffic is steady enough to pass your break-even. You need a model we do not serve. Or you need latency you can plan around.
Before you decide, count your real requests per day for a week and compute your cost per request with the formula above. That one number turns the choice into arithmetic.
Bring your model and workload requirements to our engineers, and we will help you choose the endpoint.