A priority tier on a shared AI API lets your requests skip ahead in the queue, but it adds no GPUs. When the shared pool is full, your requests still wait or get turned away. Latency you can plan for comes from GPUs reserved for your traffic: on our platform, a dedicated endpoint sized for your busiest hour.
The rule to keep: if your latency target must hold at your busiest hour, pay for capacity at that hour. A place in the queue is not enough.
A customer put it to us in a sales call: some providers have a priority option for inference. Do you offer that? Our answer was no. The rest of this article explains why, and what we offer instead.
A priority tier reorders the queue, it does not add GPUs
A priority tier is a paid option on a shared API. It puts your requests ahead of other customers' requests while they wait for a GPU. The idea comes from how inference engines schedule work. vLLM, a widely used open-source inference engine, has a priority scheduling policy. It handles requests by their given priority, and arrival time decides ties.
That policy decides the order of the queue. It does not change how many requests the GPUs can serve each second. If many requests carry the same priority, they fall back to first come, first served. If every GPU is busy, the request at the front of the queue still waits for one to free up.
Research on LLM serving names the trade-off plainly. The DistServe paper finds that strict latency targets force a choice. Serving systems must favour either the time to the first token or the time per token after it, or over-provision compute to meet both. Over-provision means holding more GPUs than the average load needs. That spare capacity is what keeps latency steady at peak, and a queue position cannot replace it.
We are Lyceum, a sovereign European AI inference provider for open-source models. Our serverless models run in Paris and Finland. Prompts and outputs are processed, not stored, and never used for training. The sections below show how we handle a full pool on serverless, and how you reserve capacity instead.
Where the wait before the first token comes from
Time to first token (TTFT) is the time from sending a request to receiving the first token of the answer. It is the delay a user feels before text starts to appear. NVIDIA's benchmarking guide notes that the full request latency also includes queueing, batching and network time.
Queueing is the part a priority tier touches. Batching means the GPU groups several requests and runs them together. Batching and the model's own reading of the prompt still happen once the request reaches a GPU. So a priority tier can only remove waiting time. And it only removes the wait caused by other customers' requests ahead of yours.
Latency targets are usually set on the p95: the time that 95 of every 100 requests stay under. An average hides the slowest requests, and the slowest requests are the ones users notice. Our guide to fixing slow outliers at p95 goes deeper on measuring it.
Engines can also protect a TTFT target by refusing work. Before a model writes, it reads the whole prompt, a step called prefill. The vLLM engine can reject new requests when the queue of prompts waiting for prefill would push TTFT past the target. A rejected request is fast, but it is not an answer. Either way, a full pool shows up as waiting or as a refusal. For the other causes of a slow first token, see our explainer on what drives time to first token.
What our serverless API does when capacity is full
Our serverless inference shares GPUs across customers and bills per token. When no slot is free for a model, our API answers with HTTP 429 and the message "inference capacity exhausted". It comes from our own capacity control, and the fix is to retry with backoff.
A 429 can also mean you sent more requests than your current limit allows. Backoff means waiting a little longer after each failed attempt before you try again. If 429s keep coming, tell us: we look at your limits with you, or at a dedicated endpoint.
import os, random, time
from openai import OpenAI, RateLimitError
client = OpenAI(
api_key=os.environ["LYCEUM_API_KEY"],
base_url="https://api.lyceum.technology/openai/v1",
)
def ask(messages, attempts=5):
for attempt in range(attempts):
try:
return client.chat.completions.create(
model="z-ai/glm-5.3-flash", messages=messages, stream=True
)
except RateLimitError:
time.sleep(2 ** attempt + random.random()) # 1 s, 2 s, 4 s ... plus a random fraction
raise RuntimeError("still rate limited after retries")The random fraction of a second spreads out retries, so many clients do not all retry at the same moment. The example also sets stream=True. Streaming sends tokens as they are generated, which lowers time to first token, and our docs recommend it for anything user-facing.
We do not publish a standard uptime SLA for serverless inference yet, and it is on our roadmap. Incidents appear on our status page. Serverless suits traffic that comes and goes, where a retried request is acceptable.
Dedicated endpoints: reserved replicas with a p95 target
A dedicated endpoint runs your model on GPUs reserved for your endpoint, planned around your production workload. Each copy of the model is called a replica, and the platform balances requests across the replicas for you. Our dedicated inference docs call it the right tool when you need predictable latency.
You pay for the time the replicas run, not per request. The replica count moves between a minimum and a maximum that you set. An autoscaler is the part of the platform that adds and removes replicas. It decides on two targets: requests per second per replica, and a p95 latency in milliseconds.
The command-line defaults are 10 requests per second per replica and a p95 target of 5,000 milliseconds. The minimum holds as long as requests keep arriving: the platform keeps at least that many replicas running, so they answer without a start-up delay. Only after 1 hour with zero requests does the deployment scale to zero and pause. The first request after a pause is slower, because the replicas have to start again.
lyceum infer deploy your-org/your-model \
-g gpu.a100 \
--min-replicas 1 \
--max-replicas 3 \
--target-rps 10 \
--target-latency 5000The p95 target is not a promise. It tells the autoscaler when to add a replica. What you can rely on is the capacity itself: the replicas serve only your traffic, so no other customer's peak fills your queue. For dedicated endpoints and reserved GPU capacity, we agree service levels with you as part of your contract. The data centre is also agreed for your workload.
Worked example: sizing replicas for a support assistant
Take a hypothetical support assistant. It peaks at 30 requests per second for 10 hours a day and runs at 5 requests per second for the other 14 hours. We use the command-line default of 10 requests per second per replica.
peak: 30 req/s ÷ 10 req/s per replica = 3 replicas × 10 h = 30 replica-hours
off-peak: 5 req/s ÷ 10 req/s per replica = 0.5, minimum of 1 replica × 14 h = 14 replica-hours
autoscaled day: 30 + 14 = 44 replica-hours
fixed at peak: 3 × 24 = 72 replica-hours
difference: 72 − 44 = 28 replica-hours a day, 39% fewerThe peak decides how much capacity you reserve: 3 replicas, so the busiest hour has room. The minimum of 1 replica keeps one copy running through the quiet hours, so the first morning request does not wait for a start. Letting the autoscaler follow the traffic saves 28 replica-hours a day against holding 3 replicas around the clock.
| Setup | Replicas at peak | Replicas off-peak | Replica-hours a day |
|---|---|---|---|
| Fixed at peak size | 3 | 3 | 72 |
| Autoscaled, minimum 1 | 3 | 1 | 44 |
These numbers are a calculated example, not a measurement. The example favours autoscaling in two ways. It assumes every replica serves exactly 10 requests per second. It also assumes new replicas are ready the moment traffic rises. It does not show how fast your model serves your prompts, which depends on the model, the prompt length and the GPU. It also leaves out prices and the scale-down window: the platform waits 300 seconds by default before it removes a replica after traffic falls. Measure your own requests per second per replica before you size the endpoint.
When to stay on serverless and when to reserve capacity
Stay on serverless inference while your traffic comes and goes and a retried request is acceptable. Per-token billing follows your traffic, so quiet hours cost nothing. Use streaming and backoff, and watch your p95.
Reserve a dedicated endpoint when your p95 target has to hold at your busiest hour. We recommend sizing it for that hour and letting the autoscaler handle the quiet ones. A dedicated endpoint is priced for the capacity you reserve, so the busier you keep it, the less each token costs. Our guide to the pay-per-token and dedicated break-even shows how to compare the two bills.
If your traffic is steady through most of the day, ask us for a price. Send us your peak requests per second and your p95 target through our inference page. We price a dedicated endpoint against your token bill.
Frequently asked questions
Do you offer a priority tier on serverless inference?
No. On our serverless API, all requests share the same pool, and a full pool returns a 429 that you retry with backoff. When latency has to hold at peak, we offer a dedicated endpoint with GPUs reserved for you.
Does a dedicated endpoint come with a latency or uptime SLA?
Service levels for dedicated endpoints and reserved GPU capacity are agreed with you in your contract. The p95 target in the deployment settings steers the autoscaler and is not itself a service level.
What happens to the first request after a quiet hour on a dedicated endpoint?
After 1 hour without requests, the deployment scales to zero and pauses, which stops billing. The next request wakes it up and is slower while the replicas start. If that delay matters, send one small warm-up request a few minutes before your traffic starts, for example before the working day.