Most eval failures start in your client, not the API
When a large eval on a serverless API loses requests, the cause is usually a timeout in your own client. Our serverless inference docs name the most common source of "cut off" errors. It is a client-side timeout that fires before the response finishes, not the server.
Why do so many requests fail when you run a large eval on a serverless inference API, and how do you run it reliably? The question comes up when you test whether an open model can replace your current one. An eval is a test run of hundreds or thousands of prompts, scored automatically. Serverless inference means you call a shared copy of the model through an API and pay per token.
A failed request leaves a hole in the score. If the failures cluster on long prompts, the score leans toward short tasks.
Lyceum is a sovereign European AI inference provider for open-source models. Sovereign here means the service is built and run in Europe. Our serverless models run in Paris and Finland, and prompts and outputs are processed, not stored, and never used for training (our data handling). Your eval runs on the same OpenAI-compatible API and in the same locations as production.
Why long answers get cut off: timeouts and retries
Long answers get cut off when the client's timeout is shorter than the time the model needs to write them. A timeout is how long your client waits for a response before it gives up.
Our serverless API allows one request up to 300 seconds of generation time. A long answer, or a large context on a non-streaming call, can take minutes.
The eval tool decides how long it waits. The open-source eval tool lm-evaluation-harness sets a 30-second timeout for API requests by default, and retries a failed request 3 times. Its num_concurrent setting is the concurrency: how many requests it sends at the same time.
A streamed response sends tokens as they are generated, instead of one reply at the end. We recommend it for long outputs. Data keeps flowing, so the connection never sits silent long enough for a timeout to fire.
Heavy load adds a second trap. The time to first token (TTFT) is how long you wait before the first word arrives. Under heavy load it can spike. Some libraries end a stream if no new token arrives within a fixed window. One is litellm, which routes calls to many LLM APIs. Raise that inactivity timeout as well as the overall request timeout.
A worked example: 1,000 prompts with a 30-second timeout
Take a hypothetical eval of 1,000 prompts. Assume 15% of them need answers that take 90 seconds each (both numbers are hypothetical). The eval tool runs with its defaults: a 30-second timeout and 3 retries.
Long prompts 1,000 x 0.15 = 150
Attempts per prompt 1 first try + 3 = 4
Attempts cut off 150 x 4 = 600
Waiting dropped 600 x 30 s = 18,000 s (5.0 hours)
Reported failures 150 / 1,000 = 15%The eval reports 15% failures, though a 90-second answer fits under our 300-second limit. The 18,000 seconds are waiting time on attempts the client started and then abandoned. With a timeout above 300 seconds, or with streaming, the same answer comes back.
The example is idealised. Every long answer takes exactly 90 seconds, and no capacity errors occur. It does not show how often capacity errors happen on a given model or day, or how long your real answers take.
Sort failed requests by error type
Sort the failed requests by error type before you blame the model or the provider. A timeout on your side, a capacity error and a rejected request each need a different fix.
The model backend is the server that runs the model behind our API. We do not pass its raw errors through. We map each failure to a stable message, so you can match on the message and the HTTP status. The table follows the error reference in our serverless docs.
| What you see | What it means | What to do |
|---|---|---|
| Your client's own timeout error | Your client stopped waiting | Raise the timeout or stream |
429 inference capacity exhausted | All our capacity for this model is in use right now | Retry with backoff |
429 inference service temporarily rate limited | The model backend returned 429 | Retry with backoff |
502 inference service temporarily unavailable | The model backend failed | Retry with backoff |
503 service temporarily unavailable | The selected backend is temporarily unavailable | Retry with backoff |
504 request timed out | Generation exceeded the 300-second limit | Stream or shorten the request, then retry with backoff |
404 model not found | Unknown model ID, or an ID listed in GET /models with no backend at the moment | Fix an unknown ID; retry a listed one with backoff |
400, 409, 413 or 422 inference request rejected | The backend rejected the request, for example a prompt longer than the model can read | Fix the request, don't retry it unchanged |
A 429 is the HTTP status for too many requests in a given amount of time. On our API it can also mean the model is under heavy load. Two hard limits make a request fail every time. One is a max_tokens value above 131,072. The other is a prompt longer than the model's context window, the most text it can read in one request.
Every response carries a trace ID in its headers: a 32-character code that identifies that one request. Quote it with the UTC timestamp when you contact us about a specific request, and our engineers can look it up.
Retry with backoff, at one layer, only what can pass
Retries help only for errors that a second try can fix, and only when they wait. Retry on 429, 502, 503 and 504 with backoff. Treat the 400, 409, 413 and 422 rejections as permanent for that request.
Exponential backoff means waiting longer after each failed try: for example 1 second, then 2, then 4. Jitter adds a random spread to each wait. The book Site Reliability Engineering says to always use randomized exponential backoff and to limit retries per request. The OpenAI Cookbook adds that random jitter keeps retries from all hitting at the same time.
Do not retry rejected or malformed requests. They fail the same way every time.
Retry at one layer only. If three layers each issue 3 retries, one user action can create 64 attempts. Each layer makes 4 attempts per call it receives:
(1 + 3) x (1 + 3) x (1 + 3) = 64 attemptsAn eval tool that retries on top of a client library that also retries is two layers. Check both, and switch one off.
import random, time
from openai import OpenAI, APIStatusError, APITimeoutError
client = OpenAI(api_key="lk_...", base_url="https://api.lyceum.technology/openai/v1",
timeout=600.0, max_retries=0) # one retry layer: this loop
RETRY = {429, 502, 503, 504}
def call(messages, model="z-ai/glm-5.3", tries=4):
for attempt in range(tries):
try:
return client.chat.completions.create(model=model, messages=messages)
except APIStatusError as e:
if e.status_code not in RETRY or attempt == tries - 1:
raise # 400/409/413/422: fix the request, never resend it unchanged
except APITimeoutError:
if attempt == tries - 1:
raise
time.sleep(2 ** attempt + random.random()) # 1, 2, 4 s plus jitterHow to run a large eval reliably on serverless inference
A large eval runs reliably on serverless inference when the client waits long enough and retries only what can pass. It also sends no more requests at once than the model can take. Work through the settings in this order.
- Set the timeout above our limit. Use more than 300 seconds, for example 600.
- Stream long answers. Raise any stream inactivity timeout too.
- Keep one retry layer. Retry 429, 502, 503 and 504 with backoff and jitter. Never resend a 400, 409, 413 or 422 rejection unchanged.
- Start with low concurrency. Raise it while capacity errors stay rare, and lower it when they climb.
- Log the error message for every failed prompt. Count failures per type before you rerun.
- Rerun only the failed prompts. A full rerun repeats work that already succeeded.
In lm-evaluation-harness, these settings go into the --model_args option. The model type local-chat-completions covers OpenAI-compatible chat APIs. The task gsm8k (school maths questions) stands in for yours. The flag --apply_chat_template sends each prompt as chat messages. The values are an example, not a recommendation for every workload.
lm_eval --model local-chat-completions \
--model_args model=z-ai/glm-5.3,base_url=https://api.lyceum.technology/openai/v1/chat/completions,num_concurrent=4,max_retries=3,timeout=600 \
--tasks gsm8k --apply_chat_templateFor the eval design itself, see our guide to evaluating an open-source model against your current one. For the wait before the first token, see what determines time to first token.
When to move the eval to a dedicated endpoint
Move the eval to a dedicated endpoint when capacity errors still block a usable result after these fixes. Move it too when it must show production latency. Our docs pick serverless for low-volume or spiky traffic, and dedicated for production traffic with predictable latency.
A dedicated endpoint runs copies of the model on GPUs reserved for you. We call each running copy a replica. You pay for the time the replicas run, not per request.
We don't publish a standard uptime SLA (service level agreement, a contract promise on availability) for serverless inference yet. For dedicated endpoints, we agree service levels in the contract. Our status page shows live status, per-model latency and past incidents. Check it against your failed requests, and see how to check a provider's reliability without an SLA.
An eval that serverless capacity could not finish is a good basis for a dedicated proof of concept.
Start with the cheapest fix. Rerun the failed part of your eval with a timeout above 300 seconds, streaming and one retry layer. Then compare the remaining errors against our error table.
Frequently asked questions
Why might I get a "429 Too Many Requests" error from the Lyceum API, and what should I do about it?
A 429 error means you've sent more requests than your current limit allows, or the model is under heavy load. Wait a moment and retry, ideally with exponential backoff. If it keeps happening, contact us and we'll look at your limits or a dedicated endpoint.
Should I lower concurrency or add retries first?
Fix the timeout first, because more retries cannot save an answer the client cuts off. Then lower concurrency if capacity errors remain, since every retry is another request while the model is busy.