Scaling LLM inference: diagnose rate limits, latency and cold starts

As traffic grows, an LLM feature often struggles first at the afternoon peak: answers start late, and the logs fill with 429 errors. A 429 is the HTTP status code for "too many requests".

Behind this can be three different problems: a capacity error, a slow first token or a paused deployment. Each has its own fix, and adding replicas solves only the capacity error. So find out which one you have before you change anything.

We follow one hypothetical support assistant. It streams answers into a chat widget. It sees 8 requests per second in the morning and 45 in the afternoon peak. Every input here is invented for the calculations below.

Separate slow requests from rejected requests

Slow requests and rejected requests are different failures, so count them separately. Google's SRE book, a guide to running reliable services, names four golden signals: latency, traffic, errors and saturation. Saturation means how full your capacity is, such as busy replicas.

A failed request often comes back fast. If you mix it into your latency numbers, the service looks faster than it feels to users. So track failed requests as their own series, next to the successful ones, as the book advises.

An average also hides the tail, the few requests that take much longer than the rest. The book's example: a service with a 100 ms average at 1,000 requests per second can still have 1% of its requests take 5 seconds. That is 10 slow requests every second. So track p95 and p99, not the mean.

You must collect the latency numbers yourself. Log the status code, the error message and the total time. Add the time until the first token arrives, called time to first token (TTFT).

Split the assistant's afternoon log into rejected requests and slow ones. Each kind has its own fix:

Locate the delay inside the request path

An LLM request spends its time in two visible phases: waiting for the first token, then generating the rest. Waiting includes queueing and processing the prompt, so it grows with load and prompt length. Generation grows with the number of output tokens.

If a person is waiting for the answer, stream it first: send each token as soon as it is produced. Our serverless inference docs recommend streaming for user-facing and long-form work. A serverless request may generate for up to 300 seconds, after which it returns a 504.

Check your client timeouts next. A timeout that fires before the response finishes is the most common cause of cut-off answers. Under heavy load, time to first token can spike, so raise the stream inactivity timeout too.

Then check our status page. It tests each model with one short streaming request every 10 minutes, so it shows spot checks, not how all customer requests are doing. Back to the assistant: say its slowest requests got slower only while waiting for the first token, and the speed after that stayed the same. Then the cause is queueing or long prompts. A network problem would slow down every phase, not just the wait. Our guide to what determines time to first token breaks the phase down.

Bound retries and timeouts before adding capacity

Every retry is another request, so unbounded retries add load exactly when capacity is short. Suppose every request fails and is retried three times. Then the 45 requests per second turn into 180: 135 of them are retries, hitting a model that is already full. Cap both the attempts and the total time.

We are Lyceum, a sovereign European inference provider. Our serverless inference runs open-source models in Paris and Finland and uses the same request format as OpenAI's API. You pay per token, with no base fee. Our docs say which errors to retry and which to fix first:

The 429 status is defined in RFC 6585, which says a server may add a Retry-After header telling the client how long to wait. The HTTP standard says the header can also accompany a 503 response. If the header is there, wait as long as it says. If not, use backoff, which waits longer after each failure. Add jitter, a random share of that wait, so clients do not all retry at once. To keep it short, the code below skips the Retry-After step.

The client below retries 429, 502 and 503, backs off with jitter and stops at a total deadline. It also retries a 504, but a 504 usually means the answer took too long to generate. Streaming or a shorter request is the real fix:

import os
import random
import time

from openai import OpenAI, APIStatusError

client = OpenAI(
    api_key=os.environ["LYCEUM_API_KEY"],
    base_url="https://api.lyceum.technology/openai/v1",
    max_retries=0,   # retries are handled below
    timeout=120.0,
)

RETRYABLE = {429, 502, 503, 504}


def complete(messages, deadline_s=20.0, max_attempts=4):
    start = time.monotonic()
    for attempt in range(max_attempts):
        try:
            return client.chat.completions.create(
                model="z-ai/glm-5.3",
                messages=messages,
            )
        except APIStatusError as err:
            trace = err.response.headers.get("traceparent", "")
            print(f"status={err.status_code} trace={trace}")
            if err.status_code not in RETRYABLE or attempt == max_attempts - 1:
                raise
            delay = min(8.0, 0.5 * 2 ** attempt) * random.uniform(0.5, 1.0)
            if time.monotonic() - start + delay > deadline_s:
                raise
            time.sleep(delay)

The 20-second deadline is hypothetical; set yours from the longest wait your users accept.

Log the trace ID with every failure. Every response carries it in two headers, x-cloud-trace-context and traceparent, and the client above prints traceparent. If you ask us about a specific request, send that ID and the time in UTC. If 429 errors continue after backoff, contact us and we will look at your limits or a dedicated endpoint.

Plan for paused deployments and replica limits

A dedicated endpoint runs one model on GPU replicas, copies of the model server that only you use. Our dedicated inference adds and removes replicas, but only between the min_replicas and max_replicas you set. Two targets decide when it scales: target_rps, the requests per second one replica should handle, and target_latency_p95_ms, the p95 latency you aim for. A stabilisation window, a pause after each scaling step, stops it from adding and removing replicas in rapid swings.

Check the defaults before you trust the autoscaler. In our CLI, --max-replicas defaults to 1, so a new deployment cannot scale out until you raise it. The other defaults are 10.0 requests per second per replica, a 5000 ms p95 target and a 300-second scale-down window.

lyceum infer deploy meta-llama/Llama-3.1-8B-Instruct \
  -g gpu.a100 \
  --min-replicas 1 \
  --max-replicas 5 \
  --target-rps 10 \
  --target-latency 5000 \
  --stabilisation 300

Calculated example: one replica handles 10 requests per second, so the peak of 45 needs ceil(45 / 10) = 5 replicas. With the default maximum of 1, 35 requests per second have no replica to serve them, and p95 climbs. The calculation assumes traffic arrives evenly. It ignores real latency and the time a new replica needs to start.

These settings tell the autoscaler what to aim for. They don't guarantee a latency.

A cold start is the slow first request after a quiet period. After one hour without requests, a dedicated deployment stops all its replicas (this is called scale to zero) and shows as paused. The next request resumes it to min_replicas, and that request waits while the replicas start. If your traffic has gaps longer than an hour, expect a slower first request in your p99. Our comparison of autoscaling approaches for GPU inference covers the mechanics.

Retest under load, then choose per-token or reserved capacity

Only trust a fix if it shows up in the same load test you ran before the change. Replay the same traffic before and after, with the same input and output sizes, request rate and simultaneous requests. Write the target first; for the assistant, a hypothetical target is a p95 time to first token under 2 seconds at the 45-request peak. Compare p95 and p99, total latency and the error rate per status code.

Then decide where the workload runs. Per-token billing follows your traffic, so quiet hours cost nothing. A dedicated endpoint is billed for the time its replicas run, not per request. The busier you keep it, the less each token costs. Our docs call it the right tool when you need predictable latency and control over the model and GPU.

We recommend paying per token while traffic varies and moving to a dedicated endpoint once traffic steadies. Our docs give no break-even figure, so ask us for a dedicated price and compare it with your per-token bill. The assistant, quiet at 8 requests per second with one afternoon peak, still belongs on per-token billing. Reliability counts too when you choose a provider. If one publishes no uptime guarantee, our guide to verifying uptime evidence shows how to check it.

If 429 errors or an unexplained tail survive the retest, bring your workload trace, latency target and error samples to our engineers.