Good average latency, slow outliers: fixing P95 on an LLM API

Our average time to first token (the wait before the reply starts) is good, but some chats still wait a long time at peak. What causes the slow chats?

Why a good average hides slow P95 requests

Time to first token (TTFT) is the wait from sending a request until the first piece of the answer arrives. In a chat, it is the pause before the reply starts to appear.

An average mixes fast and slow requests into one number. Google's SRE book points out that most requests can be fast while a long tail of requests is much, much slower. The tail is the slowest few requests. A percentile describes the tail: P95 marks where the slowest 5 in 100 requests begin, and P50 is the median. A high percentile shows a plausible worst case, and the median shows the typical case.

Here is a hypothetical support chat widget, with numbers made up to show the method. We time 20 requests at the afternoon peak.

The average looks acceptable. Three of the 20 requests take longer than 1 s. Together they account for 61% of the waiting time: 1.10 + 4.80 + 7.20 = 13.10 s of 21.4 s. Twenty requests are far too few for a real P99. A real run needs hundreds.

Bursts make the tail worse. The SRE book gives an example. Take a system that serves 200 requests a second in even-numbered seconds and none in the others. It averages 100 a second, like a system serving a constant 100. In its busy seconds it handles 200, twice the average. Long-tail behaviour gets worse at high load through queuing. It also gets harder to keep the tail low as utilisation, how busy the servers are, rises.

On our side, dedicated endpoints run on GPUs reserved for you. They autoscale their replicas, the identical copies of the model server, on a P95 target that you set. The rest of this article shows how to pick that target and the replica count.

Measure P95 time to first token in your client

Measure where your user waits, which is your own client. Latency and error metrics are not shown in our dashboard, whose serverless usage page shows tokens and cost. Our status page shows sampled model checks: one short streaming request per model every 10 minutes. These are not the success rate of all customer requests, and its performance lines are averaged. Use it for context, not for your P95.

Lyceum is a sovereign European AI inference provider for open-source models. Our serverless models run in Paris and Finland, and prompts and outputs are processed, not stored, and never used for training.

Stream the response, because streaming lowers time to first token. The script below calls the serverless endpoint with `qwen/qwen3.8-27b-instant`. It is the same model as Qwen3.8 27B, answering without a reasoning phase, so the first token it times is the first answer token. It sets `max_retries=0` so that a 429 is counted instead of retried silently.

import time
from concurrent.futures import ThreadPoolExecutor
from statistics import quantiles
from openai import OpenAI, APIStatusError

client = OpenAI(
    api_key="lk_...",
    base_url="https://api.lyceum.technology/openai/v1",
    max_retries=0,
)

def first_token(_):
    start = time.perf_counter()
    try:
        stream = client.chat.completions.create(
            model="qwen/qwen3.8-27b-instant",
            messages=[{"role": "user", "content": "Where is my order?"}],
            stream=True,
        )
        for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content:
                return "ok", time.perf_counter() - start
    except APIStatusError as err:
        return "error", err.status_code
    return "error", 0

with ThreadPoolExecutor(max_workers=16) as pool:
    results = list(pool.map(first_token, range(200)))

times = [value for kind, value in results if kind == "ok"]
errors = [value for kind, value in results if kind == "error"]
cuts = quantiles(times, n=100)
print(f"P50 {cuts[49]:.2f} s, P95 {cuts[94]:.2f} s, P99 {cuts[98]:.2f} s")
print(f"429 answers: {errors.count(429)} of {len(results)}")

Run it at the hour your users complain, with a prompt mix close to production. Keep the 429 count apart from the times, because a refused request has no first token.

Match the slow tail to its cause

Four causes produce most slow outliers. The table pairs each with what you see and the fix.

Queueing comes first. In bursty production traffic, queueing delay is frequently the largest single contributor to tail TTFT spikes. We explain why in what determines time to first token. Queueing delay is the time a request waits in the scheduling backlog.

A prefix is the start of a prompt. Prefill is the step in which the server reads your whole prompt, which it must finish before it writes the first token. When a new request shares its prefix with an earlier one, the server reuses the stored result and skips the shared part. The same documentation notes the limit: caching shortens prefill and not generation. On our serverless inference caching is automatic, with nothing to enable. It is also best-effort, so a request may or may not hit a warm cache. Check `usage.prompt_tokens_details.cached_tokens` in the response, and read it defensively because the field is absent on a miss.

A 429 on serverless has two meanings. Either we have no free capacity for that model right now, or the server running the model returned 429 to us. Retry both with backoff.

A cold start applies to dedicated endpoints. A deployment that receives no requests for 1 hour is scaled down to zero replicas, so billing stops, and is marked paused. The first request after that has higher latency while the replicas start.

Find the peak concurrency that meets your P95 target

Concurrency is the number of requests in flight at the same moment. A benchmark that raises concurrency step by step shows where P95 breaks your target.

The `vllm bench serve` tool does this for any OpenAI-compatible endpoint. Its `--max-concurrency` caps requests in flight, and a `--burstiness` below 1 makes arrivals bursty, but only together with a `--request-rate`. The `--percentile-metrics ttft` and `--metric-percentiles` options report the percentiles you want.

vllm bench serve \
  --backend openai-chat \
  --base-url https://api.lyceum.technology \
  --endpoint /api/v2/external/v1/chat/completions \
  --header "Authorization=Bearer $LYCEUM_API_KEY" \
  --model <deployment_id> \
  --request-rate 8 --burstiness 0.5 \
  --num-prompts 200 \
  --max-concurrency 16 \
  --percentile-metrics ttft \
  --metric-percentiles 50,95,99

Add your own prompt dataset with `--dataset-name` and `--dataset-path`, and check every flag against your installed vLLM version. Run the command once per step.

Back to the support widget. The benchmark steps below are hypothetical, with a P95 target of 1.0 s.

The highest step that meets the target is 16 concurrent requests per replica. Now size the fleet from it. In the formulas below, 90 is the hypothetical number of chats active at the busiest moment. The 8 s is the assumed time one request stays in flight, from sending to the last token.

replicas       = ceil(peak concurrency / best step) = ceil(90 / 16) = 6
target_rps     = best step / seconds per request    = 16 / 8      = 2.0
check: peak requests per second = 90 / 8 = 11.25, so 6 replicas carry 1.88 each, below the 2.0 target_rps

The example shows the method and nothing more. The numbers are invented. A real benchmark depends on your prompt and output lengths, where the benchmark client runs and the model. The benchmark steps assume your production traffic mix, the 8 s is a guess, and cold starts are not part of the test.

Hold the target with a dedicated endpoint

A dedicated deployment runs a model on reserved GPU replicas behind an OpenAI-compatible endpoint. You pay for the time the replicas run, not per request. Dedicated replicas are reserved for you, so other customers' traffic does not queue ahead of yours, which suits predictable latency.

A common path is to run serverless open models billed per token and to move to dedicated GPUs as the workload grows.

The replica count autoscales between `min_replicas` and `max_replicas`. `target_rps` is requests per second per replica, and `target_latency_p95_ms` is the target P95 in milliseconds. Together they define when the deployment scales up or down, and a stabilisation window, a wait before scaling down, prevents flapping, meaning rapid up-and-down scaling. Load balancing across replicas is handled for you. With the example numbers, the deploy command looks like this:

lyceum infer deploy <hugging-face-model-id> \
  -g gpu.a100 \
  --min-replicas 2 \
  --max-replicas 6 \
  --target-rps 2.0 \
  --target-latency 1000

The six replicas come from the sizing above, and two stay warm so the first request is not a cold start. The documentation calls the target "p95 latency in milliseconds" and does not say whether that is time to first token or whole-request latency. Confirm the effect with the client script above, and do not assume the two numbers match.

Call the deployment at `/api/v2/external/v1/chat/completions` and set `model` to its `deployment_id`. For dedicated inference we also run time-boxed proofs of concept with success criteria that we agree on before you commit.

Mistakes that bring the slow tail back

• Sizing from the average. Your replicas then fit the quiet hours, and the peak queues. Use the highest passing benchmark step.
• Benchmarking a calm traffic shape. A steady request rate hides the bursts that cause the queue. Set `--burstiness` below 1.
• Putting changing text first in the prompt. A timestamp or user name near the top changes the prefix, so nothing after it is reused. Keep the stable text at the front.
• Retrying without backoff. A 429 means the model is busy, so immediate retries add load. Wait and retry with exponential backoff.
• Judging from a 20-request sample. A sample that small can only show the method. Use hundreds of requests per step.

Benchmark your peak concurrency, then deploy a dedicated endpoint with a P95 target. If you are still choosing between billing models, read pay per token vs dedicated.