Our chat pauses before the first word appears. What decides how long that pause is, and what is a normal value? The question comes up when a feature feels slow although the stream itself reads fine.

What time to first token actually measures

NVIDIA's benchmarking guide defines time to first token as the time it takes to process the prompt and generate the first token. In other words, it is how long a user waits before seeing any output. The same guide says TTFT generally includes queuing time, prefill time and network latency, and grows with prompt length.

Written as a sum, the wait has four parts:

TTFT = network round trip + queue wait + prefill + first output token

network round trip  your request travelling to the API and the first chunk travelling back
queue wait          time your request waits for a free slot on the GPU
prefill             the model reading every prompt token before it writes anything
first output token  the first step of generation

end-to-end latency = TTFT + generation time
Flow diagram of one streamed request in six stages: network, queue, prefill and first token sit inside a bracket labelled time to first token. Thinking, for reasoning models only, and the first answer word follow outside it.
Time to first token covers network, queue, prefill and the first token. A reasoning model then thinks before its first answer word.

Prefill is usually the largest part you control. During prefill the model builds its KV cache (key-value cache), the record it keeps of every prompt token it has read. The model has to read the whole input before it can write, so a longer prompt means a longer prefill.

TTFT is not the speed of the stream. Inter-token latency, the average gap between tokens after the first one, measures that. Some streams open with an empty message that carries no text. Benchmark tools ignore it, so the TTFT clock stops only at the first real token.

To show what each part costs, we made 20 test calls on our API on 9 November 2026, at 10:10 in the morning, Central European Time. We tested four cases: a short prompt, the same prompt with reasoning on, a long prompt, and the long prompt again. Each case ran five times from one client, so the results show roughly how much each part adds, not a benchmark.

Typical TTFT values we measure on our API

Our status page runs completion checks across tracked models every 10 minutes. It charts each model's TTFT and throughput (output tokens per second). The table reads its 7-day series from 2 to 9 November 2026.

ModelModel IDMedian TTFTTTFT range (10th to 90th percentile)Median output tokens/s
Qwen3.8 27Bqwen/qwen3.8-27b363 ms239 to 1,615 ms59
GLM-5.3z-ai/glm-5.3434 ms296 to 1,788 ms79
DeepSeek V4.1 Flashdeepseek/deepseek-v4.1-flash441 ms252 to 1,338 ms88
GLM-5.3 Flashz-ai/glm-5.3-flash710 ms321 to 1,556 ms52
Kimi K2.7 Codemoonshotai/kimi-k2.7-code778 ms308 to 4,988 ms43
Kimi K3moonshotai/kimi-k31,768 ms403 to 5,689 ms50
Qwen3.8 2.4T A95Bqwen/qwen3.8-2.4t-a95b2,324 ms424 to 8,483 ms98
MiniMax M3minimax/minimax-m32,354 ms664 to 7,424 ms85

The probe sends a short instruction to list the integers from 1 to 200, with max_tokens 600 and temperature 0. It streams one request per model and checks warm models. Its clock stops at the first token of any kind, including a thinking token from a reasoning model. The time includes the network round trip, as our benchmark method describes. The 7-day view keeps about one reading every three hours, 55 to 57 per model. These are values for a short prompt, not a promise under your load.

Three of the eight models start in under 500 ms at the median. Usability research puts 0.1 second as the limit for a reaction that feels instant. It puts 1.0 second as the limit for keeping a user's flow of thought.

The ranking barely follows streaming speed. Qwen3.8 2.4T A95B streams fastest at 98 tokens per second but needs 2,324 ms to start. Qwen3.8 27B starts first at 363 ms and streams at 59. Sorting the table by start time and by streaming speed gives two different orders. Their rank correlation is 0.14, and a value near 0 means the two orders are almost unrelated.

Every model in the table runs on our serverless inference: open models through one OpenAI-compatible API, billed per token with no base fee. Lyceum is a sovereign European AI inference provider for open-source models, and our serverless models run in Paris and Finland. Our models page lists the current IDs and prices.

How prompt length and prefix caching change TTFT

A long prompt can make a fast model start slowly. In our test calls, a short question to GLM-5.3 Instant got its first token after 0.50 s at the median. Compare the steps between our cases, not the absolute values, because our client and prompt differ from the probe's. The same question behind a 20,301-token prompt took 2.27 s.

reading about 20,000 extra prompt tokens:  2.27 s - 0.50 s = 1.77 s

Prefix caching removes much of that cost when requests share a start. A prefix is the opening part of a prompt, such as a system prompt or a shared document. With prefix caching, a new request that starts like an earlier one reuses its KV cache and skips the shared part. A cache hit means the server still holds that record. A cache miss means it reads the whole prompt again.

We sent the same long prompt a second time. In four of five calls, the response reported 20,224 tokens served from cache. One call reported none and took 4.62 s, so it read the whole prompt again. Across all five, the median TTFT fell to 1.25 s, about 45% lower. Caching shortens prefill only. It does not make generation faster.

Horizontal bar chart of median time to first token for GLM-5.3 Instant on our API, five calls each, 9 November 2026. A short question started in 0.50 seconds. The same question behind a 20,301-token prompt took 2.27 seconds. Sent again, the long prompt hit the cache in 4 of 5 calls and started in 1.25 seconds, about 45% faster, highlighted.
In our test calls on 9 November 2026, a cache hit cut the median time to first token of a 20,301-token prompt from 2.27 s to 1.25 s on GLM-5.3 Instant.

On our API, prompt caching (our name for prefix caching) is automatic. A hit is not guaranteed, as our missed call shows. Keep the stable part of your prompt at the front and vary only the end. The field usage.prompt_tokens_details.cached_tokens shows how much came from cache, as our serverless API documentation describes. Cached tokens are also cheaper: models that list a cached input price bill those tokens at it, as our inference page states.

For GLM-5.3 we send each conversation turn to the running copy of the model that holds its history. More turns then hit the cache. Our post on cache-aware routing explains how. For long prompts that rarely repeat, read about separating prefill and decode, where decode means generating the reply.

Why reasoning models delay the first answer word

A reasoning model writes thinking text before its answer, and that text counts toward output tokens. In our test, GLM-5.3 with reasoning on sent its first token after 0.48 s at the median. That token was thinking. The first answer word came after 0.98 s, following 74 reasoning tokens in four of five calls. GLM-5.3 Instant answered the same question in 0.50 s.

In the status table, the probe stops its clock at the first thinking token. The reasoning delay therefore never shows there. If your interface shows only the answer, time the first answer token yourself.

Our API gives you two switches. Setting reasoning_effort to "none" turns reasoning off on every model that supports switching. The -instant model IDs are the same model with reasoning switched off. Some models always reason, and the serverless documentation lists which. If a reasoning model returns a blank reply, our guide on empty responses from reasoning models covers the usual cause.

Queueing and cold starts: TTFT under load

Queueing is the part of TTFT a single test call cannot show. A replica is one running copy of the model server. It works on a limited group of requests at once, called a batch. When more requests are in flight than all replicas can batch, some wait in a queue and TTFT rises.

A scheduler must start new requests quickly while keeping the token rate steady for running ones. Schedulers that always put running requests first give them GPU time in every step, so new requests wait longer for their prefill. A research scheduler called FairBatching shares GPU time more evenly. It cut the TTFT of the slowest requests by up to 2.29x and kept the stream speed on target.

Busy hours on serverless

The probe behind our status page shows the same effect across a day. We split its 2 to 9 November readings into three bands of Central European Time. Between 09:00 and 18:00, the median TTFT was higher than between midnight and 09:00 for all eight models. GLM-5.3 took 663 ms by day and 349 ms at night. DeepSeek V4.1 Flash took 830 ms against 327 ms.

The probe sends the same prompt every time, so the prompt does not explain the gap. Traffic on the service is the likely cause. Each band holds only about 20 readings, so read it as a pattern, not as a precise figure. Our test calls also ran in the morning, which fits their slower short-prompt result.

Single calls vary even inside one band. One of our five long-prompt calls took 9.26 s against a median of 2.27 s. Judge TTFT on medians and high percentiles, never on one call. Our guide to slow outliers on an LLM API covers how to track them. Under heavy load TTFT can spike. Some streaming frameworks abort a stream when no token arrives for a set time, so raise that timeout as well as the request timeout.

When you need the same start time all day

A dedicated endpoint runs a model on GPU replicas reserved for you. Our dedicated endpoint documentation recommends it for production traffic that needs predictable latency. You set a scaling target in requests per second and in p95 latency, the time 95% of requests beat. The replica count then scales between a minimum and a maximum you choose. On our inference page, the reserved GPUs are planned around your production workload before launch.

Dedicated has its own cold start. A cold start is the first request after a model has to be loaded into GPU memory. NVIDIA notes that loading delays like this hurt user experience and scalability. In NVIDIA's test, its open-source loader, Run:ai Model Streamer, loaded a 15 GB model in 7.53 seconds on fast SSD storage.

Our probe measures warm models only. A dedicated endpoint with no requests for 1 hour shuts down all its replicas and pauses. The first request after a pause has higher latency while the replicas start, as the same documentation states. Our article on serverless cold starts goes deeper.

How to measure and lower your TTFT

Measure TTFT in your own client with your own prompt, because that is where your users wait. Stream the request: on our API, stream set to true is the recommended mode for anything user-facing and lowers time to first token. The script below times the first chunk of any kind and the first answer text separately.

import os, time
from openai import OpenAI

client = OpenAI(base_url="https://api.lyceum.technology/openai/v1",
                api_key=os.environ["LYCEUM_API_KEY"])

start = time.perf_counter()
first_any = first_answer = None
stream = client.chat.completions.create(
    model="z-ai/glm-5.3",
    messages=[{"role": "user", "content": "What is 17 times 23? Answer with the number only."}],
    stream=True,
)
for chunk in stream:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta
    thinking = getattr(delta, "reasoning_content", None) or getattr(delta, "reasoning", None)
    now = time.perf_counter() - start
    if first_any is None and (thinking or delta.content):
        first_any = now
    if first_answer is None and delta.content and delta.content.strip():
        first_answer = now

print(f"TTFT {first_any:.2f} s, first answer word {first_answer:.2f} s")

Run it a few dozen times, at the hours your users are active, and read the median. A self-hosted vLLM server lets you split the sum further. Its metrics include histograms of queue time, prefill time and time to first token, plus prefix cache hits counted in tokens.

Then pull the levers from largest to smallest:

  1. Put the stable part of the prompt first, so repeated requests hit the cache.
  2. Cut prompt tokens the model does not need, because every uncached token adds prefill.
  3. Switch reasoning off, or use an -instant model, where the reply needs no thinking.
  4. Stream every user-facing request.
  5. Only then compare models on measured TTFT for your prompt.

Agents call the model many times per task, so each saving repeats in latency in tool-calling loops. For changes on the serving side, see our guide to reducing inference latency on GPUs.

If your first answer word arrives within about a second on your real prompt at your busiest hour, the pause is not your bottleneck. If it does not, time your prompt against our serverless API with the script above, once with a new prefix and once cached.

Frequently asked questions

What is a good time to first token?

For a chat reply, aim for the first answer word within about one second on your real prompt. When the model reasons, measure the first answer word rather than the first token. Long prompts that miss the cache are the usual reason a target slips.

What is the difference between TTFT and inter-token latency?

TTFT is the wait until the first token arrives. Inter-token latency is the average gap between the tokens that follow, so it describes how fast the stream reads once it has started.

Does streaming reduce time to first token?

Yes. Without streaming, the client receives nothing until the whole reply is finished, so the first word waits for the last. With streaming, tokens arrive as they are generated.

Sources

  1. developer.nvidia.com
  2. nngroup.com
  3. docs.vllm.ai
  4. arxiv.org
  5. developer.nvidia.com
  6. docs.vllm.ai