TL;DR On a shared API, an agent's next turn can land on a model copy that must reread its whole history. Our API already kept each conversation on one deployment. Now the router inside each deployment also picks a copy that holds the history, weighed against load. Since 1 October 2026, cached input on GLM-5.3 costs 41% less, input 20% less and output 2% less.

GLM-5.3 at a glance

Developer
Z.ai
Model ID
z-ai/glm-5.3
Context window
1M tokens
Max output
131,072 tokens

Price per million tokens

Input
$1.40
Cached input
$0.26
Output
$4.40
Billing
Billed per token. No base fee.
API
OpenAI-compatible
Runs in
European data centres in Paris and Finland
Serving engine
NVIDIA Dynamo with KV cache-aware routing
GLM-5.3 model page

A coding agent repeats itself. Every turn, it sends us the system prompt, the tool definitions and the conversation so far, with a small new step at the end. The model has read almost all of it before. Whether we read it again or reuse the last turn's work depends mostly on where the request lands.

Why a shared API reads the same prompt twice

When a model reads a prompt, it stores a compact record of every token in every attention layer: the KV cache (key-value cache). Building it is called prefill. If the agent's next request starts with exactly the same text, a copy of the model that still holds that cache only processes what is new. We call each copy a replica, and a group of replicas behind one router a deployment. The figures simplify and say GPU.

On a dedicated endpoint, one customer's requests go to the same few replicas, so reuse comes almost for free. A shared serverless API spreads requests across several replicas. Conventional balancers choose without knowing what each replica holds: round robin takes the next one in line, least-busy the shortest queue.

If turn 12 of a session lands on a replica that never saw turn 11, that replica prefills the history again. Take an example turn with 40,000 tokens of history, 2,000 new and 1,000 out: a miss prefills 42,000 tokens instead of 2,000, 21 times the work.

Back of the envelope at current prices, a hit costs 40,000 × $0.26/M + 2,000 × $1.40/M + 1,000 × $4.40/M = $0.0176. A miss bills all 42,000 tokens as new input, 42,000 × $1.40/M + 1,000 × $4.40/M = $0.0632, about 3.6 times as much. It also raises the time to first token, because the replica reads the whole history before it writes anything.

And the waste is hard to see: a replica recomputing a prefix just looks busy, so a least-busy balancer reads a cache miss as load.

Routing by prefix, weighed against load

Our serverless engine for GLM-5.3 builds on NVIDIA Dynamo, an open-source framework for serving models across many GPUs, and its KV cache-aware router, which tracks which prefixes each replica holds. Around it, we've added our own work to improve routing further. Our API recognises a conversation from its opening messages, keeps it on the same deployment and, when traffic peaks, reserves room for conversations already running there, so they don't lose their cache to newcomers.

But sending every request to its best match would pile traffic onto a few replicas while others sat idle. So Dynamo's router weighs how much prefill each replica would save against how busy it already is, and gives up some reuse when the best match is overloaded.

A prefix stays in GPU memory only until newer work pushes it out, so how long it lasts depends on load. In our own test calls, quick follow-ups almost always found the cache, and after a pause of one to three minutes it was hit or miss. Agents that call a tool and come straight back gain the most.

GLM-5.3 moved to this setup on 1 October, with the same model ID and endpoint.

What a simulation shows

To isolate the mechanism, we simulated both routing rules on the same made-up traffic.

The same 600 requests, routed two ways

Six coding agents on four GPUs, each with cache space for two conversations.

Simulated data

Round robin

32% of requests hit the cache

1.00× GPU time per request

Cache-aware

99% of requests hit the cache

0.34× GPU time per request

Source: Lyceum simulation, October 2026. The grids show 16 of the 600 steps; the percentages and GPU time cover all 600, with GPU time relative to round robin.

In the simulation, cache-aware routing keeps each conversation on one GPU, and the only misses are each agent's first request.

The simulation favours cache-aware routing in two ways. There is cache space for every conversation at once, so nothing is evicted. And the simulated router always picks a GPU that holds the conversation, however busy, so it never makes the trade-off above. That leaves 2 GPUs serving about twice as many requests as the other 2. In one respect it undersells the effect, because every miss counts the same, while in production a miss costs more the longer the prefix.

From GPU time to price

Less repeated prefill means less GPU time per request. Finding improvements that translate to cost savings in production took weeks of work, and those savings are what let us lower GLM-5.3's prices. Cached input fell the most.

What GLM-5.3 costs

US dollars per million tokens, excluding VAT. Before and after 1 October 2026.

GLM-5.3 prices in US dollars per million tokens, before and after 1 October 2026
Token typeBeforeNowChange
Input$1.75$1.40−20%
Cached input$0.44$0.26−41%
Output$4.50$4.40−2%

Source: Lyceum price list, 1 October 2026.

Each workload's saving is a blend of the three cuts, weighted by what it spends on each line, not by how much of its prompt repeats. In the example coding agent turn, the saving is $0.0080 ($0.0256 before, $0.0176 now), and resent context, the largest line, carries $0.0072 of it. A chat turn also reuses most of its input but spends most on output, which barely moved, so it saves the least. A retrieval-augmented generation (RAG) request is mostly new context, so its saving sits close to the input cut.

What one request costs

Three example workloads, before and now. Lower is better.

Example data
Cost of one request in US dollars, excluding VAT, before and after 1 October 2026, for three example workloads
WorkloadCached input tokensNew input tokensOutput tokensBeforeNowSaving
Chat2,000400500$0.0038$0.003314% less
Coding agent40,0002,0001,000$0.0256$0.017631% less
RAG1,0008,000600$0.0171$0.014118% less

Source: Lyceum price list, 1 October 2026. Token counts are examples, not measurements.

What we learned

In model serving, load balancing is a cache decision, and clients can help make it. If you build agents, keep the start of each prompt stable. Put the system prompt and tool definitions first, append new turns at the end, and keep changing values such as timestamps out of the early text. To see how well this works for your own traffic, read the usage in each response: usage.prompt_tokens_details.cached_tokens counts the tokens that hit the cache, and only those are billed at the cached rate.

The open problem is the case our simulation leaves out: more live conversations than cache space.

What comes next

For that case, we're working on running one copy of a model across several nodes, which leaves each GPU more memory for cached history. We plan to bring it to production soon. We're also working on much faster model loading, so a new copy can start serving sooner when traffic rises.

Thanks to the inference team, and to the maintainers of NVIDIA Dynamo, whose open-source work we build on.

Run GLM-5.3 on Lyceum

Model ID z-ai/glm-5.3. $1.40 per million input tokens and $0.26 for cached input. OpenAI-compatible, billed per token with no base fee.

app.py · Install openai and set LYCEUM_API_KEY

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LYCEUM_API_KEY"],
    base_url="https://api.lyceum.technology/openai/v1",
)

response = client.chat.completions.create(
    model="z-ai/glm-5.3",
    messages=[{"role": "user", "content": "Hello!"}],
)

print(response.choices[0].message.content)