If your agent works but users wait too long, or you can't say what one task costs, this guide is for you. It shows how to give each task a time limit and a cost limit, and how to stay inside them.
The short version: set both limits for the whole task, size them on the slow tasks rather than the average, and fix the tools and the step count before you change the model. That's where most of the time goes.
Our worked example is a hypothetical support agent, priced on our own serverless API, where each step is billed per token. The method works with any provider.
Budget the task, not the request
An agent is a program that lets a model call tools in a loop until a task is done. A step is one pass through that loop: a model call, a tool call or a search.
The user waits for the sum of the steps that run one after another, and you pay for all calls. No single request shows either number.
So set the budget at task level first, in the product owner's terms. How long does a user wait, and what may one task cost? Everything after that is allocation.
Anthropic's engineering guide says "agentic systems often trade latency and cost for better task performance."
Our running example is a hypothetical support agent that answers a question about an order. The product owner sets 30 seconds and $0.04 per task. The agent's steps fall into five types, each with its own time and cost profile:
| Step type | Where the time goes | Where the cost goes |
|---|---|---|
| Planning | Model reads the question, writes a plan | Model tokens |
| Retrieval | Embedding call and vector search (finding the closest stored texts) | Few embedding tokens |
| Tool calls | Your other services' response time | Those services' fees |
| Generation | Model reads tool results, writes the next step | Model tokens |
| Validation | A short check before the answer goes out | Few tokens, small model |
An embedding turns a text into a list of numbers, so a search can find passages with a similar meaning.
Measuring the step-count distribution first
In our example, a round is one pass of tool calls plus the model call that reads their results. That model call takes 1.5 seconds. We counted the rounds in 200 traced tasks (hypothetical numbers):
| Rounds per task | Share of tasks | Share with this many rounds or fewer |
|---|---|---|
| 1 | 30% | 30% |
| 2 | 35% | 65% |
| 3 | 25% | 90% |
| 4 | 7% | 97% |
| 5 | 3% | 100% |
The average is 2.18 rounds. The 95th percentile is the count that 95 of 100 tasks stay within: here 4 rounds.
Google's SRE (site reliability engineering) book explains why to budget on the percentile. An average hides the slow tail. Its authors "prefer to work with percentiles rather than the mean."
One round calls the order API (1.5 s) and the shipping API (3.0 s), then the model (1.5 s). A hand-off is the work between two steps: formatting a result for the next step, and queueing. The example counts 0.2 seconds before each step: 4 fixed steps plus 3 per round. Four rounds break the limit:
fixed steps = 2.5 s plan + 0.4 s retrieval + 4.0 s answer + 0.8 s check = 7.7 s
one round = 1.5 s + 3.0 s + 1.5 s = 6.0 s
hand-offs = (4 + 3 × rounds) × 0.2 s
2 rounds = 7.7 + 2 × 6.0 + 10 × 0.2 = 21.7 s
4 rounds = 7.7 + 4 × 6.0 + 16 × 0.2 = 34.9 s
No published agent latency benchmark transfers between applications, so measure your own.
Allocating latency across step types
Allocation splits the budget by step type. For the 4-round task with sequential tools (hypothetical latencies, prices read 30 October 2026):
| Step type | Time | Cost |
|---|---|---|
| Planning (1 call) | 2.5 s | $0.0041 |
| Retrieval | 0.4 s | under $0.0001 |
| Tool calls (4 rounds × 2 calls) | 18.0 s | not counted |
| Generation in rounds (4 calls) | 6.0 s | $0.0203 |
| Final answer (1 call) | 4.0 s | $0.0078 |
| Validation (1 call) | 0.8 s | $0.0003 |
| Hand-offs between steps (16 × 0.2 s) | 3.2 s | none |
| Total | 34.9 s | $0.0326 |
A model call costs its tokens times the model's price:
round call (z-ai/glm-5.3) = 3,000 input × $1.40 / 1M + 200 output × $4.40 / 1M
= $0.0042 + $0.00088 = $0.00508
4 rounds = 4 × $0.00508 = $0.0203
Base each model step's time on measured history. Time to first token (TTFT) is how long the first part of an answer takes to arrive. Throughput is how many tokens per second the model then writes.
Our status page shows both per model. A monitor sends each model a short streaming request every 10 minutes and plots the results over up to 30 days. Read these sampled checks as history, not a promise.
Finding the steps that can run in parallel
In our example, the order API and the shipping API both take the order number from the plan. Neither reads the other's answer. Run them at the same time, and a round waits for the slower call only:
tools one after another = 1.5 s + 3.0 s + 2 hand-offs = 4.9 s per round
tools in parallel = max(1.5 s, 3.0 s) + 1 hand-off = 3.2 s per round
saving = 1.7 s per round × 4 rounds = 6.8 s
parallel round = 3.2 s tools + 0.2 s hand-off + 1.5 s model = 4.9 s
4-round task = 34.9 s − 6.8 s = 28.1 s
The slow task now fits the limit, at the same cost.
Anthropic describes this as "sectioning": breaking a task into independent subtasks that run in parallel. OpenAI's function-calling guide says a model "may choose to call multiple functions in a single turn", unless you switch parallel tool calls off. On our serverless inference, parallel tool calls work on every chat model.
The LLMCompiler paper measured the gain. On HotpotQA, a benchmark of questions that need facts from two documents, with gpt-3.5-turbo, a sequential agent took 7.12 seconds. LLMCompiler, which runs independent calls together, took 3.95 seconds, 1.80 times faster.
A call is independent when its arguments come from the plan or the user, not from another call's result. It must also change nothing another call in the same round reads, such as a write or a payment.
Setting a termination rule and a partial result
An agent without a ceiling has no upper limit on cost per task. Anthropic notes that agents commonly include "stopping conditions (such as a maximum number of iterations)". Make that limit a product requirement.
The first part of the rule is a maximum of 4 rounds, the 95th percentile from the table. The second is a partial result for the 3% of tasks that need more. That result states what was verified, what is missing and the user's next action.
Check the budget before each new round. A parallel round takes 4.9 seconds and costs $0.0051 in the example. Start it only if both still fit:
import time
ROUND_SECONDS = 4.9 # measured: 3.2 s tools + 0.2 s hand-off + 1.5 s model
ROUND_COST = 0.0051 # measured: tokens × price for one round
MAX_ROUNDS = 4 # 95th percentile of rounds per task
def may_start_round(started, spent, rounds, limit_s=30.0, limit_usd=0.04):
elapsed = time.monotonic() - started
return (rounds < MAX_ROUNDS
and elapsed + ROUND_SECONDS <= limit_s
and spent + ROUND_COST <= limit_usd)
Retries count against the same budget. When our API answers 429 (no free capacity), our docs say to retry with backoff. Backoff means waiting a little longer before each new attempt, and that wait comes out of the 30 seconds.
For the latency that remains, stream progress such as "checking your order". Streaming sends output as it is produced. It changes how long the wait feels, not how long the task takes.
The model is rarely the slowest step
In the allocation table, the model accounts for less than two fifths of the task's time. So measure before you optimise. Record a span, a timed record of one step (start, end, type, tokens), for every step and sort the spans by total time. If the model leads, see our guide to reducing LLM inference latency. Distance adds to every call too, as our comparison of EU versus US inference API latency shows.
A malformed tool call costs a whole extra round, so check our list of open models with reliable function calling.
The example keeps each round at 3,000 input tokens. Real agents resend a growing history, as our EU agent infrastructure guide shows. So the example understates the model's time and cost, and it leaves out the tools' fees.
Checking the budget against measured runs
Confirm the budget with a pilot. Run real tasks and read each response's token counts. Our API reports them in the usage object. Its cached tokens field is the part of the input served from cache at a lower price:
resp = client.chat.completions.create(model="z-ai/glm-5.3", messages=messages)
print(resp.usage.prompt_tokens, resp.usage.completion_tokens)
print(resp.usage.prompt_tokens_details.cached_tokens)
We are Lyceum, a sovereign European AI inference provider for open-source models: built and run in Europe, with our serverless models in Paris and Finland. Prompts and outputs are processed, not stored, and never used for training.
Serverless inference means calling shared, pre-hosted models with no GPUs to run. Our serverless inference on pre-hosted models is billed per token with no base fee, so a task's measured calls convert directly into its cost. Prices of the example's models, read on 30 October 2026:
| Model | Step in the example | Input per 1M tokens | Output per 1M tokens |
|---|---|---|---|
| z-ai/glm-5.3 | Planning, rounds, final answer | $1.40 | $4.40 |
| z-ai/glm-5.3-flash | Validation | $0.20 | $0.50 |
| qwen/qwen3-embedding-8b | Retrieval | $0.02 | input only |
Before a backend carries an agent's budget, check three things for every step's model: its speed history, its per-token prices and its processing region. The status page holds the speed history. Our model pages list prices and region, "EU-hosted" for all three models here.
We recommend budgeting on the 95th percentile and running independent tools in parallel before you change models. In the example, that alone brought the slow task from 34.9 to 28.1 seconds. Measure your step-count distribution, allocate the budget across step types, then check each step's model on the status page before you commit.
Frequently asked questions
How do you budget latency for an AI agent?
Agree one time limit per task, subtract each step type's measured time, and keep the rest for retries.
Can agent steps run in parallel?
Yes, when no call needs another call's result. Calls that write data or take payments usually stay in order.