If your agent works but users wait too long, or you can't say what one task costs, this guide is for you. It shows how to give each task a time limit and a cost limit, and how to stay inside them.

The short version: set both limits for the whole task, size them on the slow tasks rather than the average, and fix the tools and the step count before you change the model. That's where most of the time goes.

Our worked example is a hypothetical support agent, priced on our own serverless API, where each step is billed per token. The method works with any provider.

Budget the task, not the request

An agent is a program that lets a model call tools in a loop until a task is done. A step is one pass through that loop: a model call, a tool call or a search.

The user waits for the sum of the steps that run one after another, and you pay for all calls. No single request shows either number.

So set the budget at task level first, in the product owner's terms. How long does a user wait, and what may one task cost? Everything after that is allocation.

Anthropic's engineering guide says "agentic systems often trade latency and cost for better task performance."

Our running example is a hypothetical support agent that answers a question about an order. The product owner sets 30 seconds and $0.04 per task. The agent's steps fall into five types, each with its own time and cost profile:

Step typeWhere the time goesWhere the cost goes
PlanningModel reads the question, writes a planModel tokens
RetrievalEmbedding call and vector search (finding the closest stored texts)Few embedding tokens
Tool callsYour other services' response timeThose services' fees
GenerationModel reads tool results, writes the next stepModel tokens
ValidationA short check before the answer goes outFew tokens, small model

An embedding turns a text into a list of numbers, so a search can find passages with a similar meaning.

Measuring the step-count distribution first

In our example, a round is one pass of tool calls plus the model call that reads their results. That model call takes 1.5 seconds. We counted the rounds in 200 traced tasks (hypothetical numbers):

Rounds per taskShare of tasksShare with this many rounds or fewer
130%30%
235%65%
325%90%
47%97%
53%100%

The average is 2.18 rounds. The 95th percentile is the count that 95 of 100 tasks stay within: here 4 rounds.

Hypothetical counts from 200 traced agent tasks: 30% needed 1 round, 35% 2 rounds, 25% 3 rounds, 7% 4 rounds and 3% 5 rounds. A dashed line marks the average of 2.18 rounds. The 4-round column is highlighted as the 95th percentile, the count 95 of 100 tasks stay within.
In our hypothetical sample of 200 traced tasks, the average task needs 2.18 rounds, but the 95th percentile needs 4.

Google's SRE (site reliability engineering) book explains why to budget on the percentile. An average hides the slow tail. Its authors "prefer to work with percentiles rather than the mean."

One round calls the order API (1.5 s) and the shipping API (3.0 s), then the model (1.5 s). A hand-off is the work between two steps: formatting a result for the next step, and queueing. The example counts 0.2 seconds before each step: 4 fixed steps plus 3 per round. Four rounds break the limit:

fixed steps = 2.5 s plan + 0.4 s retrieval + 4.0 s answer + 0.8 s check = 7.7 s
one round   = 1.5 s + 3.0 s + 1.5 s = 6.0 s
hand-offs   = (4 + 3 × rounds) × 0.2 s
2 rounds    = 7.7 + 2 × 6.0 + 10 × 0.2 = 21.7 s
4 rounds    = 7.7 + 4 × 6.0 + 16 × 0.2 = 34.9 s

No published agent latency benchmark transfers between applications, so measure your own.

Allocating latency across step types

Allocation splits the budget by step type. For the 4-round task with sequential tools (hypothetical latencies, prices read 30 October 2026):

Step typeTimeCost
Planning (1 call)2.5 s$0.0041
Retrieval0.4 sunder $0.0001
Tool calls (4 rounds × 2 calls)18.0 snot counted
Generation in rounds (4 calls)6.0 s$0.0203
Final answer (1 call)4.0 s$0.0078
Validation (1 call)0.8 s$0.0003
Hand-offs between steps (16 × 0.2 s)3.2 snone
Total34.9 s$0.0326

A model call costs its tokens times the model's price:

round call (z-ai/glm-5.3) = 3,000 input × $1.40 / 1M + 200 output × $4.40 / 1M
                          = $0.0042 + $0.00088 = $0.00508
4 rounds                  = 4 × $0.00508 = $0.0203

Base each model step's time on measured history. Time to first token (TTFT) is how long the first part of an answer takes to arrive. Throughput is how many tokens per second the model then writes.

Our status page shows both per model. A monitor sends each model a short streaming request every 10 minutes and plots the results over up to 30 days. Read these sampled checks as history, not a promise.

Finding the steps that can run in parallel

In our example, the order API and the shipping API both take the order number from the plan. Neither reads the other's answer. Run them at the same time, and a round waits for the slower call only:

tools one after another = 1.5 s + 3.0 s + 2 hand-offs = 4.9 s per round
tools in parallel       = max(1.5 s, 3.0 s) + 1 hand-off = 3.2 s per round
saving            = 1.7 s per round × 4 rounds = 6.8 s
parallel round    = 3.2 s tools + 0.2 s hand-off + 1.5 s model = 4.9 s
4-round task      = 34.9 s − 6.8 s = 28.1 s

The slow task now fits the limit, at the same cost.

Anthropic describes this as "sectioning": breaking a task into independent subtasks that run in parallel. OpenAI's function-calling guide says a model "may choose to call multiple functions in a single turn", unless you switch parallel tool calls off. On our serverless inference, parallel tool calls work on every chat model.

The LLMCompiler paper measured the gain. On HotpotQA, a benchmark of questions that need facts from two documents, with gpt-3.5-turbo, a sequential agent took 7.12 seconds. LLMCompiler, which runs independent calls together, took 3.95 seconds, 1.80 times faster.

A call is independent when its arguments come from the plan or the user, not from another call's result. It must also change nothing another call in the same round reads, such as a write or a payment.

Setting a termination rule and a partial result

An agent without a ceiling has no upper limit on cost per task. Anthropic notes that agents commonly include "stopping conditions (such as a maximum number of iterations)". Make that limit a product requirement.

The first part of the rule is a maximum of 4 rounds, the 95th percentile from the table. The second is a partial result for the 3% of tasks that need more. That result states what was verified, what is missing and the user's next action.

Check the budget before each new round. A parallel round takes 4.9 seconds and costs $0.0051 in the example. Start it only if both still fit:

import time
ROUND_SECONDS = 4.9   # measured: 3.2 s tools + 0.2 s hand-off + 1.5 s model
ROUND_COST = 0.0051   # measured: tokens × price for one round
MAX_ROUNDS = 4        # 95th percentile of rounds per task
def may_start_round(started, spent, rounds, limit_s=30.0, limit_usd=0.04):
    elapsed = time.monotonic() - started
    return (rounds < MAX_ROUNDS
            and elapsed + ROUND_SECONDS <= limit_s
            and spent + ROUND_COST <= limit_usd)

Retries count against the same budget. When our API answers 429 (no free capacity), our docs say to retry with backoff. Backoff means waiting a little longer before each new attempt, and that wait comes out of the 30 seconds.

For the latency that remains, stream progress such as "checking your order". Streaming sends output as it is produced. It changes how long the wait feels, not how long the task takes.

The model is rarely the slowest step

In the allocation table, the model accounts for less than two fifths of the task's time. So measure before you optimise. Record a span, a timed record of one step (start, end, type, tokens), for every step and sort the spans by total time. If the model leads, see our guide to reducing LLM inference latency. Distance adds to every call too, as our comparison of EU versus US inference API latency shows.

Calculated example with hypothetical latencies for a 4-round support-agent task. With tools called one after another the task takes 34.9 seconds: model 13.3, tools 18.0, hand-offs 3.2, retrieval 0.4. With the two tool calls in parallel it takes 28.1 seconds: model 13.3, tools 12.0, hand-offs 2.4, retrieval 0.4. A dashed line marks the 30-second limit.
In our calculated example with hypothetical latencies, tools and hand-offs take most of the 4-round task, and parallel tools bring it from 34.9 to 28.1 seconds.

A malformed tool call costs a whole extra round, so check our list of open models with reliable function calling.

The example keeps each round at 3,000 input tokens. Real agents resend a growing history, as our EU agent infrastructure guide shows. So the example understates the model's time and cost, and it leaves out the tools' fees.

Checking the budget against measured runs

Confirm the budget with a pilot. Run real tasks and read each response's token counts. Our API reports them in the usage object. Its cached tokens field is the part of the input served from cache at a lower price:

resp = client.chat.completions.create(model="z-ai/glm-5.3", messages=messages)
print(resp.usage.prompt_tokens, resp.usage.completion_tokens)
print(resp.usage.prompt_tokens_details.cached_tokens)

We are Lyceum, a sovereign European AI inference provider for open-source models: built and run in Europe, with our serverless models in Paris and Finland. Prompts and outputs are processed, not stored, and never used for training.

Serverless inference means calling shared, pre-hosted models with no GPUs to run. Our serverless inference on pre-hosted models is billed per token with no base fee, so a task's measured calls convert directly into its cost. Prices of the example's models, read on 30 October 2026:

ModelStep in the exampleInput per 1M tokensOutput per 1M tokens
z-ai/glm-5.3Planning, rounds, final answer$1.40$4.40
z-ai/glm-5.3-flashValidation$0.20$0.50
qwen/qwen3-embedding-8bRetrieval$0.02input only

Before a backend carries an agent's budget, check three things for every step's model: its speed history, its per-token prices and its processing region. The status page holds the speed history. Our model pages list prices and region, "EU-hosted" for all three models here.

We recommend budgeting on the 95th percentile and running independent tools in parallel before you change models. In the example, that alone brought the slow task from 34.9 to 28.1 seconds. Measure your step-count distribution, allocate the budget across step types, then check each step's model on the status page before you commit.

Frequently asked questions

How do you budget latency for an AI agent?

Agree one time limit per task, subtract each step type's measured time, and keep the rest for retries.

Can agent steps run in parallel?

Yes, when no call needs another call's result. Calls that write data or take payments usually stay in order.

Sources

  1. anthropic.com
  2. sre.google
  3. platform.openai.com
  4. arxiv.org