EU compliant AI agent infrastructure has three parts. The model runs in European data centres and does not store prompts. You plan memory and cost for a context that grows at every step, and you keep a log of every step yourself.
A provider that stores no prompts cannot hand you an audit trail. Your app has to write it.
So you are probably asking: which parts of my agent stack must change before an EU customer signs, and what does a task cost?
Why agents change the infrastructure question
An agent is a program that lets a model call tools in a loop until a task is done. The ReAct paper describes this as alternating reasoning and action. Each pass through that loop is one model call.
The model does not remember earlier calls. Your app sends the full history every time: instructions, tool definitions, tool results and earlier answers. The context window is the amount of text the model can read in one call. The agent fills it step by step.
Take a hypothetical eight-step task. It starts with 3,000 tokens of instructions and tool definitions. Each step adds 1,500 tokens of tool results and model output, and the model writes 300 tokens per step.
input at step k = 3,000 + 1,500 × (k − 1)
steps 1 to 8 = 3,000, 4,500, 6,000, 7,500, 9,000, 10,500, 12,000, 13,500
total input = 66,000 tokens
total output = 8 × 300 = 2,400 tokensA single call sends the 3,000-token start plus the 1,500 tokens one step adds: 4,500 tokens. The agent sends 66,000, and 66,000 ÷ 4,500 = 14.7 times as much. Cost, memory and logging all grow with it. The multi-step agent budget guide covers latency for the same pattern.
What an EU security review asks about an agent
A customer's security team asks three questions: where the data is processed, who can see it, and what is kept.
We run open models in Europe. Our serverless models run in Paris and Finland, and our GPUs run in European data centres. Each model page shows a processing region, so check it before you send data.
Our legal page shows what happens to a request at each stage. Our gateway is the server that checks your API key and passes the request to a GPU. It sees your prompt in memory, but we never log it or write it to a database. The GPU holds your prompt in memory while the model writes the answer.
After a request, we keep one record: your organisation and user, the model, token counts and the time. It never contains the prompt or the response. The only trace left is a short-term cache of recent prompt starts in GPU memory, which newer requests replace.
The DPA (data processing agreement) is part of our terms. The subprocessor list, which names the companies that process data on our behalf, is there too. Our ISO 27001 audit is in progress; ISO 27001 is the international standard for running information security.
Your conversation history stays in your app.
Agent logs you must keep under the AI Act
The AI Act requires high-risk AI systems to allow automatic recording of events over their lifetime. Deployers must keep those logs for at least six months. That applies to the logs they can access and keep, such as the ones your own app writes.
A high-risk AI system is one that can pose serious risks to health, safety or fundamental rights, such as CV sorting or exam scoring. The European Commission says these use cases (listed in the Act's Annex III) face obligations from 2 December 2027. AI built into regulated products, such as safety components (Annex I), follows on 2 August 2028. The obligations listed include "logging of activity to ensure traceability of results".
If your agent is high-risk, the log is your job, because we keep no prompt text. Enterprise reviewers ask for one either way.
Log each step, not only the final answer:
| Field | Why you need it |
|---|---|
| Task id and step number | Rebuilds the order of decisions |
| Model id and processing region | Shows where each call ran |
| Tool called and its arguments | Shows what the agent did, not only what it said |
| Input, cached and output tokens | Matches your bill and spots runaway loops |
| Time to first token and total time | Finds slow steps |
| Retention date | Proves the six-month minimum, or your own policy |
Time to first token (TTFT) is how long the model takes to send the first word of its answer.
Failing agents rarely crash. They loop, calling the same tool again, and a hard step limit catches that.
NIST, the US standards institute, released its AI Risk Management Framework on 26 January 2023 for voluntary use. Our EU AI Act infrastructure guide covers the duties beyond logging.
GPU memory for long agent contexts
The KV cache (key-value cache) is the record the model keeps of every token it has read in a request. It lives in GPU memory, and it grows with the context. Our KV cache guide shows how to calculate its size.
In our example, the eighth call carries 13,500 tokens. Ten agents at once need memory for ten growing contexts, and self-hosted stacks run out there.
The PagedAttention paper names the cause. KV cache memory "grows and shrinks dynamically", and badly managed memory is wasted by fragmentation. Fragmentation means free memory is split into gaps too small for the next request. The fix stores the cache in small fixed-size blocks that can sit anywhere in memory, as an operating system does with pages. A growing request takes the next free block, so no large gap is needed. Throughput is the number of tokens a GPU produces per second. The authors report 2 to 4 times higher throughput.
Two failures recur in self-hosted agents:
- The growing-context crash. A supervisor agent, which hands tasks to worker agents and collects their results, runs out of memory after some minutes. Prefix caching helps: the engine reuses the stored cache for the unchanged start of the prompt, so only new tokens are processed.
- The waiting queue. Parallel worker agents slow down while the GPU looks half idle. Without continuous batching, the server handles requests one after another. Continuous batching is the fix: the server adds new requests to the running batch as slots free up.
On our serverless API you do not size this memory. You pay per token instead.
What an eight-step agent costs per task
Take Kimi K3, an open model with a 1M-token context. On its model page, read 30 October 2026, input costs $3.00 per million tokens. Cached input costs $0.75 and output $15.00 per million tokens. Prompt starts are the first tokens of a prompt that repeat across calls. We bill those still in GPU memory at the cached input price, as our inference page states.
In our example, every step starts with the full prompt of the step before. If those prefixes are still cached, each step after the first pays the cached price for the previous step's input. That is 3,000 + 4,500 + … + 12,000 = 52,500 cached tokens. The remaining 66,000 − 52,500 = 13,500 tokens pay the full input price.
| Calculated example (hypothetical task) | Without cache | With cache |
|---|---|---|
| Input tokens at $3.00 per 1M | 66,000 | 13,500 |
| Cached input tokens at $0.75 per 1M | 0 | 52,500 |
| Output tokens at $15.00 per 1M | 2,400 | 2,400 |
| Cost per task | $0.234 | $0.116 |
Keeping the start of the prompt identical halves the cost of this task. Put fixed instructions and tool definitions first and append new results at the end.
This example favours the cache. It assumes every prefix stays in GPU memory, which is not promised, and it ignores retries. Your usage record shows the real cached share for each request.
Serverless API or dedicated endpoint for agents
Start on serverless inference: open models through one OpenAI-compatible API, billed per token with no base fee. Quiet hours cost nothing.
from openai import OpenAI
client = OpenAI(base_url="https://api.lyceum.technology/openai/v1", api_key="YOUR_LYCEUM_API_KEY")
response = client.chat.completions.create(
model="moonshotai/kimi-k3",
messages=[{"role": "user", "content": "Summarise the open tickets for account 4711."}],
)
print(response.usage)Move to a dedicated endpoint when traffic is steady all day. A replica is one copy of the model on GPUs reserved for you. A dedicated endpoint runs your model on one or more replicas. It is billed for replica time. The price is agreed per contract for capacity you commit to in advance. The data centre is agreed for the workload, and you can ask for a specific EU location.
If your agents run around the clock, price a dedicated endpoint against your token bill. Otherwise stay on per-token billing. Either way, check availability on our status page.
To try it, point your agent framework at our OpenAI-compatible API and log the usage fields from the first call.
Frequently asked questions
Does a provider in Europe make my agent compliant with the AI Act?
No. Where the model runs is one input. If your system is high-risk, the logging, documentation and oversight duties sit with you. The AI Act calls the company that builds the system the provider and the one that uses it the deployer.
Do you train on the prompts my agents send?
No. Prompts and outputs are processed, not stored, and never used for training.