An AI gateway, the proxy between your app and its model providers, can send overflow to a backup without breaking EU data rules. The backup needs what your main provider has: a data processing agreement (DPA), your written approval and EU processing if you promised it. Only capacity errors, where a provider has no free slot, should move a request there.
A fallback that fires rarely still hands over whole prompts. In our hypothetical example below, 0.5% of requests reach the backup, and that is 30,000 prompts a month.
Can our gateway send overflow to a second provider when our main one hits its limits, and still keep processing in the EU? Teams ask us this often. It comes up when a launch runs into capacity errors. The availability fix is one line of config. The data question sits in your DPA.
Why overflow routing raises a data question
Overflow routing raises a data question because every request carries its prompt in full. Overflow means requests the main provider cannot take right now. A fallback is the gateway rule that sends them to another provider.
Under the GDPR, a processor is a person or body that processes personal data on behalf of the controller. The controller is the company that decides why the data is processed, which is usually you. If your users' prompts contain personal data, every provider on the fallback list becomes a processor.
Under our DPA, you are the controller and we are the processor. A backup has to meet the same bar as your main provider. Otherwise the fallback weakens what your main provider guarantees.
How AI gateways decide to fall back
AI gateways fall back after a failed call, and each gateway defines a failure differently. In LiteLLM, an open-source gateway, a call that still fails after its retries goes to the next model group on the list. A model group is a name in the gateway config that points at one or more model deployments. LiteLLM tries the fallbacks in the order you list them.
Portkey, a hosted gateway, falls back on any non-2xx status code by default. A non-2xx code is any HTTP answer other than a success. Its on_status_codes setting narrows the trigger, for example to 429 and 503.
LiteLLM also keeps a separate list for prompts that are too long for a model, called context window fallbacks. That list sends exactly your largest prompts to another model. If that model sits at another provider, your users' longest documents are the ones that leave.
The code that wires two endpoints into one client is in our multi-provider failover guide. Running the gateway on your own servers is covered in our self-hosted gateway guide.
Which errors should trigger the fallback
Only capacity errors should trigger the fallback, because those are the failures a second provider can fix. HTTP 429 means too many requests. HTTP 503 means a service is temporarily unavailable. Backoff means waiting a little longer before each new retry.
Our serverless inference docs say what each error means on our API. Your gateway can use them to tell capacity from a bad request.
| Status | Message | What it means on our API | Send to the backup? |
|---|---|---|---|
| 429 | inference capacity exhausted | Our capacity control has no free slot for this model right now | Yes, after retries with backoff |
| 429 | inference service temporarily rate limited | The model backend behind our API returned its own 429, not our capacity control | Yes, after retries with backoff |
| 503 | service temporarily unavailable | The selected backend is temporarily unavailable | Yes, after retries with backoff |
| 400, 409, 413, 422 | inference request rejected | The request itself was rejected, for example a prompt larger than the context window | No, fix the request |
Our docs say to retry 429, 502, 503 and 504 with backoff, and to treat other 4xx rejections as permanent for that request. A rejected request needs a fix in your code, not a second provider. Forwarding it hands its prompt to another company for an error you caused.
If 429s keep coming after retries, the fix may be capacity rather than a backup. A dedicated endpoint runs your model on reserved GPU replicas, which are copies of the model on their own GPUs. The number of replicas scales between a minimum and a maximum that you set.
What the backup provider receives: a worked example
The backup provider receives every overflow prompt in full, however rare the fallback is. Take a hypothetical service. We chose its three inputs to show the arithmetic.
- 200,000 requests a day
- 1% of them get a capacity error on the first try
- half of those succeed when the gateway retries with backoff
first-try capacity errors = 200,000 x 1% = 2,000 a day
sent to the backup = 2,000 x 50% = 1,000 a day
prompts per month = 1,000 x 30 days = 30,000
share of traffic = 1,000 / 200,000 = 0.5%In this example, the backup handles 0.5% of requests and still processes 30,000 prompts a month. Each prompt arrives with everything your app put in it: the user's message, retrieved documents and the conversation so far. For those prompts, the backup is a processor, whatever its share.
The example does not show real error rates on any provider, the latency retries add, or what the backup costs. Take real numbers from your gateway logs.
The checks every backup provider must pass
Every backup provider must pass the same checks as your main provider before its first request. This is general guidance on the GDPR, not legal advice for your setup.
- A contract. Under Article 28, a controller may only use processors that give sufficient guarantees for appropriate technical and organisational measures. In practice, that means a signed DPA with the backup.
- Approval in the chain. A processor may not engage another processor without the controller's written authorisation. If your gateway is a hosted service, the gateway operator is itself your processor. The backup is then the operator's subprocessor, a processor engaged by your processor. The operator must give the backup the same data protection duties by contract.
- Location. The EEA (European Economic Area) is the EU plus Iceland, Liechtenstein and Norway. If the backup processes prompts in a third country outside the EEA, sending them there is a transfer. Article 44 allows a transfer only when the GDPR's transfer conditions are met. If you promised your customers EU processing, pick a backup that processes in the EU.
- Your customers' contracts. If your own DPA with customers lists your providers, the backup belongs on that list before it gets traffic.
The same checks apply to us as your main provider. Our DPA is Annex 1 of our terms, so it applies when you accept them. We tell you before we add or replace a subprocessor, and you can object within 10 business days. Our subprocessor list is available on request. How to read such a list is covered in our guide to subprocessors in AI inference.
Building an EU-only fallback list
An EU-only fallback list holds only backups that pass all four checks, and it fires only on capacity errors. Here is an example config for Portkey's fallback strategy. Both targets are providers you set up in Portkey first. The second one is a placeholder.
{
"strategy": {
"mode": "fallback",
"on_status_codes": [
429,
503
]
},
"targets": [
{
"provider": "@lyceum-serverless"
},
{
"provider": "@eu-backup-with-signed-dpa"
}
]
}We are a sovereign European AI inference provider for open-source models. Our GPUs run in European data centres, and prompts and outputs are processed, not stored, and never used for training. Point the first target at our OpenAI-compatible base URL, https://api.lyceum.technology/openai/v1, with your Lyceum API key as a Bearer token. Keep retries with backoff in front of the fallback, so short spikes stay with your main provider. Log each fallback with its error code, so you can show which prompts went where.
If no backup passes the checks, leave the list empty. A failed request then returns its error to your app, and the prompt stays where you promised.
Add a backup when capacity errors persist after retries and you have a contracted EU provider for it. When they persist and you have none, the fix is capacity. Watch per-model latency and past incidents on our status page, then talk to our engineers about a dedicated endpoint.
Frequently asked questions
Does a fallback provider that handles only a small share of requests need its own DPA?
Yes, if the prompts it receives contain personal data. The share of traffic does not change its role, because it processes those prompts on your behalf.
Why might I get a "429 Too Many Requests" error from the Lyceum API, and what should I do about it?
A 429 error means you've sent more requests than your current limit allows, or the model is under heavy load. Wait a moment and retry, ideally with exponential backoff. If it keeps happening, contact us and we'll look at your limits or a dedicated endpoint.
Does Lyceum offer an uptime SLA, and what availability guarantees apply to serverless inference, dedicated endpoints and GPU capacity?
We don't publish a standard uptime SLA for serverless inference yet, and it's on our roadmap. For dedicated endpoints and reserved GPU capacity, we agree service levels with you as part of your contract.