Why route between open and proprietary models?
Routing pays when part of your traffic does not need your most expensive model. A proprietary model is one whose weights only its vendor serves, through the vendor's own API. An open model publishes its weights, so any provider can serve it. Many applications send everything to one proprietary model, including short answers and tagging jobs a smaller open model can handle.
Decide the routing rule yourself, and call each model by its exact name while you measure. Then a change in cost or quality comes from the split, not from a model that changed underneath you.
How do I route requests between open-source and proprietary models in one application? The question usually comes up when the bill grows, or when an EU customer asks where prompts are processed.
Research on splitting traffic between a strong and a weak model is encouraging. The RouteLLM project trained routers that choose between a stronger and a weaker model for each request. In their tests, the strong model was GPT-4 Turbo and the weak one was Mixtral 8x7B, an open-weight model. On MT Bench, a set of multi-turn test questions, one router kept 95% of GPT-4's score with 26% of calls going to GPT-4. That was about 48% cheaper than a router that picks the model at random and reaches the same score. These are their benchmark results, not a promise for your traffic.
The gap you route across is also small now. Since January 2026, the strongest open-weight models have trailed the best closed models by about four months. That is the gap on Epoch AI's capability index (data insight of 29 May 2026). So the open side of your router can take real work, not only leftovers.
Which routing rules work: task type, data sensitivity and cost
A routing rule is a few lines of your own code that read a request and return a model. Three rules cover most applications.
Task type. Label each call site with its job: answer an FAQ, tag a ticket, extract fields, or reason through a long case. Short, well-defined jobs go to the open model. Long reasoning stays where your evaluations say it works best. The label comes from your code, so it costs nothing at request time.
Data sensitivity. Requests that carry personal data go to a model on EU-hosted infrastructure. This is the share of traffic we built Lyceum for. Lyceum is a sovereign European AI inference provider for open-source models: our serverless models run in Paris and Finland. Prompts and outputs are processed, not stored, and our Qwen3.8 Flash Next page states that they are never used for training. The rule is a flag set where the data enters, such as a form with customer details. Your proprietary provider never sees those requests.
Cost. Once task labels exist, cost tells you which labels to move first. Here is a hypothetical support assistant with 1,000,000 requests a month. Each request reads about 1,500 input tokens and writes about 300 output tokens. A token is a piece of a word, and it is the unit you are billed in. The rule sends FAQ answers and ticket tagging, 70% of traffic, to `qwen/qwen3.8-flash-next`. The other 30% stays on the proprietary model.
Our model page lists $0.20 per 1M input tokens and $0.50 per 1M output tokens. Cached input costs $0.05 per 1M tokens (prices read on 23 October 2026). Cached input is the start of a prompt that repeats exactly, billed at a lower rate.
Calculated example, hypothetical traffic, prices read 23 October 2026
Routed requests: 1,000,000 x 70% = 700,000
Input: 700,000 x 1,500 tokens = 1,050M tokens x $0.20 = $210.00
Output: 700,000 x 300 tokens = 210M tokens x $0.50 = $105.00
Routed share per month $315.00
If the first 1,000 tokens repeat and every one is a cache hit:
Cached input: 700M tokens x $0.05 = $35.00
New input: 350M tokens x $0.20 = $70.00
Output: $105.00
Routed share per month $210.00The first total is what the routed share costs with no caching. The second total assumes that every request's first 1,000 tokens are a cache hit. A cache hit is best effort, so $210.00 is the lowest case, not a forecast. The example does not show answer quality or speed. It also leaves out what your proprietary model charges for the same 700,000 requests: take that number from your own invoice.
Routing by how hard a request is works too, but it needs a classifier and its own tests. We cover it in routing requests by difficulty.
Where the router lives: your code or an open-source proxy
The router can live in your application or in a proxy that sits between your application and the providers. Start in your code. It is one function, and you can read every decision it makes.
Our serverless inference uses the OpenAI-compatible base URL `https://api.lyceum.technology/openai/v1`, and you authenticate with your Lyceum API key. So the open side of the router is a second OpenAI client with another base URL and key.
import os
from openai import OpenAI
proprietary = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
open_models = OpenAI(
api_key=os.environ["LYCEUM_API_KEY"],
base_url="https://api.lyceum.technology/openai/v1",
)
OPEN_TASKS = {"faq_answer", "ticket_tag"}
OPEN_MODEL = "qwen/qwen3.8-flash-next"
PROPRIETARY_MODEL = os.environ["PROPRIETARY_MODEL"] # the model you call today
def route(task: str, has_personal_data: bool):
if has_personal_data or task in OPEN_TASKS:
return open_models, OPEN_MODEL
return proprietary, PROPRIETARY_MODEL
def complete(task: str, messages: list, has_personal_data: bool = False):
client, model = route(task, has_personal_data)
return client.chat.completions.create(model=model, messages=messages)A proxy makes sense once several services share the same routes. LiteLLM, an open-source proxy, can send requests to any OpenAI-compatible endpoint. Our guide to a self-hosted LLM gateway covers that setup.
We also serve routing aliases: model IDs that point at another model and are billed as that model. They let your rule name a tier, such as simple or complex, instead of a model. Today `lyceum/simple` sends each request to Qwen3.8 Flash Next. `lyceum/complex` and `lyceum/router` both send each request to MiniMax M3, and `lyceum/reasoning` sends to GLM-5.2. Each model page states the target with the word "today", so it can change. While you compare, call the full model ID, and read our note on version pinning.
How fallbacks between providers work
A fallback is a second model that gets the request when the first call fails. Typical triggers are a timeout, a rate-limit error or a server error. Keep the list short and in a fixed order, and log every time it fires.
The trap is data. Say a request with personal data fails on the open model and falls back to the proprietary one. Your sensitivity rule sent it away from that provider, and the fallback sends it straight back. So a sensitive request should fall back to a second EU-hosted model, or fail and retry later. On our API, that second model is one model ID away. If `qwen/qwen3.8-flash-next` fails, retry on `minimax/minimax-m3`, which also runs on EU-hosted infrastructure behind the same base URL and key. It costs more per token ($0.40 per 1M input and $2.00 per 1M output tokens), so log how often it fires.
Our article on failover across two endpoints shows the retry code. For the state of our models, read our status page. It shows sampled model checks, not the success rate of all customer requests.
How to keep outputs consistent across models
Define the shape of each answer once, as a JSON Schema, and check every response against it. A JSON Schema is a written description of the fields and types a JSON answer must have. The check does not care which model answered, so it catches drift on both sides of the router.
When a response fails the check, retry it once on the same model, then send it down the fallback list. Count the failures per model, because that count is one of the numbers you decide on.
The rest of the request stays the same on both sides. Chat completions, streaming, tool calling and usage accounting work on our API as they do with OpenAI. One request can generate up to 131,072 tokens and run for up to 300 seconds. For long outputs, set `stream: true`.
How to measure whether routing pays off
Routing pays off when the routed share costs less and still passes your checks. Compare a week before and a week after the switch on three numbers, with the rule unchanged:
| Number | Where it comes from |
|---|---|
| Cost per routed request | The `usage` field of each response, times the price on the model page |
| Share of answers that pass the schema check | Your validator's log |
| How often the fallback fires | Your router's log |
Each response reports its token counts in `usage`. The cached part sits in `usage.prompt_tokens_details.cached_tokens`, which is missing when the cache missed. That tells you whether the routed share lands near $315.00 or near $210.00 in the example. If several teams share a proxy, LiteLLM can also track spend per key, user and team.
Keep the split when the cost falls and the pass rate holds. Move a task back when its pass rate drops.
To start, send your FAQ and tagging traffic to `qwen/qwen3.8-flash-next` for one week, then compare it on the three numbers in the table. Your proprietary model keeps the long reasoning jobs, and the routed share runs on our EU infrastructure. The model IDs and prices are on our models page.