Switching to open models: what to check, in what order

How do you switch from OpenAI or Anthropic to an open model without breaking your product? Most teams change the base URL, see a 200 response and call the work done. This article follows one hypothetical support assistant through five checks. The assistant answers order questions and calls one function, `lookup_order`.

We are Lyceum, a sovereign European AI inference provider for open-source models. Teams call models such as Kimi, GLM, DeepSeek and Qwen through one OpenAI-compatible API, billed per token, on GPUs in European data centres. Our serverless models, which you call per request without managing servers, run in Paris and Finland. Prompts and outputs are processed, not stored, and never used for training.

Define the behaviour your users need to keep

Start with what must stay true, not with a model name. For the assistant, that means a correct order status and a function call when an order number appears. It also means a reply that ends in a few seconds.

Turn each point into an acceptance test: a fixed set of real prompts with a pass rule. Take 30 to 100 prompts from your logs and add the awkward ones, such as a missing order number. Keep the old model's answers as the baseline. These numbers are a proposal, not a standard.

Then fix the pass bar in numbers, because a small test set is noisy. Suppose the baseline passes 48 of 50 prompts and the candidate passes 47. That is 96% against 94%. The 95% range is the span in which the true pass rate probably lies. Here it is 86.5% to 98.9% for the baseline and 83.8% to 97.9% for the candidate. The two ranges share everything from 86.5% to 97.9%, so this test cannot show a gap that small.

The Wilson method, described in the NIST handbook, gives these ranges. For the baseline, p = 48/50 = 0.96 and n = 50, which gives 86.5% to 98.9%.

centre = (p + z^2 / 2n) / (1 + z^2 / n)
half   = z * sqrt(p(1-p)/n + z^2 / 4n^2) / (1 + z^2 / n)
range  = centre - half  to  centre + half
p = passes / prompts, n = prompts, z = 1.96 for a 95% range

With 500 prompts each, 480 passes (96%) give 93.9% to 97.4% and 470 passes (94%) give 91.6% to 95.8%. The ranges narrow but still overlap between 93.9% and 95.8%. The example is hypothetical and treats every prompt as an independent pass or fail. It measures how sure a pass rate is, not answer quality. Decide the size of gap you care about, then add prompts until the ranges separate.

Shortlist open models for the actual task

Open models trail the best closed models. Epoch AI found that since January 2026 the most capable open-weight models have lagged frontier closed models by an average of four months. Plan for a gap on your hardest prompts, and let your tests show whether it matters.

Pick candidates by what your task needs: context length, images, tool calls, price. Our models page lists what we serve. These four fit the assistant. Prices are list prices read on 19 October 2026, in US dollars per million tokens.

A token is the unit a model reads and writes. Check the licence of the exact model you pick, because licences differ between models and Hugging Face tells users to read them before use. Do not rank the four from this table. Rank them on your prompts.

Evaluate quality, latency and total cost together

Send the same prompts through both APIs. For each answer, record whether it passed, how long it took to start and finish, and what it cost. Judge the three together, because a cheaper model that fails more prompts costs more per solved task.

Some of these models write a reasoning trace: thinking text that comes before the answer. It counts as output tokens, so it enters the bill. The calculation below prices one day of the assistant.

cost of a request = input tokens x input price + output tokens x output price
output tokens     = answer tokens + reasoning trace tokens

This is a calculated example with hypothetical inputs, not a measurement. The assistant handles 10,000 requests a day. Each request sends 2,000 input tokens and gets a 300-token answer after a 900-token trace. Trace and answer together are 1,200 output tokens, so `max_tokens` must be at least 1,200, because it caps trace and answer together. Otherwise the whole budget can go into thinking and the visible answer comes back empty or cut off.

The trace is 900 of the 1,200 output tokens, or 75%, so it drives the output cost on every row. The last column prices the same request with no trace. The example assumes the same 900-token trace for every model, but real trace lengths differ. It also leaves out quality, speed and your integration work. Add the engineering hours for the switch and the upkeep afterwards, then divide the total by the share of prompts that passed. Take tokens alone. A model that costs $36.80 a day and passes 94% of prompts costs $39.15 a day per unit of passing work, before engineering hours. Compare that number across models.

Check API and tool behaviour before rollout

Our docs say that streaming, tool calling and usage accounting work the same as with OpenAI. Compatibility is not parity, though. A forced tool call is the sharpest example. In the OpenAI guide, `tool_choice: "required"` means the model must call one or more functions. Our docs state that this is not enforced on `deepseek/deepseek-v4-pro-0813`, `deepseek/deepseek-v4-flash-0731` and `z-ai/glm-5.3`. Those models can answer in plain text with no function call, while the reply field `finish_reason` still says `tool_calls`. Your code then believes a call happened when none did. `moonshotai/kimi-k3`, `z-ai/glm-5.3-flash`, `minimax/minimax-m3` and `qwen/qwen3.8-27b` honour both forms. Our docs name the models that fail and date their checks. Because compatibility differs by model, test the API before the pilot.

The base URL is `https://api.lyceum.technology/openai/v1`. The snippet below changes the base URL and the model, then sends a prompt that needs no tool with `tool_choice` set to `required`. If you depend on a forced call, check `message.tool_calls` and fall back when it is empty.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LYCEUM_API_KEY"],
    base_url="https://api.lyceum.technology/openai/v1",
)
tools = [{"type": "function", "function": {
    "name": "lookup_order",
    "description": "Look up an order by its number",
    "parameters": {"type": "object",
                   "properties": {"order_id": {"type": "string"}},
                   "required": ["order_id"]}}}]

resp = client.chat.completions.create(
    model="moonshotai/kimi-k3",
    messages=[{"role": "user", "content": "Hello, who are you?"}],
    tools=tools,
    tool_choice="required",
    max_tokens=1200,
)
msg = resp.choices[0].message
print("forced tool call honoured:", bool(msg.tool_calls))

Run each check below on every candidate. The API returns the trace as an extra field on the reply message. Most models use `message.reasoning_content` and some backends use `message.reasoning`, so read both.

A single request may run up to 300 seconds of generation time, so stream long or user-facing answers. A listed model can briefly have no backend and then answers `404 model not found`. Retry that case with backoff, because it can pass. Retry 429, 502, 503 and 504 as well, and treat other 4xx errors as permanent. If you come from Anthropic in Claude Code, our inference page describes an Anthropic-compatible endpoint, which accepts the API format Claude Code expects. For an application, use the OpenAI-compatible API above.

Pilot the switch with a working rollback

A canary release rolls a change out to a small share of users first, which limits the damage if something is wrong. Route a small share of the assistant's traffic to the new model and keep the old model's key and prompts working until the pilot ends. Keep the model name in a setting, such as a config flag or router rule, so one change sends all traffic back. That change is the rollback.

Write the triggers before you start. Proposed examples: the pass rate on your sampled live answers falls below the bar, answers come back empty, or tool calls fail. When one fires, send the traffic back. In the dashboard, Inference → Serverless → Usage shows input, output and cached tokens per model, member and API key. Cached tokens are input billed at the lower repeated-text rate. Use that page to compare the pilot's cost per model with your baseline.

Widen the share only when the pilot holds your bar for a period you chose in advance. If you are not sure the bar is met, keep the old model for all traffic. Three guides come next: what breaks when switching models behind an OpenAI-compatible API, how to read open model benchmarks and what switching inference providers costs.

Book a 30-minute conversation with our team, bringing your acceptance tests and expected traffic: talk to an expert.