Switching to an open model: five checks before you go live
Changing your API settings is only the first step.
Before switching, check that the new model gives useful answers, works with your application and meets your requirements for speed and cost.
Lyceum is a European AI inference provider. Teams access open models through one OpenAI-compatible API, billed per token.
We’ll use a support assistant as an example. It answers order questions and calls a function called lookup_order to check an order’s status.
Here, “open model” means a model whose trained weights are available to download and run. Check the licence for the specific model you’re considering.
- Choose01Define the jobA small test set can reveal obvious problems, but it won’t reliably distinguish similar results.→
- 02Shortlist modelsModel descriptions and benchmarks can help you choose candidates; your own tests should determine which one you use.→
- Test03Compare on your chatsA lower token price may not save money if more requests fail or need another attempt.→
- 04Test the integrationAn API can accept your request and still behave differently from your current provider.→
- Roll out05Go live smallMake switching back a single configuration change, and test that it works.→
What reasoning adds to the bill
You pay for 1,200 output tokens, even though the user only sees a 300-token answer.
Some models generate reasoning tokens before producing the visible answer. These can add to both the bill and the time users wait. This hypothetical assistant handles 10,000 requests a day. Each request uses 2,000 input tokens, 900 reasoning tokens and 300 tokens for the visible answer.
cost = input tokens × input price
+ output tokens × output price
output = answer + reasoningSo what Check how the API limits output. Where reasoning and the answer share a token limit, leave enough room for both: a limit that’s too low can produce an empty or incomplete answer.
Define what the model must do
A small test set can reveal obvious problems, but it won’t reliably distinguish similar results.
- Write down what your users need the model to get right. For the support assistant: calling
lookup_orderwhen the user provides an order number, asking for it when it’s missing, giving the correct status from the lookup, responding within an acceptable time. - Turn these requirements into tests. Start with 30–100 examples from real conversations, including difficult or incomplete requests. Run them through your current model to establish a baseline.
- For each example, define what counts as a pass. Then set your minimum requirements for accuracy, response time and critical behaviour.
Choose models that fit your task
Model descriptions and benchmarks can help you choose candidates; your own tests should determine which one you use.
- Use your requirements to build a shortlist. Check whether each model supports the amount of text your application sends, images if your application uses them, the function calls your workflow needs and a price range that could work at your expected volume.
- For the support assistant, correct order lookups and clear replies matter more than performance on unrelated coding tasks.
- Check the licence of each candidate before committing to it.
| Model | Text | Function calls | Price |
|---|---|---|---|
| A → test | ✓ | ✓ | ✓ |
| B | ✓ | ✕ | ✓ |
| C → test | ✓ | ✓ | ✓ |
| D | ✕ | ✓ | ✕ |
Compare quality, speed and total cost
A lower token price may not save money if more requests fail or need another attempt.
- Send the same test requests to your current model and each candidate. For every request, record quality against your pass criteria, time to start, time to finish and cost, including reasoning tokens.
- Reasoning tokens add to both the bill and the time users wait: see the worked example.
- Compare cost per successful task: divide the total cost by the number of tasks completed successfully.
- Include the engineering work needed to switch and maintain the integration when estimating the overall cost.
Test the integration
An API can accept your request and still behave differently from your current provider.
- For our assistant, test the complete order lookup: the model requests
lookup_order, the application runs it and the model uses the result to answer. - Also test what happens when a required function call is missing. Check
message.tool_callsdirectly: a successful HTTP response or afinish_reasonvalue alone doesn’t prove the call happened. - The diagnostic below requires a function call even though the greeting doesn’t need one. Replace the model name to test each candidate. It does not execute the function or validate its arguments.
- On our API, retry
404 model not found(temporary when a listed model has no available backend),429,502,503and504with backoff; other 4xx are permanent. The generation limit is 300 s per request: stream long answers.
| Feature | What to check |
|---|---|
| Streaming | Responses arrive progressively and finish without timing out. |
| Function calls | The model calls the correct function with usable arguments, then handles its result; the response contains a call whenever your request requires one. |
| Output limits | The model has enough room to finish the visible answer. |
| Error handling | Your application handles temporary failures and retries appropriately. |
| Usage reporting | Your code handles missing optional fields. |
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["LYCEUM_API_KEY"],
base_url="https://api.lyceum.technology/openai/v1",
)
tools = [{"type": "function", "function": {
"name": "lookup_order",
"description": "Look up an order by its number",
"parameters": {"type": "object",
"properties": {"order_id": {"type": "string"}},
"required": ["order_id"]}}}]
resp = client.chat.completions.create(
model="moonshotai/kimi-k3",
messages=[{"role": "user", "content": "Hello, who are you?"}],
tools=tools,
tool_choice="required",
max_tokens=1200,
)
msg = resp.choices[0].message
print("forced tool call honoured:", bool(msg.tool_calls))Start small and keep a way back
Make switching back a single configuration change, and test that it works.
- Send a small share of live traffic to the new model first. Keep the current model available while you check the results.
- Before starting, decide how much traffic the new model will receive, how long you’ll observe it, which results would make you switch back and how you’ll return traffic to the current model.
- For the support assistant, reasons to switch back might include failed order lookups, empty answers or accuracy falling below your minimum requirement.
- During the pilot, compare answer quality, response times and cost with your baseline.
