Switching to an open model: five checks before you go live

Changing your API settings is only the first step.

Before switching, check that the new model gives useful answers, works with your application and meets your requirements for speed and cost.

The short version
about lyceum

Lyceum is a European AI inference provider. Teams access open models through one OpenAI-compatible API, billed per token.

worked example

What reasoning adds to the bill

You pay for 1,200 output tokens, even though the user only sees a 300-token answer.

Some models generate reasoning tokens before producing the visible answer. These can add to both the bill and the time users wait. This hypothetical assistant handles 10,000 requests a day. Each request uses 2,000 input tokens, 900 reasoning tokens and 300 tokens for the visible answer.

cost.formula
cost = input tokens  × input price
     + output tokens × output price
output = answer + reasoning

So what Check how the API limits output. Where reasoning and the answer share a token limit, leave enough room for both: a limit that’s too low can produce an empty or incomplete answer.

Exhibit 2Reasoning is $21.60 of a $36.80 day: 75% of the output tokens are never shown to the user$ per day, 10,000 requests · rows: reasoning tokens per request
Input · $0.40 / 1MAnswer · $2.40 / 1MReasoning · $2.40 / 1M
0 tokens
$15.20
900 tokens
$36.80
900-token row in detail
input$8.0020 M tok × $0.40
answer$7.203 M tok × $2.40
reasoning$21.609 M tok × $2.40
Source Illustrative calculation, not measured performance. Uses list prices recorded on 19 October 2026. Reasoning length is hypothetical. The zero-reasoning row shows the cost difference; it does not mean every model supports turning reasoning off.
Step 01 / 05 · Choose

Define what the model must do

A small test set can reveal obvious problems, but it won’t reliably distinguish similar results.

  • Write down what your users need the model to get right. For the support assistant: calling lookup_order when the user provides an order number, asking for it when it’s missing, giving the correct status from the lookup, responding within an acceptable time.
  • Turn these requirements into tests. Start with 30–100 examples from real conversations, including difficult or incomplete requests. Run them through your current model to establish a baseline.
  • For each example, define what counts as a pass. Then set your minimum requirements for accuracy, response time and critical behaviour.
Before moving on: have a reusable test set, results from your current model and clear requirements for the replacement.
Exhibit 348 against 47 of 50 is only one answer apart, not yet a meaningful differenceTests passed, test set of 50
one model
48 / 50
another model
47 / 50
Source Illustration. Test more examples before treating a one-answer gap as a meaningful difference.
Step 02 / 05 · Choose

Choose models that fit your task

Model descriptions and benchmarks can help you choose candidates; your own tests should determine which one you use.

  • Use your requirements to build a shortlist. Check whether each model supports the amount of text your application sends, images if your application uses them, the function calls your workflow needs and a price range that could work at your expected volume.
  • For the support assistant, correct order lookups and clear replies matter more than performance on unrelated coding tasks.
  • Check the licence of each candidate before committing to it.
Before moving on: have a shortlist of models that meet your basic requirements and are ready to test.
Exhibit 4A single ✕ keeps a candidate off the shortlist: only A and C go on to testingShortlist
ModelTextFunction callsPrice
A → test✓✓✓
B✓✕✓
C → test✓✓✓
D✕✓✕
Images and licence: ✓ for all four
Source Illustration, hypothetical candidates.
Step 03 / 05 · Test

Compare quality, speed and total cost

A lower token price may not save money if more requests fail or need another attempt.

  • Send the same test requests to your current model and each candidate. For every request, record quality against your pass criteria, time to start, time to finish and cost, including reasoning tokens.
  • Reasoning tokens add to both the bill and the time users wait: see the worked example.
  • Compare cost per successful task: divide the total cost by the number of tasks completed successfully.
  • Include the engineering work needed to switch and maintain the integration when estimating the overall cost.
Before moving on: identify which candidates meet your quality and speed requirements at an acceptable total cost.
Exhibit 5$36.80 for 10,000 requests is $3.91 per 1,000 successful tasks when 9,400 passcost.per_success
cost$36.8010,000 requests
÷ pass9,400of 10,000 pass your tests
per 1,000 successful tasks$3.91
Source Illustrative calculation, not measured performance. Pass rate hypothetical. Engineering costs are excluded.
Step 04 / 05 · Test

Test the integration

An API can accept your request and still behave differently from your current provider.

  • For our assistant, test the complete order lookup: the model requests lookup_order, the application runs it and the model uses the result to answer.
  • Also test what happens when a required function call is missing. Check message.tool_calls directly: a successful HTTP response or a finish_reason value alone doesn’t prove the call happened.
  • The diagnostic below requires a function call even though the greeting doesn’t need one. Replace the model name to test each candidate. It does not execute the function or validate its arguments.
  • On our API, retry 404 model not found (temporary when a listed model has no available backend), 429, 502, 503 and 504 with backoff; other 4xx are permanent. The generation limit is 300 s per request: stream long answers.
Before moving on: confirm that the full workflow works and that your application handles failures.
Exhibit 6An accepted request is not a working integration: check each feature on every candidate
FeatureWhat to check
StreamingResponses arrive progressively and finish without timing out.
Function callsThe model calls the correct function with usable arguments, then handles its result; the response contains a call whenever your request requires one.
Output limitsThe model has enough room to finish the visible answer.
Error handlingYour application handles temporary failures and retries appropriately.
Usage reportingYour code handles missing optional fields.
Source Lyceum; serverless documentation
forced_tool_check.pyPython
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LYCEUM_API_KEY"],
    base_url="https://api.lyceum.technology/openai/v1",
)
tools = [{"type": "function", "function": {
    "name": "lookup_order",
    "description": "Look up an order by its number",
    "parameters": {"type": "object",
        "properties": {"order_id": {"type": "string"}},
        "required": ["order_id"]}}}]

resp = client.chat.completions.create(
    model="moonshotai/kimi-k3",
    messages=[{"role": "user", "content": "Hello, who are you?"}],
    tools=tools,
    tool_choice="required",
    max_tokens=1200,
)
msg = resp.choices[0].message
print("forced tool call honoured:", bool(msg.tool_calls))
Step 05 / 05 · Roll out

Start small and keep a way back

Make switching back a single configuration change, and test that it works.

  • Send a small share of live traffic to the new model first. Keep the current model available while you check the results.
  • Before starting, decide how much traffic the new model will receive, how long you’ll observe it, which results would make you switch back and how you’ll return traffic to the current model.
  • For the support assistant, reasons to switch back might include failed order lookups, empty answers or accuracy falling below your minimum requirement.
  • During the pilot, compare answer quality, response times and cost with your baseline.
Before expanding the rollout: confirm that the pilot meets your requirements and that you can still switch back.
Exhibit 7Increase traffic only after the agreed period; switching back stays one configuration changeLive traffic: current model → new model
pilotSmall share of live trafficcurrent model stays available
observeCompare with your baseline↩ switch back: single configuration change → current model
increaseAfter the period you agreed in advanceonly once the new model meets your requirements
Source Illustration.

Your next workload starts here