Your current model is the benchmark, not a leaderboard. Before the test, decide how far below your current model the open model may score. Then compare each result with that limit.
How do I evaluate open-source LLMs against my proprietary model before I switch production traffic? A candidate looks good on paper. Before any user sees it, you need evidence from your own work.
The switch is a real option now. Open-weight models are models whose trained weights anyone can download and run. Epoch AI tracks model capability as one combined score across many benchmarks. Since January 2026, the strongest open models have reached the score of the best closed models about four months later, on average. Epoch's data was updated on 29 May 2026.
We are Lyceum, a sovereign European AI inference provider for open-source models. We run open models in European data centres, and you call them over an API. Our serverless models, the ones you pay per token without renting a server, run in Paris and Finland. You test the same model and the same API your production traffic would use later.
Why public benchmarks don't answer the switching question
A public benchmark is a fixed set of test questions that everyone scores models on. Your product asks different questions.
There is a second problem. Some benchmark scores may reflect contamination, where data close to the test questions leaked into a model's training data. Zhang and colleagues (2024) built GSM1k, a fresh set of school math problems in the style of GSM8k, an older public math benchmark. Leading models scored up to 8% lower on it than their GSM8k results suggested.
Use public scores to pick two or three candidates. Our guide on reading open-model benchmarks shows how. Then test those candidates on your own prompts.
We follow one hypothetical example through this article. A team runs a support assistant on a closed model. It wants to know whether DeepSeek V4.1 Flash, one of the open models we serve, can take over. The numbers in the example are invented for illustration, not measured.
Compare quality, latency, cost and errors against your current model
HELM, a Stanford evaluation by Liang and colleagues, scores models on 7 metrics instead of accuracy alone. Its aim is to keep trade-offs visible. A switch needs the same habit, with four measures:
- Quality: is the answer right, complete and in the required format?
- Latency: how long do your slowest requests take? Track p95, the time within which 95 of every 100 requests finish. Our guide on fixing p95 latency explains why the average hides problems.
- Cost per request: tokens in and out, times the price per token.
- Errors: timeouts, rate-limit errors and empty answers. Our status page lists incidents, and our guide on verifying provider reliability covers what to log.
Measure your current model on the same prompts, in the same week, from the same servers. That baseline is what the candidate has to match.
Build the test set from your own production traffic
A test set is the list of prompts both models answer. Sample it from recent production logs, so it holds the requests your users really send.
Split it by route, meaning each kind of request your product makes. Each route can pass or fail on its own. The example team samples 300 prompts from 30 days of logs:
- 200 open answers to customer questions;
- 60 extractions, where the model returns JSON (a structured text format) for the ticket system;
- 40 tool calls, where the model asks your code to run a function, such as a lookup.
Add the hard cases on purpose: long threads, rare languages, past incidents. Keep the set fixed afterwards. It then works as a regression test, a test you re-run after every model or prompt change.
Logs from a live product hold customer data. Check your contracts before that data reaches a new provider. Our checklist on customer data before an LLM proof of concept lists what to settle. With us, your prompts and outputs are processed, not stored, and never used for training.
Score outputs with rules, an LLM judge and human review
Score in three layers, cheapest first.
- Rules. A script checks what has one right answer: does the JSON parse, are the required fields there, is the right tool called?
- An LLM judge. For open answers, a third model compares both answers to the same prompt and picks the better one. Our guide on LLM-as-a-judge covers how to swap the answer order and check the judge.
- People. Engineers read every prompt the candidate lost, and a sample of its wins.
Run each prompt more than once. In one test, Horace He at Thinking Machines sent the same prompt 1,000 times at temperature 0. That setting should always pick the most likely next word. The model still returned 80 unique completions out of those 1,000.
So one run per prompt can mislead you. Miller (2024) recommends running each question several times and averaging its scores before you compare models. The paper also advises against lowering the temperature to reduce that spread. A lower temperature changes the answers themselves, and as the test above shows, it does not stop the variation anyway. Keep the temperature you use in production. The example team runs every prompt three times on both models.
Test tool calling and JSON output separately
Tool calls and JSON break in ways a judge reading the text can miss. A call can look fine and still name the wrong function or skip a required argument.
The Berkeley Function Calling Leaderboard (BFCL) tests how accurately models call tools. Its first version introduced AST (abstract syntax tree) evaluation as a metric, a check on the structure of each call rather than its wording. Do the same with your own tools: compare the function name and every argument with the expected call.
Read the model page before you test. Our DeepSeek V4.1 Flash page says the model takes tools in the standard OpenAI format. It also says that even when you require a tool call, the model may answer in plain text instead. The page advises you to check the response's list of tool calls and fall back when it is empty. Count an empty list as a failed test, and your evaluation catches what production would hit.
Any OpenAI-compatible client works with our API: set the base URL, your API key and a model ID. Here is a sketch of the tool-call check, pointed at our API:
import json, os
from openai import OpenAI
client = OpenAI(base_url="https://api.lyceum.technology/openai/v1",
api_key=os.environ["LYCEUM_API_KEY"])
def tool_call_passes(messages, tools, expected_name, expected_args):
r = client.chat.completions.create(
model="deepseek/deepseek-v4.1-flash",
messages=messages,
tools=tools,
)
calls = r.choices[0].message.tool_calls
if not calls: # plain-text answer: a failed test
return False
call = calls[0].function
return call.name == expected_name and json.loads(call.arguments) == expected_argsRun the same check on your current model. For JSON routes, validate every answer against your schema.
Set pass/fail limits before the run, then decide per route
Write the limits down before the first request, so a result cannot move them. Set each one as a margin against your current model. The example team chose these limits and got these results:
| Route (hypothetical) | Current model | DeepSeek V4.1 Flash | Limit | Verdict |
|---|---|---|---|---|
| Open answers, 200 | baseline | loses 48 (24%) | loses at most 30% | pass |
| JSON extraction, 60 | 59 valid (98.3%) | 57 valid (95.0%) | at most 2 percentage points lower | fail |
| Tool calls, 40 | 38 right (95.0%) | 37 right (92.5%) | at most 5 percentage points lower | pass |
A 24% loss rate on 200 prompts is an estimate, so it comes with an error margin. The standard error says how far a score could move by chance with a different sample of prompts. Miller recommends reporting it next to every score. He also recommends comparing the two models on each prompt, then averaging those differences, instead of comparing two overall scores. Both models then face the same easy and hard prompts, so the margin shrinks.
Hypothetical: the candidate loses 48 of 200 open answers
p = 48 / 200 = 0.24
standard error = sqrt(p x (1 - p) / n) = sqrt(0.24 x 0.76 / 200) = 0.0302
margin = 1.96 x 0.0302 = 0.059
# 1.96 standard errors either side cover 95% of the results chance alone would give
95% interval = 24% +/- 5.9% = 18.1% to 29.9%Even the top of that range stays under the limit, so open answers pass. Had the range crossed the limit, the team would need more prompts.
Tool calls pass too: 37 of 40 were right, and one of the 3 wrong ones was a plain-text answer with no tool call. JSON extraction fails at 3.3 percentage points below the current model. The team reads the 3 failed extractions, fixes the prompt and retests that route.
The test itself costs little. Here is the arithmetic, with prices from our DeepSeek V4.1 Flash page and an assumed request size.
Hypothetical inputs: 300 prompts, 900 input and 250 output tokens each
Input: 300 x 900 = 270,000 tokens x $0.50 per 1M = $0.135
Output: 300 x 250 = 75,000 tokens x $1.50 per 1M = $0.1125
One pass: $0.2475
Three passes: 3 x $0.2475 = $0.7425, about $0.74The example does not show the real quality or latency of either model. Other workloads give other numbers.
Give the test its own API key. An owner or admin can set a spend limit on that key, as a monthly cap or a fixed budget. You pay per token with no base fee, so the test needs no commitment.
How long should it run? Our recommendation, not a measurement: the offline passes take hours. Then send a copy of live requests to the candidate for at least a full week, while users still see only the current model. That is called shadow traffic, and a full week covers both weekday and weekend patterns. After that, move a small share of real traffic. Our plan for a two-week proof of concept lays out these weeks.
In a proof of concept, we benchmark one to three open models against the model you use today, on your typical use case. Move a route when the candidate stays inside every limit across repeated runs and a week of shadow traffic. To benchmark an open model on your own workload, talk to our engineers.
Frequently asked questions
How many prompts do I need?
It depends on how small a difference you need to see. Miller gives a sample-size formula for that. In the example, 200 prompts left an error margin of about 6 percentage points either way, so a route close to its limit needs more.
Should I test at temperature 0?
Test at the settings you will use in production. A lower temperature does not make answers fully repeatable, so repeated runs are still needed.