A judge's verdicts are a measurement, not a ruling. Pick a judge from neither model's family and swap the answer order. Decide only once you know how often the judge agrees with your engineers.
How do I use LLM-as-a-judge to compare model outputs when I switch providers? You ask once a candidate open model looks good on public benchmarks. Now it has to hold up on your prompts, and nobody reads thousands of answer pairs by hand.
Open models are now a real option for that switch. Since January 2026, the most capable open-weight models have trailed frontier closed models by four months on average. That is Epoch AI's measure, with data updated 29 May 2026. An average says little about your task, though. The judge measures your task.
We are Lyceum, a sovereign European AI inference provider for open-source models. Our serverless models run in Paris and Finland, and you call them through one OpenAI-compatible API.
How LLM-as-a-judge compares two models
LLM-as-a-judge means a language model grades the answers of other models. Zheng et al. (2023) describe three setups.
- Pairwise comparison: the judge sees a question and two answers, then picks the better one or calls a tie.
- Single answer grading: the judge gives one answer a score.
- Reference-guided grading: the judge also gets a reference solution to compare against.
In the same study, strong judges matched human preferences with over 80% agreement, the same level of agreement as between humans. That is a result on the study's question sets, not a promise for your prompts.
Take a hypothetical team that wants to move off its current closed-API model. The candidate is GLM-5.3, one of the open models we serve. They pick 200 prompts from their product and collect both models' answers. A third model, DeepSeek V4 Pro 0813, judges each pair. We come back to this example in every section.
Pairwise verdicts with a rubric fit a provider switch
A switch asks one question: is the new model at least as good as the old one on your prompts? Pairwise comparison asks exactly that.
Single answer grading can miss subtle differences between two answers. Its absolute scores also move more than pairwise results when the judge model changes.
Give the judge a rubric, the written criteria it applies to every pair. Name what matters for your product: correct facts, followed instructions, the right format. Add "length is not a criterion", for a reason the next section explains.
Leave hard floors, such as "the JSON must parse", to a script.
Ask the judge for a structured verdict, so a script can count it. A JSON object with the winner (A, B or tie) and a one-line reason is enough. Here is a sketch to adapt, using the OpenAI SDK pointed at our API:
import json, os
from openai import OpenAI
client = OpenAI(base_url="https://api.lyceum.technology/openai/v1",
api_key=os.environ["LYCEUM_API_KEY"])
RUBRIC = """You compare two answers to the same user prompt.
Criteria: correct facts, follows the instructions, required format.
Length is not a criterion. Reply only with JSON:
{"winner": "A" | "B" | "tie", "reason": "<one sentence>"}"""
def judge(prompt, answer_a, answer_b):
r = client.chat.completions.create(
model="deepseek/deepseek-v4-pro-0813",
temperature=0,
messages=[{"role": "system", "content": RUBRIC},
{"role": "user", "content":
f"Prompt:\n{prompt}\n\nAnswer A:\n{answer_a}\n\nAnswer B:\n{answer_b}"}])
return json.loads(r.choices[0].message.content)["winner"]
def verdict(prompt, current, candidate):
first = judge(prompt, current, candidate) # current shown first
second = judge(prompt, candidate, current) # order swapped
if first == "A" and second == "B":
return "current"
if first == "B" and second == "A":
return "candidate"
return "tie" # split or tie counts as a tiePosition, verbosity and self-preference bias, and their fixes
A judge is a model, so it has habits. Three of them can tilt a switch.
Position bias is when a judge favours the answer in a certain position, often the first one. Zheng et al. tested it with two very similar answers. Only one of the three judges they tried gave the same verdict in more than 60% of cases after the swap. Wang et al. (2023) went further. By reordering the answers alone, they made one model beat another on 66 of 80 queries.
The fix is the swap in the sketch above. Judge each pair in both orders, and declare a win only when both orders agree. A split counts as a tie. Wang et al. propose a related fix, Balanced Position Calibration: score each pair in both orders and combine the results.
Verbosity bias is when a judge prefers the longer answer, even when it is not clearer or more accurate. A switch often changes answer length, so this bias alone can decide the result. The rubric line on length is the first defence. Reading a sample of the wins is the second.
Self-preference is when a judge scores its own outputs higher than others' answers of equal quality. The next section deals with it.
Math and reasoning prompts need one more step. A judge can be misled by the answers it grades, even on a problem it can solve alone. For those prompts, use reference-guided grading and give the judge the correct answer.
In the example, 120 of the 200 pairs got the same verdict in both orders. The other 80 count as ties.
Choosing a judge from neither model's family
Pick a judge that wrote neither answer. If the candidate judges itself, self-preference pushes its own answers up. If your current model judges, the same bias protects the status quo.
So the judge comes from a third model family. In the example, the current model is a closed-API model and the candidate is GLM-5.3 from Z.ai. The judge is DeepSeek V4 Pro 0813, a model from a different vendor than either. Neither compared model judges its own answers.
A stronger judge is not automatically a fair one. Even the most consistent judge in Zheng et al.'s position test kept its verdict in only about two thirds of swapped pairs. The swap and the rubric apply to every judge.
On our API, the judge is one more model ID behind the same base URL. Its price is on its page: DeepSeek V4 Pro 0813 costs $2.00 per 1M input tokens and $4.00 per 1M output tokens.
Calibrating the judge against your engineers' grades
Calibration means checking the judge against people before you trust it. LLM evaluators inherit the problems of the models they evaluate, so they need human validation.
Have two engineers grade a subset of the pairs blind, without seeing the judge's verdict. Then count how often the judge agrees with them.
Expect the rubric to change during that round. Shankar et al. (2024) call it criteria drift: people need criteria to grade outputs, but grading outputs helps them define the criteria. Update the rubric after the first round, then let the judge run again.
In the example, the engineers grade 50 of the 200 pairs. The judge agrees with their grade on 41 of them, which is 82%. The team decides that is close enough to the human agreement in Zheng et al. to use the judge for the rest. That threshold is the team's choice, not a standard.
Read the 9 disagreements too. If they share a pattern, such as long answers, fix the rubric.
Turning judge verdicts into a switch decision
Set the decision rule before you run the judge, so the result cannot move it. The example team sets a loss limit: the candidate may lose on at most 30% of prompts, and on none tagged critical.
Here are the example verdicts after the swap. These are hypothetical numbers, not a measurement of either model.
| Outcome after the swap (hypothetical) | Prompts | Share |
|---|---|---|
| GLM-5.3 wins in both orders | 62 | 31% |
| Current model wins in both orders | 58 | 29% |
| Split or tie | 80 | 40% |
The current model wins 58 prompts, or 29%. That stays within the 30% limit. Before the switch, someone reads those 58 losses and checks that no critical prompt is among them.
The test itself is cheap. Here is the arithmetic, with prices from our model pages and assumed token counts per call:
Hypothetical inputs: 200 prompts, judged in both orders
Judge calls: 200 x 2 = 400
Judge input: 400 x 1,500 tokens = 600,000 x $2.00 per 1M = $1.20
Judge output: 400 x 300 tokens = 120,000 x $4.00 per 1M = $0.48
Judge total: $1.68
Candidate input: 200 x 500 tokens = 100,000 x $1.40 per 1M = $0.14
Candidate output: 200 x 400 tokens = 80,000 x $4.40 per 1M = $0.352
Candidate total: $0.49
Whole run: $2.17GLM-5.3's prices are on its model page. The example does not show the real quality of either model or latency. A larger or harder prompt set changes every line.
Give the judge run its own API key. An owner or admin can set a spend limit on that key, as a monthly cap or a fixed budget. Once it is reached, new requests with that key are refused with a 402 error. The limit is soft: a request already running is never cut. The usage view in our dashboard groups cost and tokens by API key, so the eval spend stays separate from production.
Your eval set will usually come from product traffic, so it carries customer data. We don't keep your prompts or outputs, and we never use them for training. Public benchmarks help you pick candidates, but your own test decides. After the decision comes the integration work, which our guide on what breaks when switching to an OpenAI-compatible API covers. The judge run fits inside a two-week proof of concept.
Switch when the candidate stays within your loss limit and loses no critical prompt, with a judge your engineers have checked. To run your eval set against an open model on your own workload, talk to our engineers.
Frequently asked questions
Can the candidate model judge its own answers?
It should not. A judge tends to rate its own outputs above equally good answers, which would tilt the test toward the candidate.
Does LLM-as-a-judge work for code and math answers?
Only with help. A judge can accept a wrong answer it would have solved correctly on its own. Give it the correct answer or the test result, and for code, run the tests rather than asking the judge.
How many human-graded pairs do I need before I trust the judge?
No fixed number exists. Grade enough pairs to cover each prompt type you care about, then study the disagreements.