What a two-week PoC with a new inference provider should include

Compare providers and models on the cost of each answered request, not on the price per token. In our test calls, Qwen3.8 27B with its default reasoning step cost about 3.4 times as much as the same model with reasoning off.

What do you need to test in two weeks before you move production traffic to a new inference provider? That question usually comes with a deadline, after capacity problems, a customer's question about data or a growing bill.

The two-week plan below is one you run yourself on pay-as-you-go serverless inference. The proofs of concept we run for dedicated inference are a separate, shorter format, described in the FAQ below.

What a two-week inference PoC has to prove

A two-week inference PoC has to prove that your own workload runs on the new provider at least as well as it runs today. Inference is the step where a hosted model answers a request. The PoC tests your prompts, your traffic and your error handling, not the provider in general.

Four questions cover most workloads:

• Quality: do the answers pass the checks you already run on today's provider?
• Latency: how long do users wait, including on the slowest requests?
• Failures: what happens when a request is rejected, times out or hits a limit?
• Cost: what does one answered request cost, with every token counted?

Testing a new provider matters more now that open models are close behind closed ones. Open-weight models publish their trained model files, so several providers can serve the same model. Frontier closed models are the most capable models whose files stay private, such as GPT and Claude. Epoch AI scores models on one capability index and measures how long open-weight models take to reach a closed model's score. Since January 2026, the best open-weight models have matched a frontier closed model's score about four months after it. That is Epoch AI's finding, from data of 29 May 2026.

Sort out data agreements before any real traffic flows. Our PoC data checklist covers that part.

Write the exit criteria before the first request

Exit criteria are the pass or fail limits that decide the PoC, written down before any test traffic. Set one limit per question, in numbers, and measure your current provider against the same limits. Without that baseline, a result of 2 seconds means nothing.

Two latency measures do most of the work. Time to first token (TTFT) is the time from sending a request to receiving the first token of the answer. It includes queuing, prefill (the model reading your prompt) and network time, and longer prompts raise it. End-to-end latency is the time until the last token arrives.

Report each as a median and as a p95. The p95 is the time that 95 of every 100 requests beat. The slowest 5 sit above it, and those are the requests users notice.

Fix the budget the same way. On our API, an owner or admin can put a spend limit on an API key. Once the key reaches it, new requests get HTTP 402 with the error code api_key_spend_limit_reached. In the dashboard, it sits under Account → API keys → Set spend limit. A PoC key with its own limit cannot overrun the test budget.

Week 1: integration, answer quality and latency percentiles

Week 1 connects your existing client to the new endpoint and runs your own prompts through it. Lyceum is a sovereign European AI inference provider for open-source models. Our serverless models run in Paris and Finland. Prompts and outputs are processed, not stored, and never used for training.

Serverless inference means you call shared open models through one OpenAI-compatible API, billed per token with no base fee. Pointing a client at us changes three values: the base URL, the API key and the model. The script below sends one streamed request and times it.

import os
import time

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LYCEUM_API_KEY"],
    base_url="https://api.lyceum.technology/openai/v1",
    timeout=600.0,
)

start = time.time()
first_token = None
usage = None
stream = client.chat.completions.create(
    model="qwen/qwen3.8-27b",
    messages=[{"role": "user", "content": "How do I reset my password?"}],
    max_tokens=200,
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices and first_token is None:
        delta = chunk.choices[0].delta
        if delta.content or getattr(delta, "reasoning_content", None):
            first_token = time.time() - start
    if chunk.usage:
        usage = chunk.usage

print(f"TTFT {first_token:.2f} s, end-to-end {time.time() - start:.2f} s")
print(f"input {usage.prompt_tokens}, output {usage.completion_tokens}")

Run your quality checks on the same prompts you use today, with the same scoring. A model that fails them is a no-go, whatever its latency or price.

Then log TTFT and end-to-end time for every request. The open-source serving engine vLLM ships a benchmark tool that reports latency the same way. It gives percentiles of time to first token and related measures. Stream anything a user watches: streaming on our API sends tokens as they are generated and lowers time to first token.

Our own test calls show why the p95 matters. On 23 October 2026, we sent 20 streamed requests in a row to Qwen3.8 27B. Each had the same system prompt of about 440 tokens plus a short customer question. The median TTFT was 384 ms. The p95 was 5,080 ms, and the slowest request waited 14,802 ms for its first token.

Twenty requests from one client at one time of day are a small sample, not a benchmark. In your PoC, send far more and spread them across the day.

Week 2: load, 429s, timeouts and failure handling

Week 2 replays your traffic at production rates and checks how your client behaves when requests fail. Start at your normal peak rate, then go above it, and log every status code.

HTTP 429 Too Many Requests means a client sent too many requests in a given time. The response may carry a Retry-After header that says how long to wait. Count your 429s, and check that your client waits and retries instead of failing the user's request.

Timeouts are the next check. A single request on our API may generate for up to 300 seconds. Most cut-off answers come from a client-side timeout that fires first. Under heavy load, time to first token can spike, so a stream inactivity timeout can also end a request early. That is a limit, set in your client library or proxy, on how long a stream may go without a new token.

Test one rarer case as well. A listed model can briefly have no backend available. Our API then answers 404 model not found, the same as for an unknown model ID. Your client should retry with backoff before it treats the model as gone.

Note how fast and how precisely the provider answers your questions during the PoC. Compare your latency numbers with our status page, where a monitor sends a short streaming request to each model every 10 minutes.

The hidden costs a PoC should surface

The cost a price list hides is the tokens you did not plan for. Only a test shows how many tokens an answered request needs.

Many open models write a reasoning trace (text the model writes for itself) before the answer. It counts as output tokens. On Qwen3.8 27B, reasoning is on by default. Setting the request field reasoning_effort to "none", or calling qwen/qwen3.8-27b-instant, turns it off at the same price.

Our 23 October test calls showed the effect. We set max_tokens, the cap on output tokens per request, to 200, as in the script above. With reasoning on, the median request used 184.5 output tokens. Counted separately, the median reasoning part was 133.5 tokens and the median visible answer 48.5, so the two medians do not add up exactly. Eight of 20 requests used the whole cap, and one returned an empty answer because reasoning took all of it. In your PoC, set the cap high enough for both the reasoning and the answer. The same questions on the Instant variant used a median 36 output tokens, and none reached the cap.

Prompt caching (billing repeated prompt text at a lower rate) works on the input side. When requests start with the same text, such as a fixed system prompt, we bill the repeated part at the cached input price. A cache hit (finding and reusing the repeated part) is best effort. In our runs, requests 2 to 20 of each model all reported cached input tokens. Request 1 reported none.

Cost per request follows from three token counts and three prices:

cost per request = (uncached input tokens × input price
                    + cached input tokens × cached input price
                    + output tokens × output price) / 1,000,000

Our worked example prices 100,000 requests, a hypothetical volume, at the medians from our test calls. It uses the Qwen3.8 27B list prices of 23 October 2026: $0.40 input, $0.10 cached input and $2.40 output per 1M tokens.

Our API reported 480 prompt tokens per request with reasoning on and 444 with it off, so the input columns differ. The last column assumes no cache hits, so every input token is billed at the full input price.

Output tokens explain the gap. They are the most expensive token type here, and reasoning made about five times as many of them.

The example favours caching, because a long shared system prompt with short questions is the easiest case for cache hits. It says nothing about answer quality, or about whether reasoning improves your answers. Its requests also ran one at a time, so it shows no effects of load. Read your own token counts per model and API key in the dashboard under Inference → Serverless → Usage. For costs beyond tokens, see what switching providers costs.

Go or no-go: reading the PoC results

Go when every exit criterion passes against your baseline, on your own traffic. Extend the test when one result sits close to its limit and the sample is small. Stop when quality or p95 latency fails by a clear margin, because a lower token price will not fix either.

If reasoning drives the cost, run your quality checks once more with reasoning off before you decide. Some workloads keep their quality without it. If p95 is the weak point, our guide on fixing a slow p95 covers the usual causes.

The PoC also shows your traffic shape. Per-token billing follows your traffic, so quiet hours cost nothing. A dedicated endpoint is priced for the capacity you reserve, so the busier you keep it, the less each token costs. If the PoC shows steady traffic all day, ask us to price a dedicated endpoint against your token bill.

Create an API key with a spend limit in your Lyceum dashboard and replay one day of your own traffic against an open model.