A token cut is only a saving if your answers stay as good. Score a fixed set of real requests before and after every cut. Keep a cut only when the score stays within a limit you set in advance.
How do I reduce token usage without hurting output quality? It comes up once traffic grows and the bill grows with it. Every request resends instructions, history and documents, and you pay for every token.
Where the tokens in one request go, and what 1,000 tokens cost
A token is a short piece of text, often part of a word, and providers bill per token. A request has input tokens (everything you send) and output tokens (everything the model writes back). Output is the pricier line on the model below.
Take a hypothetical support assistant on Qwen3.8 27B, one of the open models we serve. Our list prices, read 4 November 2026, are $0.40 per 1M input tokens and $2.40 per 1M output tokens. Cached input costs $0.10 per 1M. So 1,000 input tokens cost $0.0004 and 1,000 output tokens cost $0.0024.
Each request in the example carries four parts of input and two parts of output. These token counts are assumptions, not measurements.
| Part of the request | Tokens | Billed as |
|---|---|---|
| System prompt with tool definitions | 3,000 | input |
| Conversation history | 6,000 | input |
| Retrieved chunks (8 × 600) | 4,800 | input |
| The user's question | 200 | input |
| Reasoning trace | 900 | output |
| Answer | 500 | output |
Tool definitions are the descriptions of the functions the model may call. Retrieved chunks are passages your search step adds to the prompt, the core of retrieval-augmented generation (RAG). A reasoning trace is the text a model writes while it thinks, before the answer.
The baseline cost follows from the token counts and the prices:
input_tokens = 3_000 + 6_000 + 4_800 + 200 # 14,000
output_tokens = 900 + 500 # 1,400
cost = input_tokens * 0.40 / 1e6 + output_tokens * 2.40 / 1e6
# 0.0056 + 0.00336 = $0.00896 per request
# x 10,000 requests = $89.60The question itself is 200 of 15,400 tokens. Everything else is a candidate for a cut.
Trim conversation history, retrieved chunks and tool definitions
Context management means deciding which earlier messages and documents each request carries. It is the first cut, because context is most of the input.
Conversation history. Keep the last few turns in full and replace older turns with a short summary. In the example, history drops from 6,000 tokens to 1,800: the last four turns plus a 300-token summary.
Retrieved chunks. Send the best-ranked chunks and drop the rest. Fewer chunks can also help quality. Liu et al. found that language models use information at the start or end of a long input best. They do significantly worse when it sits in the middle. In the example, the top 4 chunks replace 8, so retrieval drops from 4,800 tokens to 2,400.
Tool definitions. Send only the tools the current step can use. Each unused definition is input you pay for on every turn.
Prompt compression rewrites the prompt itself into fewer tokens. LLMLingua (Jiang et al.) reached up to 20x compression with little performance loss on its test datasets. That is a benchmark result, so test it on your prompts. Trim what you send before you compress what is left.
After step 1, the example sends 7,400 input tokens instead of 14,000. The bill falls from $89.60 to $63.20 per 10,000 requests.
Cut output and reasoning tokens with max_tokens and answer formats
Output tokens cost six times as much as input on this model, so every word the model skips counts. Two levers shrink them: a shorter answer format and less reasoning.
Answer format. Ask for the shape you need, such as a two-sentence answer or a JSON object with fixed fields. Research shows how far this goes, and where it stops. Chain-of-thought prompting asks the model to reason step by step before it answers. The concise version (Renze and Guven) asks it to keep those steps brief. It cut average response length by 48.70% with negligible impact on a multiple-choice benchmark. On math problems, one tested model took a performance penalty of 27.69%. Han et al. found that reasoning is often longer than it needs to be. A token budget in the prompt can shorten it, but how much it saves depends on choosing the right budget size. So shorten output per task, never across the board.
Reasoning. On our API, a reasoning trace counts as output tokens, so it is billed at the output price. Qwen3.8 27B reasons by default. Its instant variant, qwen/qwen3.8-27b-instant, is the same model at the same price without a reasoning phase.
max_tokens. This request field caps how many tokens the model may write. If you leave reasoning on, set it high enough for both the reasoning and the answer. Otherwise the whole budget can go to reasoning, and the answer comes back empty or truncated. Our guide to empty answers from reasoning models covers that case.
We are Lyceum, a sovereign European AI inference provider for open-source models. Sovereign here means built and run in Europe. Serverless means we run the servers and you only send requests. Our serverless models run in Paris and Finland, and you call them through one OpenAI-compatible API, billed per token with no base fee. Here is a sketch of the step-2 request, to adapt:
import os
from openai import OpenAI
client = OpenAI(base_url="https://api.lyceum.technology/openai/v1",
api_key=os.environ["LYCEUM_API_KEY"])
resp = client.chat.completions.create(
model="qwen/qwen3.8-27b-instant", # same model and price, no reasoning phase
messages=messages, # system prompt first, new turn last
max_tokens=500, # cap for a short answer, no reasoning to cover
)In the example, reasoning off and a short answer format take output from 1,400 tokens to 350. The bill falls from $63.20 to $38.00 per 10,000 requests.
Prompt caching makes the tokens you must resend cheaper
Prompt caching means the provider reuses the work it did on a prompt start it has seen before. On our API it is automatic. A prefix is the opening text of a prompt. When consecutive requests start with an identical prefix, that part is billed at the lower cached input rate. Caching does not reduce the number of tokens you send. It lowers the price of the ones you must keep.
A cache hit means the prefix was still stored. Three habits raise the hit rate:
- Put the system prompt and tool definitions first.
- Append new turns at the end, and vary only the tail.
- Keep changing values, such as timestamps, out of the early text.
Caching is best-effort. A prefix stays in GPU memory only until newer work pushes it out, so how long it lasts depends on load. In our own test calls, quick follow-ups almost always found the cache. After a pause of one to three minutes, it was hit or miss.
Check the hits in each response rather than assuming them. Only the tokens counted in cached_tokens are billed at the cached rate. The prompt_tokens_details field is absent on a cache miss. Read it defensively:
details = resp.usage.prompt_tokens_details
cached = details.cached_tokens if details else 0
print(resp.usage.prompt_tokens, cached)In the example, a hit on the 3,000-token prompt start bills it at $0.10 per 1M instead of $0.40. The bill falls from $38.00 to $29.00 per 10,000 requests. Our dashboard's Serverless → Usage view shows input, output and cached tokens and the cache share by model, member or API key.
Check output quality after every cut
Build the check before the first cut, then rerun it after each one.
- Record a fixed set of real requests, for example 200, and grade each answer pass or fail.
- Set the limit in advance. Here it is at most 1 point below the baseline pass rate, and at most 1% of answers cut off.
- Apply one cut, rerun the same set, and keep the cut only if both numbers stay within the limit.
The hypothetical example ends like this:
| Step | Input tokens | Output tokens | Cost per 10,000 requests | Pass rate |
|---|---|---|---|---|
| Baseline | 14,000 | 1,400 | $89.60 | 91.0% |
| 1. Trim context | 7,400 | 1,400 | $63.20 | 90.5% |
| 2. Cut output and reasoning | 7,400 | 350 | $38.00 | 90.0% |
| 3. Cache hit on prompt start | 7,400 (3,000 cached) | 350 | $29.00 | 90.0% |
Across the three steps, cost falls by 68% (calculated example). One of 200 answers stopped at the token limit after step 2, within the 1% limit. Every step stays inside the limit, so the team keeps all three.
The example assumes a cache hit on every request and grades made-up answers. It does not show real quality on your traffic, how often your own cache hits, or latency. For latency, measure your own requests and check our status page. A cheaper model is a different lever. Find the cheapest model that clears your quality bar, or try routing requests by difficulty. Our full cost-lever overview covers both.
Cut tokens when your graded set holds within its limit, and stop at the first cut that breaks it. To run that test on your own workload, benchmark an open model with our engineers.
Frequently asked questions
How much do 1,000 tokens cost?
It depends on the model and on the token type. At our list prices read 4 November 2026, 1,000 tokens on Qwen3.8 27B cost $0.0004 as input and $0.0024 as output. Cached input costs $0.0001. On GLM-5.3 they cost $0.0014 as input and $0.0044 as output.
Do I have to turn on prompt caching?
No. Caching is automatic on our API. You get the cached rate when consecutive requests start with identical text and that text is still in GPU memory.
Does turning off reasoning make answers worse?
It depends on the task. Simple lookups and fixed-format replies often pass without reasoning, while multi-step problems can fail. Turn it off per task, and keep the change only if your graded set still passes.