The per-token price of a RAG query is easy to read off a pricing page. The cost of the same query on a GPU you run yourself is not. It depends on traffic, memory and how many queries the GPU works on together. What does one query through my RAG pipeline cost, and when does a GPU of my own make it cheaper?
Embedding vs retrieval vs generation cost
Of the three steps in a RAG pipeline, generation carries almost all of the cost. Generation is the step where the language model writes the answer. A RAG pipeline answers a question in three steps. It turns the question into a vector, a list of numbers that captures its meaning: that is the embedding. It searches stored vectors for the closest passages: that is retrieval. Then the model reads the passages and writes the answer.
Take one question to the support assistant from our RAG infrastructure guide. The inputs are illustrative, not a customer measurement. The question has 25 tokens. Retrieval returns eight passages of 500 tokens each. A 500-token system prompt, the fixed instructions sent with every request, comes on top. The model writes a 400-token answer. Prices were read on our model pages on 30 October 2026.
input tokens: 25 question + 8 × 500 passages + 500 system prompt = 4,525
embedding: 25 tokens × $0.02 / 1M = $0.0000005
generation: 4,525 × $0.40 / 1M = $0.001810 input
+ 400 × $2.40 / 1M = $0.000960 output
= $0.002770 per query
| Step | Runs on | Cost per query in the example |
|---|---|---|
| Embedding the question | Qwen3 Embedding 8B on our API, $0.02 per 1M input tokens | $0.0000005 |
| Vector search | Your search server, usually on CPUs | Part of the server you already run |
| Generation | Qwen3.8 27B on our API: $0.40 input, $0.10 cached input, $2.40 output per 1M tokens | $0.002770 |
Input costs a sixth of output per token, yet it makes up $0.00181 of the $0.00277. A RAG prompt is long, so the retrieved text dominates the bill. Pick the embedding model for retrieval quality, not price. The MTEB benchmark (Massive Text Embedding Benchmark) covers 8 embedding tasks over 58 datasets and 112 languages. Its public leaderboard ranks the current models.
Speed matters too. On the Qwen3.8 27B model page, our status checks show a median time to first token of 291 ms. The median output speed is 45 tokens per second. Time to first token is the wait before the answer starts. Both medians cover the last 7 days of short requests, measured on 30 October 2026. A 400-token answer at 45 tokens per second takes about 8.9 seconds to stream.
Batching
On a GPU you rent, the cost of one query depends on how many queries the GPU processes at the same time. That count is the batch size. vLLM, an open-source inference server, uses continuous batching: new requests join the running batch after every generated token. It does not wait for a whole batch to finish.
Run the same query on a rented NVIDIA H100, a data-centre GPU with 80 GB of memory. Our GPU virtual machines are billed per second, so the on-demand H100 costs $0.0013 per GPU-second. The query runs on Qwen3.8 27B in 16-bit weights, two bytes per parameter. With 27.8 billion parameters, the weights take 55.6 GB.
The estimate below is a roofline: the fastest the GPU could go if memory and compute ran at the datasheet limits. It favours the GPU. It ignores scheduling overhead, idle time and code that runs below peak speed.
| Queries in the batch | GPU-seconds per query | Cost per query | Against $0.00277 on our API |
|---|---|---|---|
| 1 | 6.94 | $0.0090 | 3.3 times more |
| 4 | 1.97 | $0.0026 | About the same |
| 8 | 1.14 | $0.0015 | 0.54 times |
| 32 | 0.52 | $0.0007 | 0.24 times |
Each row adds the prompt reading to the answer writing, which the batch shares. The next section shows where the 0.25 s and the step times come from:
GPU-seconds per query = 0.25 s prefill + 400 answer tokens × time per step / queries in the batch
batch of 4 = 0.25 s + 400 × 17.1 ms / 4 = 1.97 s → × $0.0013028 = $0.0026
batch of 8 = 0.25 s + 400 × 17.7 ms / 8 = 1.14 s → × $0.0013028 = $0.0015
Batching cuts the cost per query about thirteenfold between one query and 32. Memory sets the ceiling. Each request in flight needs its own KV cache (key-value cache): the record the model keeps of every token in that request. A model is a stack of layers, each one a step of computation applied to every token. Qwen3.8 27B keeps this record in 16 of its 64 layers, in 4 key-value heads per layer: parallel slots of 256 numbers each.
Its other 48 layers use linear attention: each keeps one fixed-size summary per request instead of a record per token. That summary is 48 heads of 128 × 128 numbers per layer (linear_num_value_heads, linear_key_head_dim and linear_value_head_dim in the config). The config stores it as float32, 4 bytes per number (mamba_ssm_dtype). A KiB is 1,024 bytes. vLLM uses 92% of the GPU's memory unless you set otherwise, which is 73.6 GB of the 80.
KV cache per token = 16 layers × 4 heads × 256 numbers × 2 (key and value) × 2 bytes = 64 KiB
fixed summary = 48 layers × 48 heads × 128 × 128 numbers × 4 bytes = 151 MB
full request = (4,525 input + 400 output tokens) × 64 KiB + 151 MB = 0.47 GB
requests that fit = (73.6 GB − 55.6 GB of weights) / 0.47 GB = 38
Our KV cache guide runs the same formula for other models.
Where the bottleneck sits
The bottleneck moves as the batch grows, and that decides which lever cuts your cost. A request runs in two phases. Prefill is the first phase: the model reads the whole prompt in one pass. Decode is the second phase: the model writes the answer one token per step.
Decode is memory-bound: its speed is set by how fast the GPU reads memory, not by how fast it computes. To write one token, the GPU reads every weight once, and every request in the batch shares that read. The PagedAttention paper, which introduced vLLM's memory manager, says this sequential generation "makes the workload memory-bound, underutilizing the computation power of GPUs". The H100 reads 3.35 TB per second. Each request also adds a read of its own cache. While it writes, a request holds about 0.46 GB on average: the cache for the prompt and half the answer, plus the 151 MB summary.
weight read per step = 55.6 GB / 3,350 GB per second = 16.6 ms
cache read per request = 0.46 GB / 3,350 GB per second = 0.14 ms
batch of 1 = 16.6 ms + 1 × 0.14 ms = 16.7 ms per step, at most 60 tokens a second
batch of 4 = 16.6 ms + 4 × 0.14 ms = 17.1 ms per step
batch of 8 = 16.6 ms + 8 × 0.14 ms = 17.7 ms per step
batch of 32 = 16.6 ms + 32 × 0.14 ms = 21 ms per step, at most 48 tokens a second
These speeds are upper bounds from the datasheet, so batching costs each stream a little speed. The 45 tokens per second on our model page is a measurement on our API, which this estimate does not model.
Prefill is compute-bound: the GPU's arithmetic, not its memory, sets the pace. One forward pass, the model's full computation for one token, costs about two operations per parameter. The H100 datasheet lists 1,979 teraFLOPS, trillions of operations per second, for 16-bit math. That figure is marked "with sparsity", a shortcut for weights with a fixed pattern of zeros. A dense model gets half, so we count 989.
prefill per query = 2 × 27.8 billion × 4,525 tokens / 989 trillion per second = 0.25 s
decode, batch of 1 = 400 steps × 16.7 ms = 6.69 s
decode, batch of 32 = 400 steps × 21 ms / 32 queries = 0.26 s
prefill share = 0.25 / (0.25 + 6.69) = 4% at 1, 0.25 / (0.25 + 0.26) = 49% at 32
So decode is 96% of the GPU time for a lone query, and prefill is half of it for a full batch. Prefill grows with every retrieved passage, so the retrieved text drives cost on both routes. On our API it is 65% of the price.
Two levers follow. Send fewer passages: four instead of eight save $0.0008 per query on our API, $1,200 a month at the infrastructure guide's 1.5 million questions. Then let our cached-input price apply to the repeated prompt start. On our API, a repeated prompt start that is still in GPU memory is billed at the cached input rate. For Qwen3.8 27B that is $0.10 instead of $0.40 per million tokens. A cache hit means a request found its prompt start still in memory. So put the fixed system prompt first and the passages after it. Hits are best-effort, and the response reports the cached tokens.
RAG pipeline GPU or per-token API: the break-even
A rented GPU beats per-token pricing only when you keep it full for every hour you pay for. One H100 hour is 3,600 GPU-seconds, and dividing its cost by the per-query price gives the rate it has to sustain:
break-even = $0.0013028 per GPU-second × 3,600 seconds / $0.00277 per query = 1,693 queries an hour
Your traffic has to reach that rate around the clock. The batch also has to hold about four queries, the row in the batching table where both routes cost the same. Our recommendation is to start on per-token and price your own GPUs only once steady traffic clears that rate. The roofline leaves out idle hours, restarts, model updates and the engineer who runs the server. It also assumes the same answer quality on both routes.
Lyceum is a sovereign European AI inference provider for open-source models. Our serverless models, the open models we serve per token through one API, run in Paris and Finland. Prompts and outputs are processed, not stored, and never used for training. When your traffic steadies, a dedicated endpoint reserves GPUs for one model, with commitment pricing agreed per contract. Ask us to price it against your token bill.
To check your own numbers, send one real question with its retrieved passages and read the token counts in the response:
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["LYCEUM_API_KEY"],
base_url="https://api.lyceum.technology/openai/v1",
)
answer = client.chat.completions.create(
model="qwen/qwen3.8-27b",
messages=[
{"role": "system", "content": "Answer only from the passages."},
{"role": "user", "content": "Passages: Refunds take five working days.\n\nHow long do refunds take?"},
],
)
u = answer.usage
cost = u.prompt_tokens * 0.40 / 1e6 + u.completion_tokens * 2.40 / 1e6
print(u.prompt_tokens, u.completion_tokens, f"${cost:.6f}")
To size the GPU side, use Size a model on our GPU compute page. Enter your model, the tokens per request and the requests at once. It shows the memory needed and which GPU fits.
Frequently asked questions
Where can I see my real cost per query?
In the dashboard, Serverless › Usage shows tokens (input, output and cached) and cost per model, per member or per API key. Divide the cost by your number of questions for the period.