So size the GPUs by the memory left after the weights, not by whether the weights fit. Llama 3.1 70B in 16-bit on two 80 GB GPUs fits, but leaves room for only about two long requests at a time.
How many GPUs does Llama 3 70B need behind an API, and do you have to run the serving software yourself? This guide sizes Llama 3.1 and Mistral Large 2, then deploys and calls the model.
Sizing GPU memory for Llama 3 and Mistral Large 2
GPU memory for Llama 3 has two parts: the weights and the KV cache. The weights are the model's learned numbers. The KV cache (key-value cache) is the record the model keeps of every token it has read in a request. Our KV cache guide explains it in depth.
The weights take a fixed amount. Each parameter takes 2 bytes in a 16-bit format and 1 byte in an 8-bit format such as FP8.
weights (GB) = parameters (billions) x bytes per parameter
Llama 3.1 8B, 16-bit: 8 x 2 = 16 GB
Llama 3.1 70B, 16-bit: 70 x 2 = 140 GB
Llama 3.1 70B, 8-bit: 70 x 1 = 70 GB
Meta publishes Llama 3.1 in 8B, 70B and 405B sizes. Each reads up to 128k tokens per request, its context window. Mistral Large 2 has 123 billion parameters and the same 128k context. In 16-bit that is 246 GB of weights.
The serving software does not hand all GPU memory to the model. vLLM, the open-source serving engine, uses 92% of each GPU's memory by default. So two 80 GB GPUs give 147.2 GB to work with. The 140 GB of weights leave 7.2 GB.
That 7.2 GB is the KV cache for every request in flight. Three numbers from the model's config file set its size per token. Layers are the stacked processing steps a token passes through. KV heads are the parts of each layer that store the cache. Head size is how many numbers each KV head stores per token. The leading 2 counts one key and one value per token, as the KV cache guide shows:
KV cache per token = 2 x layers x KV heads x head size x bytes
Llama 3.1 70B, 16-bit: 2 x 80 x 8 x 128 x 2 = 327,680 bytes
Llama 3.1 8B, 16-bit: 2 x 32 x 8 x 128 x 2 = 131,072 bytes
One 8,192-token request: 8,192 x 327,680 bytes = 2.68 GB
The 80 layers and 8 KV heads come from the Llama 3.1 70B config. The config does not list the head size directly. It lists the width of each layer, called the hidden size, as 8,192 numbers, split over 64 attention heads. 8,192 divided by 64 gives a head size of 128. The 64 attention heads share the 8 KV heads, eight to each, which keeps the cache small. That 8,192 is a coincidence: it is not the request length.
So 7.2 GB holds two requests of 8,192 tokens each. A third request waits until one finishes.
The table runs the same sum for other layouts. A100 and H100 are NVIDIA GPU models with 80 GB of memory each. The H200 has 141 GB and the B200 192 GB.
| Model and format | GPUs | Usable memory (92%) | Weights | Left for KV cache | 8,192-token requests at once |
|---|---|---|---|---|---|
| Llama 3.1 8B, 16-bit | 1 x A100 or H100 (80 GB) | 73.6 GB | 16 GB | 57.6 GB | 53 |
| Llama 3.1 70B, 16-bit | 2 x 80 GB | 147.2 GB | 140 GB | 7.2 GB | 2 |
| Llama 3.1 70B, 16-bit | 4 x 80 GB | 294.4 GB | 140 GB | 154.4 GB | 57 |
| Llama 3.1 70B, 8-bit | 1 x H200 (141 GB) | 129.7 GB | 70 GB | 59.7 GB | 22 |
| Mistral Large 2, 16-bit | 4 x 80 GB | 294.4 GB | 246 GB | 48.4 GB | not calculated |
| Mistral Large 2, 8-bit | 1 x B200 (192 GB) | 176.6 GB | 123 GB | 53.6 GB | not calculated |
These are calculated examples, not measurements. The sum leaves out activations, the temporary numbers the GPU holds while it computes each step. It also leaves out the overhead of tensor parallelism, which splits each layer across several GPUs that then exchange results. Both lower the counts. For the 8-bit rows, we assume the weights are 8-bit but the KV cache stays 16-bit. We did not calculate requests for Mistral Large 2, because its config file is gated and we did not read it. The 8B row uses its own config: 32 layers, 8 KV heads and a head size of 128. That gives 131,072 bytes per token.
Storing weights in fewer bits is called quantisation. 8-bit weights halve the memory, but they change the model's numbers, so test answer quality on your own prompts first. Our quantisation guide compares the formats. The largest size, Llama 3.1 405B, runs on one server of H200 GPUs in 16-bit, according to NVIDIA.
To skip the arithmetic, our Size a model tool reads a Hugging Face model's config and shows the memory and requests.
Deploying Llama 3 to a dedicated endpoint
A dedicated endpoint runs one model on GPUs reserved for you and gives it one OpenAI-compatible API address. A replica is one copy of the model on its own GPUs, and an endpoint can run several. You name the model by its Hugging Face model ID, and you pick a hardware profile, the GPU type each replica runs on.
Meta's Llama repositories on Hugging Face are gated: you have to accept Meta's terms before you can download the weights. For a gated model, you pass your Hugging Face token when you deploy. Our dedicated inference docs show the command for Llama 3.1 8B:
lyceum infer deploy meta-llama/Llama-3.1-8B-Instruct \
-g gpu.a100 \
--min-replicas 1 \
--max-replicas 3 \
--hf-token "$HF_TOKEN"
The replica count scales between the minimum and the maximum. Two targets decide when it scales. The first is requests per second per replica, 10 by default in the deploy flags. The second is the p95 latency, the time 95% of requests stay under, 5,000 ms by default. We balance requests across the replicas behind one address.
Check progress with lyceum infer status <deployment_id>. The deployment moves from created through provisioning to running once the first replica is healthy.
Llama 3.1 8B fits one GPU, so you can deploy it yourself with this command. The docs name A100 and H100 profiles. On those 80 GB GPUs, Llama 3.1 70B and Mistral Large 2 need more than one GPU per replica, as the table shows. For those, talk to us through our inference page, and we agree the model configuration with you before launch. The data centre is agreed for your workload too, and you can ask for a specific EU location.
Calling your Llama 3 API with the OpenAI SDK
A Llama 3 deployment answers in the OpenAI Chat Completions format, so the openai Python client works unchanged. You set the base URL to our endpoint and the model field to your deployment ID:
from openai import OpenAI
client = OpenAI(
api_key="lk_...",
base_url="https://api.lyceum.technology/api/v2/external",
)
resp = client.chat.completions.create(
model="<deployment_id>",
messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)
We are Lyceum, a sovereign European AI inference provider for open-source models. By sovereign European we mean the base URL above points at our GPUs in European data centres. Your prompts and outputs are processed, not stored, and never used for training.
LangChain, LlamaIndex and any other client that speaks this format work the same way. The deployment also accepts the OpenAI Batch API, which takes a file of many requests and returns the answers later.
Licences for Llama 3.1 and Mistral Large 2
The licence decides whether you may serve a model commercially at all. Llama 3.1 ships under the Llama 3.1 Community License, which Meta's model card describes as a custom commercial licence. Meta names commercial and research use as intended uses.
Mistral Large 2 is different. Its model card puts it under the Mistral Research License, which allows research and non-commercial use. A company may not use it in its own products without a commercial licence from Mistral. Settle this before you size GPUs for it.
Dedicated endpoint or per-token API
A dedicated endpoint is billed for the time its replicas run, not per request. A deployment with no requests for one hour scales to zero and pauses, which stops billing. The next request resumes it, and that first request waits while the replicas start.
Per-token billing follows your traffic, so quiet hours cost nothing. A dedicated endpoint is priced for the capacity you reserve, so the busier you keep it, the less each token costs. Traffic that comes and goes fits our serverless inference: open models such as Kimi, GLM, DeepSeek and Qwen, billed per token. Our models list shows which ones run there.
Deploy Llama 3 yourself when you need that exact model, your own fine-tuned copy or predictable latency. Start with the 8B command above and watch how many requests reach it. For 70B or Mistral Large 2, bring the model, your request length and your peak traffic to us, and we size the replicas with you.
Frequently asked questions
Can I deploy a fine-tuned or private Llama 3 model?
Yes. A dedicated endpoint takes any Hugging Face model ID, and a token gives access to gated or private repositories.
How do I know a replica is healthy?
Each replica reports a status of pending, running, stopped or failed. A health flag shows the result of the regular health checks.
Can one endpoint serve Llama 3 and Mistral Large 2?
No. Each deployment runs one model. Deploy the second model as its own deployment, with its own deployment ID.