GGUF is the single-file format of llama.cpp, an open-source program that runs models on laptops, CPUs and Apple Silicon. GPTQ and AWQ are 4-bit checkpoints (saved model weights) that vLLM, an engine that serves one model to many users on GPUs, loads directly. FP8, an 8-bit number format, is the third choice on L40S and H100 GPUs.
The format also sets how many conversations fit next to the model on one GPU.
So which file should you download: the original 16-bit model, or its GGUF, GPTQ, AWQ or FP8 version? The question comes up the moment you leave a hosted API and run an open model yourself.
What GGUF, GPTQ, AWQ and FP8 files contain
Quantization means storing each weight of a model in fewer bits than the 16 it was released in. A weight is one of the billions of learned numbers inside the model. Fewer bits make the file smaller, so it needs less VRAM, the memory on the GPU.
The four formats differ in three ways: which program reads them, which GPUs run them fast, and how the small numbers were chosen. GPTQ and AWQ first run sample text, called calibration data, through the model to decide how to round each weight. Hugging Face's quantization guide splits methods the same way: some need calibration, others quantize on the fly.
Rounding costs some quality. The llama.cpp project measures that loss as perplexity, a score of how surprised the model is by real text, where lower is better.
| Format | Program that runs it | Bits per weight | NVIDIA GPUs in vLLM |
|---|---|---|---|
| GGUF Q4_K_M | llama.cpp | 4.89 on Llama 3.1 8B | Experimental plugin only |
| GPTQ | vLLM, Transformers (Hugging Face's Python library) | 4, plus a scale per group | Volta (V100) to Hopper (H100) |
| AWQ | vLLM, Transformers | 4, plus a scale per group | Turing (T4) to Hopper (H100) |
| FP8 | vLLM, SGLang (another serving engine) | 8, plus 16-bit layers | Ada (L40S) and Hopper (H100) |
Volta, Turing, Ampere, Ada and Hopper are NVIDIA's GPU generations, from oldest to newest. The A100 is Ampere.
GGUF runs on llama.cpp, CPUs and Apple Silicon
GGUF files run on llama.cpp, whose README lists Apple Silicon as a first-class target through Apple's Metal graphics interface. It also splits a model between CPU and GPU when the model is larger than the GPU's memory.
The common 4-bit GGUF type is Q4_K_M. Q4 means 4-bit, K names llama.cpp's block-wise k-quant scheme, and M is its medium variant. In llama.cpp's own table for Llama 3.1 8B, it stores 4.89 bits per weight on average. The file shrinks from 14.96 GiB in 16-bit to 4.58 GiB.
One command downloads a GGUF file from Hugging Face and starts a server with an OpenAI-compatible API. UD in the file name marks Unsloth's dynamic variant, which keeps some layers at higher precision.
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_MGGUF is a weak fit for serving many users from a GPU. The vLLM documentation calls its GGUF support "highly experimental and under-optimized", and it now ships as a separate plugin. For a shared GPU endpoint, use one of the next two formats.
GPTQ and AWQ are 4-bit checkpoints for vLLM on GPUs
GPTQ and AWQ both store most weights as 4-bit integers. For each small group of weights, they also keep a scale, a multiplier that restores the original range. They differ in how they decide the rounding.
GPTQ rounds the weights layer by layer in one pass. It uses approximate second-order information: an estimate of how much the layer's output changes when several weights move together. GPTQ uses it to pick the rounding that changes the output least. Its authors quantized a 175-billion-parameter model in about four GPU hours, at 3 or 4 bits per weight.
AWQ starts from activations, the values that flow through a layer when it reads real text. Its authors found that protecting about 1% of the weights, the salient ones, cuts most of the rounding error. AWQ finds them from the activations. It multiplies those weights by a scale before rounding and divides the matching activations by it, so the product stays the same. A larger number loses a smaller share of its value when rounded, and every weight stays 4-bit. The authors say this avoids overfitting to the calibration text.
The 4-bit label does not describe the whole file. The most downloaded AWQ checkpoint of Qwen3.8 27B on Hugging Face is 21.02 GB, and its GPTQ counterpart is 19.42 GB. Both leave the model's linear-attention layers in 16-bit. A full-attention layer keeps a record of every token it has read. A linear-attention layer squeezes all of it into a fixed-size state instead. The configuration files list those layers as ignored, which means not quantized.
AWQ: 21.02 GB × 8 bits ÷ 27.78 billion weights = 6.05 bits per weight
GPTQ: 19.42 GB × 8 bits ÷ 27.78 billion weights = 5.59 bits per weightBoth files keep those layers in 16-bit, so the gap between them comes from the scales. The AWQ file keeps a scale for every 32 weights, the GPTQ file one for every 128.
vLLM reads the quantization method from the checkpoint's configuration file, so no extra flag is needed.
vllm serve cyankiwi/Qwen3.8-27B-AWQ-INT4FP8 needs an Ada or Hopper GPU
FP8 stores each weight as an 8-bit floating-point number. A floating-point number keeps a small exponent, so very small and very large values both survive the rounding. Qwen, the model's authors, publish their own FP8 checkpoint of Qwen3.8 27B.
That file is not all FP8 either. Hugging Face lists 24.70 billion parameters in FP8 and 3.08 billion kept in BF16, the 16-bit format of the original.
FP8 part: 24.70 billion × 1 byte = 24.70 GB
BF16 part: 3.08 billion × 2 bytes = 6.16 GB
Total: 30.87 GB = 8.89 bits per weightvLLM calls this format FP8 W8A8: 8-bit weights and 8-bit activations. It runs it only on Ada GPUs such as the L40S and Hopper GPUs such as the H100. An A100 needs AWQ or GPTQ instead. The model card gives the serving command:
vllm serve Qwen/Qwen3.8-27B-FP8How many conversations each format leaves room for
GPU memory holds two things: the weights, and a KV cache (key-value cache) for every conversation. The KV cache is the record the model keeps of every token it has read. Whatever the weights leave free decides how many conversations run at once. By default, vLLM uses 92% of the GPU's memory.
Qwen3.8 27B mixes two layer types, as its configuration file shows. Its 16 full-attention layers keep a record of every token. KV heads are the parts of a layer that hold that record. Each layer has 4 of them, and each stores one key and one value of 256 numbers per token. The other 48 layers use linear attention. Each keeps a fixed state per conversation: 48 heads, each a grid of 128 × 128 numbers. Our KV cache guide explains the parts.
KV cache per token = 16 layers × 2 (key, value) × 4 KV heads × 256 numbers × 2 bytes = 65,536 bytes (64 KiB)
Fixed state = 48 layers × 48 heads × 128 × 128 numbers × 2 bytes = 75.5 MB
One conversation = 8,192 tokens × 64 KiB = 536.9 MB, plus 75.5 MB fixed state = 0.612 GB
Room on an H100 = 80 GB × 0.92 = 73.60 GB, minus 21.02 GB of AWQ weights = 52.58 GB
Conversations = 52.58 GB ÷ 0.612 GB = 85The same steps for each file give this table. It is a calculated example for conversations of 8,192 tokens.
| Weights file (Qwen3.8 27B) | Size | Conversations on H100 (80 GB) | Conversations on L40S (48 GB) |
|---|---|---|---|
| BF16, original | 55.56 GB | 29 | Does not fit |
| FP8, from Qwen | 30.87 GB | 69 | 21 |
| AWQ 4-bit, community | 21.02 GB | 85 | 37 |
| GPTQ 4-bit, community | 19.42 GB | 88 | 40 |
The count is an upper bound. It leaves out activations, the working memory each step needs. It also leaves out CUDA graphs, recorded GPU operations that vLLM replays to speed up small batches. It also says nothing about answer quality or speed. If vLLM still runs out of memory, our guide to vLLM out-of-memory errors covers the settings.
Which quantization format to run where
Pick the format by where the model runs, then check the memory it leaves:
- Laptop, CPU or Mac: GGUF Q4_K_M with llama.cpp.
- GPU server with A100s: AWQ or GPTQ with vLLM.
- GPU server with L40S or H100: FP8 when the model author publishes it, 4-bit when you need room for more conversations.
- No file to manage: call the model per token instead.
We are Lyceum, a sovereign European AI inference provider for open-source models. Serverless models run in Paris and Finland, and prompts and outputs are processed, not stored, and never used for training. Our serverless inference is a shared API billed per token. There, Qwen3.8 27B costs $0.40 per 1M input tokens and $2.40 per 1M output tokens (read 30 October 2026).
To run an AWQ, GPTQ or FP8 file yourself, deploy it as a dedicated endpoint: the model on GPUs reserved for you. The deploy command takes any Hugging Face model ID, so a quantized checkpoint deploys like the original. The hardware profile sets the GPU type: AWQ and GPTQ run on the A100 or H100 profile, FP8 only on the H100. A deployment with no requests for an hour pauses and stops billing for idle replicas, the copies of the model that serve requests.
lyceum infer deploy cyankiwi/Qwen3.8-27B-AWQ-INT4 \
-g gpu.h100 \
--min-replicas 1 \
--max-replicas 2For GGUF files with llama.cpp, or your own vLLM build, our GPU virtual machines give you root access, billed per second. The AWQ and GPTQ files above are community uploads, so test their answers on your own prompts first.
Pick the file for your GPU, then follow the dedicated endpoint guide to deploy it on an H100.
Frequently asked questions
Is GGUF Q4_K_M the same as a 4-bit AWQ file?
No. They round weights differently and need different programs: llama.cpp reads GGUF, and vLLM reads AWQ. In llama.cpp's own table, Q4_K_M averages 4.89 bits per weight rather than 4.