Q4_K_M vs Q8_0: which GGUF file to download
Open a model on Hugging Face, or in LM Studio, and it comes as a dozen GGUF files from Q2_K to Q8_0. They are the same model compressed by different amounts: the fewer bits each weight keeps, the smaller and faster the file, and the more it loses. Here is what the names mean and how to choose.
Updated 9 November 2026
The short answer
- Q4_K_M is the usual choice: close to the original at under a third of its size.
- If a bigger one fits in your graphics card’s memory with room to spare, Q5_K_M or Q6_K is closer still. Q8_0 is as good as the original, and worth it mainly for small models.
- Go below 4 bits only to fit a large model, 30B parameters or more: small models lose much more from heavy compression than large ones.
- Whatever you choose, all of it should fit on the card. A Q4 that fits runs several times faster than a Q8 that spills onto the processor.
How big each one is
Sizes for Llama 3.1 8B, from llama.cpp’s own measurements. Other models scale with their size: a model with four times the parameters takes four times the memory at the same setting.
| File | Bits per weight | Llama 3.1 8B | Quality |
|---|---|---|---|
| F16 / BF16 | 16 | 15.0 GB | The original weights |
| Q8_0 | 8.5 | 8.0 GB | As good as the original |
| Q6_K | 6.6 | 6.1 GB | Practically as good as the original |
| Q5_K_M | 5.7 | 5.3 GB | Very close to the original |
| Q4_K_M | 4.9 | 4.6 GB | Close to the original; the usual download |
| Q4_K_S | 4.7 | 4.4 GB | A little below Q4_K_M |
| IQ4_XS | 4.5 | 4.2 GB | About Q4_K_S, slightly smaller |
| Q3_K_M | 4.0 | 3.7 GB | Noticeably worse, more so in small models |
| Q2_K | 3.2 | 3.0 GB | Clearly worse; only for very large models |
The file is not all the memory a model needs. The conversation it holds takes more on top — from a few hundred megabytes for a short chat to many gigabytes for a long one — and the graphics driver keeps about a gigabyte. Can my PC run AI? adds it all up for any model and card.
What the names mean
- Q and a number: quantized, to about that many bits per weight. The original model uses 16.
- K: the k-quant method, which compresses weights in small blocks, each with its own scale, and gives the parts of the model that matter most a few more bits. Nearly every file people download now is a K or an I file.
- S, M and L: small, medium and large versions of the same level, which differ in how many of those important parts get the extra bits. M is the usual balance.
- _0 and _1: the older, simpler method. Q8_0 is still the standard 8-bit file; Q4_0 has been overtaken by Q4_K_M.
- IQ: newer methods that keep more quality at very low sizes, using a sample of text to decide what to keep. They shine at 2 and 3 bits, and can run a little slower on processors and older cards.
- UD, XL and similar: variants from particular uploaders, which mix levels layer by layer. Their pages say what they did.
Q4_K_M or Q4_K_S?
Q4_K_M, unless Q4_K_S is what makes the model fit: S is about 5% smaller and a little worse. The same goes for Q5_K_M and Q5_K_S.
Q4_K_M or Q5_K_M, Q6_K or Q8_0?
Each step up is a little closer to the original and takes more memory: Q5_K_M about 17% more than Q4_K_M, Q6_K about a third more, Q8_0 about three quarters more. It is also slower, because a model has to read all its weights for every word it writes: on the same card, a Q8_0 writes at a little over half the speed of a Q4_K_M. So take the biggest one that fits on the card with room for the conversation — but if the bigger one would push part of the model off the card, the smaller one is far better.
Below 4 bits
A model with 30 billion or more parameters at 3 bits usually beats a smaller model at 4 bits, and the very largest models are often run at 2 bits. A 7B or 8B model at 3 bits loses noticeably, and at 2 bits a lot; for those, a smaller model at Q4_K_M is the better choice.
Models that come one way
Some models are released already compressed, such as gpt-oss in MXFP4, and come in one size. Their Q4_K_M and Q8_0 files on Hugging Face are about the same size as the original, so take the file the maker published.
In Ollama and LM Studio
- Ollama: a model’s default tag is usually Q4_K_M. Its page on ollama.com lists the
others, such as tags ending in
q8_0. - LM Studio: the download list for a model shows each file with its size; choose by the sizes above.
How Bottleneck helps
Can my PC run AI? picks the best compression that fits your graphics card and memory for each of 88 models, and Bottleneck’s free app fills in your parts from a scan. Its Check my local AI sees whether the model you run actually fits on the card.