When you see a file like model.Q4_K_M.gguf, the Q4 is doing
a lot of work. It's the quantization level – the knob that trades model size and speed
against answer quality. In GGUF explained we covered
what the letters mean. Here's the deeper version: what quantization actually does to
the model, how much quality you give up, and how it shows up in real speed numbers.
What quantization actually does
A neural network's weights are numbers – normally stored as 16-bit floating point
(or even 32-bit). Quantization stores those numbers at lower precision. Instead of
a weight being 0.7321024…, it becomes something like
0.73, or gets snapped to one of 16 possible values (4 bits).
That sounds destructive, and it is – but the trick is that models are extremely robust to it. Neural networks have enormous redundancy, so you can throw away most of the precision in the weights before the model's behaviour meaningfully degrades. That's why a 4-bit model can still be remarkably good.
The size math is simple. A 7-billion-parameter model at 16 bits is roughly
7B × 2 bytes ≈ 14 GB. At 4 bits it's 7B × 0.5 bytes ≈ 3.5 GB.
That's the difference between "doesn't fit on my laptop" and "runs fine on my laptop."
The trade-off, quantified
Quantization level affects three things: file size, speed, and quality. The size and speed are easy to reason about; quality is the fuzzy one.
| Level | Bits per weight | Relative size | Quality |
|---|---|---|---|
| Q2_K | ~2.6 | ~25% | Poor – only for tiny budgets |
| Q3_K | ~3.4 | ~32% | Mediocre – visible degradation |
| Q4_K_M | ~4.3 | ~42% | Good – the default |
| Q5_K_M | ~4.8 | ~49% | Very good – noticeably sharper |
| Q6_K | ~6.0 | ~58% | Near-original |
| Q8_0 | ~8.0 | ~78% | Indistinguishable |
| F16 | 16 | 100% | Original weights |
Important nuance: the differences between Q4, Q5, Q6, and Q8 are mostly visible in difficult tasks – long reasoning chains, complex code, math, or factual recall where a single wrong token cascades. For casual chat and summarization, you'll struggle to tell Q4 from Q6. This is why Q4_K_M is the default for everyone: for most real use, the quality gap is imperceptible.
How it shows up in speed
Smaller weights mean faster inference, because the bottleneck is memory bandwidth – how fast the GPU can feed bytes to the compute units. Decode speed is roughly:
decode_tok/s ≈ (memory_bandwidth × 0.8) / bytes_per_token
A 7B model at Q4 needs ~0.5 bytes/weight × 7B ≈ 3.5 GB per token pass. At Q8 it's ~7 GB. So Q4 is roughly twice as fast as Q8 at decode, all else equal – and about 4× faster than F16. That's why a quantized model feels so much snappier.
You can see this effect in the VelsTech benchmark database – our tested Qwen 27B at 3.7bpw ran at ~18–22 tok/s on a 12 GB RX 6800M, and that's with partial offload. A bigger quantization would fit less and run slower.
Why k-quants (the "K" and "_K_M") are clever
Older quantizations applied the same bit width to every weight. K-quantization is smarter: it applies fewer bits to less-important weights and more bits to the ones that matter, grouped per block of weights. The result is that a "4-bit" k-quant file is often better than a naive 4-bit file of the same size – and sometimes even beats a slightly larger naive quantization.
The _S (small) and _M (medium) suffixes change how
aggressively this is applied. Q4_K_S is the smallest "good" 4-bit option;
Q4_K_M spends a bit more size for noticeably better quality. For most
people, _M is the right call.
When to go higher (and when not to)
- Stay at Q4_K_M for general chat, writing, and most coding – the default, and the best size/quality balance.
- Step up to Q5_K_M or Q6_K when the model is for reasoning-heavy or factual work, and you have the RAM/VRAM to spare.
- Skip Q8/F16 unless you're benchmarking or have huge memory. The quality gain over Q6 is minimal and you pay double the size and speed cost.
- Drop to Q3 only when the model won't fit at Q4 and you need it to work at all.
Want to know exactly what fits your card? Run the numbers through the LLM VRAM Calculator.
Bottom line
Quantization is the reason local AI works at all. It's a compression with a quality tax that's small for most tasks and significant only for hard ones. Q4_K_M is the smart default, Q5_K_M is the quality upgrade, and everything above Q6 is usually wasted on consumer hardware. Understand that and you'll never waste a download again.