# Quantization deep dive: how Q4, Q5, and Q8 change quality and speed

> How quantization levels (Q4, Q5, Q8) affect LLM quality, size, and speed – with real numbers and practical advice for local AI.

*Source: https://velstech.net/quantization-deep-dive · Updated: 2026-08-30 · Category: AI · Tags: Quantization, GGUF, Local AI*

*Markdown version of [Quantization deep dive: how Q4, Q5, and Q8 change quality and speed](https://velstech.net/quantization-deep-dive). [Read the full guide with interactive tools](https://velstech.net/quantization-deep-dive).*
*Also as Markdown: [Hindi](https://velstech.net/quantization-deep-dive.hi.md) · [Tamil](https://velstech.net/quantization-deep-dive.ta.md).*

---

When you see a file like `model.Q4_K_M.gguf`, the `Q4` is doing
a lot of work. It's the quantization level – the knob that trades model size and speed
against answer quality. In [GGUF explained](https://velstech.net/gguf-explained) we covered
what the letters mean. Here's the deeper version: what quantization actually does to
the model, how much quality you give up, and how it shows up in real speed numbers.

## What quantization actually does

A neural network's weights are numbers – normally stored as 16-bit floating point
(or even 32-bit). Quantization stores those numbers at lower precision. Instead of
a weight being `0.7321024…`, it becomes something like
`0.73`, or gets snapped to one of 16 possible values (4 bits).

That sounds destructive, and it is – but the trick is that **models are
extremely robust to it**. Neural networks have enormous redundancy, so you can
throw away most of the precision in the weights before the model's behaviour meaningfully
degrades. That's why a 4-bit model can still be remarkably good.

The size math is simple. A 7-billion-parameter model at 16 bits is roughly
`7B × 2 bytes ≈ 14 GB`. At 4 bits it's `7B × 0.5 bytes ≈ 3.5 GB`.
That's the difference between "doesn't fit on my laptop" and "runs fine on my laptop."

## The trade-off, quantified

Quantization level affects three things: **file size**,
**speed**, and **quality**. The size and speed are easy to
reason about; quality is the fuzzy one.

| Level | Bits per weight | Relative size | Quality |
| --- | --- | --- | --- |
| Q2_K | ~2.6 | ~25% | Poor – only for tiny budgets |
| Q3_K | ~3.4 | ~32% | Mediocre – visible degradation |
| Q4_K_M | ~4.3 | ~42% | Good – the default |
| Q5_K_M | ~4.8 | ~49% | Very good – noticeably sharper |
| Q6_K | ~6.0 | ~58% | Near-original |
| Q8_0 | ~8.0 | ~78% | Indistinguishable |
| F16 | 16 | 100% | Original weights |

**Important nuance:** the differences between Q4, Q5, Q6, and Q8 are
mostly visible in *difficult* tasks – long reasoning chains, complex code,
math, or factual recall where a single wrong token cascades. For casual chat and
summarization, you'll struggle to tell Q4 from Q6. This is why Q4_K_M is the default
for everyone: for most real use, the quality gap is imperceptible.

## How it shows up in speed

Smaller weights mean faster inference, because the bottleneck is memory bandwidth –
how fast the GPU can feed bytes to the compute units. Decode speed is roughly:

```
decode_tok/s ≈ (memory_bandwidth × 0.8) / bytes_per_token
```

A 7B model at Q4 needs ~0.5 bytes/weight × 7B ≈ 3.5 GB per token pass. At Q8 it's
~7 GB. So **Q4 is roughly twice as fast as Q8 at decode, all else equal** –
and about 4× faster than F16. That's why a quantized model feels so much snappier.

You can see this effect in the [VelsTech benchmark
database](https://velstech.net/benchmarks/index) – our tested Qwen 27B at 3.7bpw ran at ~18–22 tok/s on a 12 GB RX 6800M,
and that's with partial offload. A bigger quantization would fit less and run slower.

## Why k-quants (the "K" and "_K_M") are clever

Older quantizations applied the same bit width to every weight. K-quantization is
smarter: it **applies fewer bits to less-important weights** and more bits
to the ones that matter, grouped per block of weights. The result is that a "4-bit"
k-quant file is often *better* than a naive 4-bit file of the same size –
and sometimes even beats a slightly larger naive quantization.

The `_S` (small) and `_M` (medium) suffixes change how
aggressively this is applied. `Q4_K_S` is the smallest "good" 4-bit option;
`Q4_K_M` spends a bit more size for noticeably better quality. For most
people, `_M` is the right call.

## When to go higher (and when not to)

- Stay at Q4_K_M for general chat, writing, and most coding – the default, and the best size/quality balance.

- Step up to Q5_K_M or Q6_K when the model is for reasoning-heavy or factual work, and you have the RAM/VRAM to spare.

- Skip Q8/F16 unless you're benchmarking or have huge memory. The quality gain over Q6 is minimal and you pay double the size and speed cost.

- Drop to Q3 only when the model won't fit at Q4 and you need it to work at all.

Want to know exactly what fits your card? Run the numbers through the
[LLM VRAM Calculator](https://velstech.net/llm-vram-calculator).

## Bottom line

Quantization is the reason local AI works at all. It's a compression with a quality
tax that's small for most tasks and significant only for hard ones. Q4_K_M is the
smart default, Q5_K_M is the quality upgrade, and everything above Q6 is usually
wasted on consumer hardware. Understand that and you'll never waste a download again.

## FAQ

**Is Q4 much worse than Q8?**

For most tasks – chat, summarization, code generation – the difference between Q4_K_M and Q8 is imperceptible. The gap only shows in hard reasoning, math, or factual recall where a single wrong token cascades.

**Which quantization should I use?**

Q4_K_M is the default for everyone. Step up to Q5_K_M if you have headroom and want the best quality. Drop to Q3 only if the model won't fit at Q4.

---

*VelsTech – technology explained for everyone. Original: https://velstech.net/quantization-deep-dive*