# GGUF explained: the format that makes local LLMs work

> GGUF is the file format that makes running LLMs locally possible. Here's what it is, what quantization levels mean, and how to pick the right one for your hardware.

*Source: https://velstech.net/gguf-explained · Updated: 2026-08-30 · Category: AI · Tags: GGUF, Quantization, Local AI*

*Markdown version of [GGUF explained: the format that makes local LLMs work](https://velstech.net/gguf-explained). [Read the full guide with interactive tools](https://velstech.net/gguf-explained).*
*Also as Markdown: [Hindi](https://velstech.net/gguf-explained.hi.md) · [Tamil](https://velstech.net/gguf-explained.ta.md).*

---

If you've downloaded a model to run locally – with Ollama, llama.cpp, or anything
else – you've almost certainly run into the letters **GGUF**.
It's the file extension on most open-weight models you can actually run yourself:
`llama-3.2-3b.Q4_K_M.gguf`, `qwen2.5-7b.Q5_K_M.gguf`, and so on.

GGUF is the reason local AI is possible at all. Here's what it actually is, what those
`Q4_K_M` labels mean, and how to pick the right one for your hardware.

## What is GGUF?

GGUF is a **file format for storing quantized LLMs** – a container that
packs a model's weights, tokenizer, and metadata into a single file that llama.cpp and
its derivatives (Ollama, LM Studio, etc.) can load directly. It's the successor to
GGML, and it's become the de facto standard for running models on consumer hardware.

The clever part is what "quantized" means. A model's weights are normally stored as
16-bit floating-point numbers. GGUF files store them at far lower precision – 8-bit,
4-bit, even 2-bit – which shrinks the file dramatically. That's what lets a 27-billion
parameter model fit in 12 GB of VRAM instead of needing 50+ GB.

## The trade-off: size vs quality

Quantization is a compression with a price. Lower precision = smaller file + faster
inference + lower quality. Higher precision = bigger file + slower + better quality.
The skill is finding the sweet spot where quality holds up but the model fits your
hardware.

GGUF files encode this in their name. Take `qwen2.5-7b.Q4_K_M.gguf`:

- Q4 – 4-bit quantization (the "Q" stands for quantization).

- K – "k-quant" (k-quantization), a smarter scheme that uses fewer bits on less important weights.

- M – the size: S (small), M (medium), L (large). Larger = closer to full quality.

So `Q4_K_M` means "4-bit k-quant, medium" – the most popular all-rounder,
and the default recommendation for most people.

## Common quantization levels

| Level | Approx. size vs full | Quality | When to use it |
| --- | --- | --- | --- |
| Q2_K | ~25% | Poor | Only when you must squeeze into tiny RAM |
| Q3_K | ~32% | Meh | Very tight budgets |
| Q4_K_S / Q4_K_M | ~40–44% | Good | Default choice – best size/quality balance |
| Q5_K_M | ~49% | Very good | When you have headroom; notably sharper |
| Q6_K | ~58% | Near original | Plenty of RAM, want quality |
| Q8_0 | ~78% | ~Indistinguishable | Only for the KV cache or when size is no object |
| F16 / F32 | 100% | Original | Rarely needed for inference; huge files |

## Which one should you download?

The rule of thumb is simple: **start with Q4_K_M.** It's the best
quality-per-gigabyte, runs on most hardware, and for most tasks the quality difference
from Q5 or Q6 is hard to notice. If the model runs comfortably and you have spare RAM,
try Q5_K_M and see if you can tell the difference. If it's too slow or doesn't fit,
drop to Q4_K_S or Q3_K_M.

To figure out exactly what fits your card, run the numbers through the
[LLM VRAM Calculator](https://velstech.net/llm-vram-calculator) – it accounts for weights,
KV cache, and quantization so you know before you download.

## GGUF + llama.cpp

GGUF files are designed to be run by llama.cpp – the engine underneath Ollama and
LM Studio. You rarely interact with GGUF files directly if you use Ollama (it handles
the download and format for you), but the concepts are identical: you're always
choosing a model size and quantization. See the
[llama.cpp guide](https://velstech.net/llama-cpp-guide) for the command-line details.

## Bottom line

GGUF is the container that makes local AI practical. The letters in the filename tell
you everything: `Q4_K_M` is the safe default, `Q5_K_M` is a
worthwhile upgrade if you have headroom, and anything smaller is for squeezing into
tight hardware. Pick Q4_K_M, check it against the
[VRAM calculator](https://velstech.net/llm-vram-calculator), and you'll have a local model
running in minutes.

## FAQ

**What is a GGUF file?**

GGUF is a container format for quantized LLMs – it packs a model's weights, tokenizer, and metadata into one file that llama.cpp, Ollama, and LM Studio can load directly. The quantization is what shrinks a model enough to run on consumer hardware.

**Which GGUF quantization should I download?**

Q4_K_M is the safe default – the best quality-per-gigabyte. If you have spare RAM, try Q5_K_M for noticeably sharper output. Drop to Q3 or Q2 only if the model doesn't fit.

---

*VelsTech – technology explained for everyone. Original: https://velstech.net/gguf-explained*