# KV cache explained: why context windows eat memory

> Why longer context windows eat memory, how the key-value cache grows with tokens, and how to calculate it for your own GPU.

*Source: https://velstech.net/kv-cache-explained · Updated: 2026-08-30 · Category: AI · Tags: KV Cache, VRAM, Local AI*

*Markdown version of [KV cache explained: why context windows eat memory](https://velstech.net/kv-cache-explained). [Read the full guide with interactive tools](https://velstech.net/kv-cache-explained).*
*Also as Markdown: [Hindi](https://velstech.net/kv-cache-explained.hi.md) · [Tamil](https://velstech.net/kv-cache-explained.ta.md).*

---

"This model supports 128K context" sounds great – until you try to run it and wonder
where all your memory went. The culprit is the **KV cache**: the part of
an LLM that grows with every token of context, and the reason a big context window
costs real gigabytes.

This guide explains what the KV cache is, why it scales the way it does, and how to
calculate exactly how much memory a context window needs on *your* hardware.

## What the KV cache is

When an LLM processes a token, it does a lot of computation – and crucially, it needs
the results from *earlier* tokens to make sense of the later ones. Each token's
attention to the past is computed from two things per layer: a **key**
(what this token offers) and a **value** (what it contains).

Rather than recomputing those keys and values for every previous token on every new
token – which would make generation quadratically slow – the model stores them. That
store is the **KV cache**. Every token in the context adds a fixed amount
to it, across every layer and every attention head.

## Why it eats memory linearly with context

Here's the key fact: **the KV cache grows with the number of tokens, not the
size of the model.** More context = more cached keys and values. Double the
context and you roughly double the KV cache.

The rough size for a dense model is:

```
KV cache ≈ layers × kv_heads × head_dim × 2 × bytes_per_value × context
```

The exact numbers depend on the model architecture. The practical point: for a
mid-size model like a 7B or 13B, the KV cache at a large context (say 32K or 128K)
can easily match or exceed the size of the weights themselves.

## Real numbers

To see this in practice, the [LLM VRAM Calculator](https://velstech.net/llm-vram-calculator)
breaks down weights + KV cache for your exact model, quantization, and context. A
couple of worked examples:

- A 7B model at Q4 with 8K context: KV cache is roughly 1 GB – a small slice of the ~4 GB total.

- The same model at 32K context: KV cache jumps to ~4 GB, now matching the weights.

- A 32B model at Q4 with 64K context: the KV cache alone can be 8–12 GB – more than many GPUs have.

That's why "runs at 8K context" and "runs at 128K context" are completely different
hardware requirements, even for the same model.

## How to reduce it

If the KV cache is eating your memory, you have a few levers:

- Lower the context window (-c) – the single biggest lever. Use the context you need, not the maximum the model supports.

- Quantize the KV cache (q8_0, q4_0) – storing keys/values at 8-bit or 4-bit roughly halves or quarters the cache, with minimal quality loss for most tasks.

- Use MoE or long-context-optimized models – MoE models like the Ornith 35B share KV across tokens, making the cache smaller than a dense model's.

- Offload – llama.cpp can keep part of the KV cache in system RAM, at the cost of speed.

You can see the q8 KV effect in our [Qwen
27B Lab test](https://velstech.net/qwen-27b-ridge-rocm-vs-vulkan), where q8 KV at 16K fit (just barely) on a 12 GB card with partial
offload.

## MoE vs dense (why they differ)

A dense model processes every token through all its parameters. An MoE model routes
each token through only a few "expert" layers – but the KV cache is a different story.
Because attention is shared across tokens regardless of which experts handled them,
MoE models can keep a *smaller* KV cache per token in some configurations.
That's one reason a 35B MoE can run at 262K context where a 35B dense model never could.
See the [MoE vs Dense Lab test](https://velstech.net/moe-vs-dense-rx6800m-16k-vs-262k) for
the head-to-head.

## Bottom line

The KV cache is the hidden memory cost of LLMs, and it grows with every token of
context. When a model says it "supports 128K," that's an upper limit, not a
recommendation. Use the context you actually need, quantize the KV cache, and check
the [VRAM calculator](https://velstech.net/llm-vram-calculator) before you commit – and
your model will run in a fraction of the memory.

## FAQ

**Does the KV cache affect speed?**

Yes – a larger KV cache means more data to read per token, which slows decode. Quantizing the KV cache (q8, q4) helps both memory and speed.

**How much memory does the KV cache use?**

It depends on the model architecture and context. For a 7B model at 8K context, roughly 1 GB. At 32K, about 4 GB. At 128K, it can be 12 GB or more.

---

*VelsTech – technology explained for everyone. Original: https://velstech.net/kv-cache-explained*