The KV cache grows with every token of context. This calculator shows how much memory the cache will use at different context lengths and quantizations, so you know what fits your GPU before you run the model.
Why the KV cache matters. Every time an LLM generates a token, it needs to remember all previous tokens in the conversation. The key-value (KV) cache stores this information, and its size scales linearly with context length. A 32K context uses roughly twice the cache memory of a 16K context. For a 7B model with 8 KV heads, the cache alone can use 2–4 GB at 32K context – and that's before the model weights.
How to reduce KV cache memory. Modern backends like llama.cpp support KV cache quantization (q8 or q4) which halves or quarters the cache size with minimal quality impact. Some models also use grouped-query attention (GQA), which reduces the number of KV heads and directly cuts cache memory. This calculator lets you adjust heads, layers, and quantization to see the impact on your specific setup.
Rough estimate: layers × kv_heads × head_dim × 2 × bytes_per_value × context.
For precise numbers including model weights, use the
LLM VRAM Calculator.
See the KV cache guide for how to reduce it.