Pick a model and quantization, set your context length, and this estimates the total GPU memory you'll need – model weights plus the KV cache plus a small overhead. Enter your GPU's VRAM to get a straight runs / partial offload / won't fit verdict. Runs entirely in your browser.
Rough estimates. Real usage varies with the exact model build, tokenizer, CUDA/ compute graph overhead, and your inference engine. KV cache assumes FP16.
How we calculate this – assumptions & formulas
This is an estimate, not a lab measurement. Real usage varies with tokenizer, model build, CUDA graph, and your inference engine (llama.cpp, vLLM, etc.). Numbers below match the live calculator logic.
1. Model weights
weights = parameters × bytes_per_weight / GiB
Bytes per weight comes from quantization: FP32 4.0, FP16/BF16 2.0, Q8_0 1.0625, Q6_K 0.8125, Q5_K_M 0.66, Q4_K_M 0.61, Q4_0 0.56, INT4 0.5, INT8 1.0. Example: 8B params at Q4_K_M ≈ 8e9 × 0.61 / 1.07e9 ≈ 4.5 GB.
2. KV cache
kv_per_token = 2 × layers × kv_heads × head_dim × 2 bytes kv_cache = kv_per_token × context × batch / GiB
The ×2 covers K and V; the trailing ×2 bytes assumes FP16 KV (the default for most engines). For a 32-layer, 8-KV-head, 128-dim model at 8192 context and batch 1, that's ~134 MB per 1K tokens. Long context dominates memory – 32K needs ~4× the KV of 8K.
3. Overhead
overhead = (weights + kv_cache) × 0.06 + 0.25 GB
6% covers CUDA context / engine buffers, plus a fixed 0.25 GB floor. Small but noticeable on 8 GB cards.
4. Verdict
total = weights + kv_cache + overhead. Fits if total ≤ vram × 0.95. Partial offload if spill fits in ram × 0.9; otherwise won't fit. Full details: How much VRAM do you need?