Pick a GPU, a model and quantization, then drag the context slider. The bar chart shows how your VRAM budget breaks down: model weights, KV cache, and overhead. A green verdict means everything fits on the GPU; yellow means partial CPU offload; red means it won't run at all.
Decode speed assumes ~80% memory bandwidth utilization and that the full model fits in VRAM. KV cache assumes FP16 with 2 bytes per value. Overhead is ~6% of weights + KV cache. Real values vary by engine, drivers, and architecture.
How we calculate this – assumptions & formulas
This is a visual budget planner, not a lab measurement. Real usage varies with tokenizer, model build, CUDA graph, and inference engine.
1. Model weights
weights (GB) = parameters × bytes_per_weight / GiB
Bytes per weight: Q4 ≈ 0.56, Q8 ≈ 1.0625, 3.7 bpw ≈ 0.4625, etc.
2. KV cache
kv_per_token = 2 × layers × kv_heads × head_dim × 2 bytes kv_cache (GB) = kv_per_token × context / GiB
Assumes FP16 KV cache (2 bytes per value). Architecture estimates vary by model size.
3. Overhead
overhead = (weights + kv_cache) × 0.06
6% covers CUDA context, engine buffers, and small runtime allocations.
4. Verdict
Fits if total ≤ VRAM × 0.95. Partial offload if spill ≤ 16 GB RAM. Otherwise won't fit.
5. Speed estimate
decode (tok/s) = (bandwidth × 0.8) / (weights + kv_per_token / GiB) prefill (tok/s) = (TFLOPS × 0.55) / (2 × params)
Decode is memory-bandwidth bound; prefill is compute bound.