Pick a GPU and a model, and this estimates how fast local inference will actually feel: decode speed (tokens per second while the model answers – this is memory-bandwidth bound) and prompt processing speed (compute bound). Estimates only – real numbers depend on your engine, drivers, and settings.
Decode assumes weights + KV cache stay in VRAM and uses ~80% of memory bandwidth; prompt processing assumes ~55% compute efficiency. MoE models compute on their active parameters, so real prefill can be faster than shown.
How we calculate this – assumptions & formulas
Real performance depends on engine, kernels, driver, and cooling. These are memory- and compute-bound upper bounds scaled by realized efficiency, not benchmarks.
1. Does it fit?
weights = params × bytes_per_quant / GiB kv_cache ≈ params × 16384 × context / GiB total = weights + kv_cache
KV here uses a rough 8B-arch approximation (~128 KB per token for 8B). If total > vram we mark won't fit; real KV from layers/heads/dim is more precise – use the VRAM calculator for that.
2. Decode speed (memory bound)
bytes_per_token = weights × GiB + kv_per_token decode_tok/s = (memory_BW × 0.8) / bytes_per_token
Every generated token reads the whole model + KV. 0.8 models ~80% achievable bandwidth; quantization helps because fewer bytes per token.
3. Prefill / prompt processing (compute bound)
prefill_tok/s = (TFLOPS × 0.55) / (2 × params)
Prompt processing is a large matmul: ~2 FLOPs per parameter per token. 0.55 models 55% compute utilization. MoE models use active params, so actual prefill can be higher.
4. Times
TTFT = prompt_tokens / prefill_tok/s gen_time = answer_tokens / decode_tok/s
See also: How much VRAM do you need? and Best GPU for local LLMs.