"What GPU do I need for AI?" is the wrong first question. The right one is "will an LLM run on the machine I already have?" – and the answer is a checklist, not a product pick. VRAM decides whether a model loads; system RAM, CPU, storage, and power decide whether it runs well. This guide walks the full checklist in order, with the numbers from our own VRAM guide and Lab benchmarks on a 12 GB RX 6800M.
If you finish this and decide you need a new card instead, the best GPU for local LLMs picks up where this leaves off. If you want the exact number for your setup, the LLM VRAM Calculator does the math with a fit verdict for your GPU.
The short version
- VRAM first: 8 GB runs 7B, 12–16 GB runs 14B, 24 GB runs 32B – all at 4-bit quantization. Everything else is secondary.
- System RAM second: 16 GB minimum, 32 GB if you ever want CPU offload as a fallback.
- CPU third: 6+ modern cores; it sets prompt speed and carries offloaded layers.
- Storage fourth: model files run 5–40 GB each – a 512 GB NVMe SSD is the practical minimum.
- PSU and cooling last: a loaded GPU draws its full TDP for hours; a tired PSU causes the weirdest "AI crashes."
1. VRAM: the gatekeeper
A model either fits in VRAM or it doesn't – there is no "minimum settings" mode that
makes a 70B model run on an 8 GB card. At the community-standard
Q4_K_M quantization, plus ~1–3 GB for KV cache and overhead:
| Model size | Q4 weights | Total needed | Fits on |
|---|---|---|---|
| 7–8B | ~4–5 GB | ~6 GB | 8 GB card (RTX 4060, RX 7600) |
| 13–14B | ~9–10 GB | ~11 GB | 12–16 GB card |
| 27–32B | ~18–21 GB | ~22 GB | 24 GB card (RTX 4090, RX 7900 XTX) |
| 70B | ~40–45 GB | ~48 GB | Datacenter card or cloud – not home hardware |
Two footnotes that bite beginners: long context eats VRAM (128K context can add 3–6 GB of KV cache – a whole tier), and higher quantization costs VRAM (Q8 roughly doubles the weights column). Our quantization guide shows what you gain for it.
2. System RAM: the safety net
System RAM matters twice: the OS, browser, and engine need their share (8 GB just to breathe), and RAM is where overflow goes when VRAM runs out. 16 GB is the minimum for any local-AI machine; 32 GB if you want partial CPU offload as a fallback – a 30B Q4 can limp along on a 16 GB card with 32 GB of system RAM, at 5–15 tok/s. Slow, but it runs, which beats not running.
3. CPU: speed, not fit
The CPU doesn't decide whether a model loads – but it decides prompt-eval speed (how fast long documents get absorbed) and how painful offload is. Rule of thumb: 6+ modern cores (any Ryzen 5 / Core i5 from the last five years). Below that, even a good GPU feels sluggish on long prompts because the CPU can't feed it.
4. Storage: bigger and faster than you think
Model files are large and you will collect them: a 7B quant is ~5 GB, a 32B quant ~20 GB, and you will keep three "just to compare." 512 GB NVMe minimum, 1 TB comfortable. Speed matters at load time (NVMe vs SATA is seconds vs a minute per model load) and for offload swap – never put models on a spinning drive you also boot from.
5. PSU, cooling, OS: the unglamorous trio
- PSU: a GPU at full inference load draws near its TDP for hours. A 650W quality unit for mid-range cards, 850W+ for 24 GB flagships. Random crashes under load are a dying PSU until proven otherwise.
- Cooling: sustained 100% GPU utilization is hotter than gaming bursts. Undervolting a Radeon costs ~5% speed for ~20% less heat – worth it on laptops especially.
- OS and drivers: Linux + ROCm for AMD, any OS + CUDA for NVIDIA. Our ROCm vs Vulkan guide and ROCm install guide cover the AMD path step by step.
Three machines, verdicts
- Thin-and-light laptop (integrated graphics, 16 GB RAM): 7B Q4 via CPU at a few tok/s, or small models only. Usable for chat, painful for agents.
- 2021 gaming laptop (RTX 3060 / RX 6800M, 12 GB VRAM, 32 GB RAM): the sweet spot – 14B fully, 27–32B with tuned quants (our Lab runs exactly this class daily).
- Desktop with a 24 GB card: 32B fully with long context. The point where local stops feeling like a compromise for most work.
Bottom line
Check in order: VRAM → RAM → CPU → storage → PSU. Most "can I run local AI?" questions are answered at step one – 8 GB means 7B, 16 GB means 14B, 24 GB means 32B. Run your exact model through the VRAM calculator before spending money, and only then ask which card to buy.