# llama.cpp guide: run GGUF models locally

> Complete guide to llama.cpp – install it, run GGUF models, enable GPU acceleration, and read the benchmark output. The engine behind Ollama, explained.

*Source: https://velstech.net/llama-cpp-guide · Updated: 2026-08-30 · Category: AI · Tags: llama.cpp, GGUF, Local AI*

*Markdown version of [llama.cpp guide: run GGUF models locally](https://velstech.net/llama-cpp-guide). [Read the full guide with interactive tools](https://velstech.net/llama-cpp-guide).*
*Also as Markdown: [Hindi](https://velstech.net/llama-cpp-guide.hi.md) · [Tamil](https://velstech.net/llama-cpp-guide.ta.md).*

---

llama.cpp is the engine that Ollama, LM Studio, and most other local AI tools run
on under the hood. It's a C++ library that loads GGUF models and runs them on your
CPU, GPU, or both, with support for NVIDIA CUDA, AMD ROCm, and Vulkan.

This guide covers the basics: how to install it, how to run a model, how to enable
your GPU, and how to read the benchmark output. By the end you'll know exactly what
`1713 ms / 373 tok = 4.60 ms/tok = 217.6 tok/s` means (and whether it's
fast enough).

## Installation

The easiest way to get llama.cpp is to build it from source. You need a C++ compiler
and `cmake`:

```
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake .. -DLLAMA_METAL=OFF
make -j$(nproc)
```

For GPU acceleration, add the right flag:

- NVIDIA CUDA: cmake .. -DLLAMA_CUDA=ON

- AMD ROCm: cmake .. -DLLAMA_HIPBLAS=ON

- Vulkan: cmake .. -DLLAMA_VULKAN=ON

- Apple Metal: cmake .. -DLLAMA_METAL=ON

If you're on Ubuntu with an AMD card, the
[ROCm installation guide](https://velstech.net/install-rocm-ubuntu) covers the prerequisites.

## Running a model

Download a GGUF file (see [GGUF explained](https://velstech.net/gguf-explained) for which
one to pick), then run:

```
./build/bin/llama-cli -m path/to/model.q4_k_m.gguf -p "Hello, who are you?" -n 256
```

This loads the model, runs the prompt, and generates 256 tokens. The output includes
timing information that tells you how fast it ran:

```
1713 ms / 373 tok = 4.60 ms/tok = 217.6 tok/s
eval time = 1713 ms / 373 tok (4.60 ms/tok, 217.6 tok/s)
```

The first number is **prompt eval** (how fast the model reads your
question). The second is **decode** (how fast it generates the answer).
For chat, decode speed is what matters – it's the stream of text you see. For long
documents, prompt eval matters more.

## GPU acceleration

By default llama.cpp runs on CPU. To offload layers to your GPU, use the
`-ngl` flag (number of GPU layers):

```
./build/bin/llama-cli -m model.gguf -p "Hello" -n 256 -ngl 999
```

`-ngl 999` offloads all layers to the GPU. If you have limited VRAM, use
`-ngl N` where N is the number of layers that fit. The
[LLM VRAM Calculator](https://velstech.net/llm-vram-calculator) can help estimate.

For AMD GPUs with ROCm, you may need `--load-mode none` to avoid splitting
weights across GPU and CPU. The [Qwen
ROCm vs Vulkan benchmark](https://velstech.net/qwen-27b-ridge-rocm-vs-vulkan) shows how this plays out in practice.

## Benchmarking your own hardware

llama.cpp's built-in timing output is all you need. To compare against known results,
use the same parameters (model, quantization, context, backend) and note the decode
tok/s. Then check the [VelsTech benchmark database](https://velstech.net/benchmarks/index)
to see how your hardware stacks up.

## Key flags cheat sheet

| Flag | What it does |
| --- | --- |
| -m | Path to GGUF model file |
| -p | Prompt text |
| -n | Number of tokens to generate |
| -c | Context size (e.g. -c 16384 for 16K) |
| -ngl | Layers to offload to GPU (999 = all) |
| -t | Thread count (default: all cores) |
| --repeat-penalty | Penalize repetition (default 1.1) |
| --temp | Sampling temperature (0.0 = deterministic, 1.0 = creative) |

## Next steps

llama.cpp is the foundation. Most people never use it directly – they use Ollama,
which wraps it in a user-friendly CLI and API. But knowing how llama.cpp works helps
you understand what Ollama is doing, and it's essential for
[benchmarking](https://velstech.net/benchmarks/index) and fine-tuning. If you're new to
local AI, start with the [getting started](https://velstech.net/how-to-get-started-local-ai)
guide – it uses Ollama, which is built on llama.cpp.

## FAQ

**Do I need llama.cpp if I use Ollama?**

No – Ollama wraps llama.cpp and handles installation, downloads, and GPU detection for you. Learning llama.cpp is useful for benchmarking and fine-tuning, but not required to run local AI.

**How do I speed up llama.cpp?**

Offload more layers to your GPU with -ngl 999 if VRAM allows, make sure you built with the right backend (CUDA/ROCm/Vulkan), and use the highest quantization that still fits in memory.

---

*VelsTech – technology explained for everyone. Original: https://velstech.net/llama-cpp-guide*