RAG (Retrieval-Augmented Generation) lets an LLM answer questions using your own documents. It retrieves the few relevant chunks for a question, then has the model answer using only that context – no retraining needed.

Can RAG run entirely locally?

Yes. With a local LLM and a small embedding model (both via Ollama), the whole load-embed-retrieve-generate pipeline runs offline and private.

RAG from scratch: ask questions about your own documents

An LLM knows what it was trained on – but not your notes, your PDFs, your code, or your company's docs. RAG (Retrieval-Augmented Generation) fixes that by letting a model answer questions using your documents, without retraining. It's how "chat with your PDF" apps work, and you can build it entirely on your own machine with a local LLM.

This guide explains the four moving parts of RAG and walks through a working, local-first example.

The four parts of RAG

Load – read your documents (PDF, markdown, text) into chunks of text.
Embed – turn each chunk into a vector (a list of numbers that captures its meaning) using an embedding model.
Retrieve – when a question comes in, embed it too and find the chunks whose vectors are closest to it.
Generate – give the LLM the retrieved chunks plus the question, and let it answer using only that context.

That's the whole trick. The LLM never sees your whole document set – only the few relevant chunks, which it uses as grounded context. That's why answers are accurate and why you don't need to retrain anything.

Step 1: Load and chunk

Split your documents into chunks of a few hundred characters, with a little overlap between them. Why overlap? So that context that straddles a chunk boundary isn't lost. A simple approach in Python:

import re
text = open("notes.md").read()
chunks = [text[i:i+500] for i in range(0, len(text), 400)]  # 500 chars, 100 overlap

Tools like LlamaIndex and LangChain handle smarter chunking out of the box, but the idea is the same.

Step 2: Embed

An embedding model converts text into a vector – a fixed-size list of numbers – where similar meaning lands near each other. A tiny embedding model runs fast on any CPU, so this is cheap. Ollama has several, e.g. nomic-embed-text:

ollama pull nomic-embed-text

Embed every chunk and store the vectors (a simple JSON or SQLite file is fine for personal use; vector databases like Chroma or Qdrant are for scale).

Step 3: Retrieve

When someone asks a question, embed it with the same model, then find the chunks whose vectors are most similar. "Similarity" is usually cosine similarity – smaller angle between vectors means more related meaning. Pick the top 3–5 chunks:

import numpy as np
# assume emb(q) is the question vector, emb(chunks) is a matrix
scores = emb(chunks) @ emb(q) / (norms * norm_q)   # cosine sim
top = np.argsort(scores)[::-1][:5]                  # top 5 chunk indices

If the question and a chunk contain similar words or concepts, they'll rank highly – that's how RAG finds the right passages without the LLM seeing everything.

Step 4: Generate

Finally, prompt the LLM with the retrieved chunks as context. The magic is the instruction: answer using only this context.

system: You are a helpful assistant. Answer the question using ONLY the
context below. If the answer isn't there, say you don't know.

Context:
--- chunk 3 ---
--- chunk 7 ---
--- chunk 11 ---

Question: What did the notes say about X?

With a local model (e.g. a 7B via Ollama), this runs fully offline. The local AI guide shows how to run the model; the prompting guide explains why the "only use the context" instruction matters so much.

A simpler shortcut: Ollama + Open WebUI

If you don't want to write the code, Open WebUI has RAG built in: upload a document, and it chunks, embeds, and retrieves for you automatically against your local Ollama models. It's the fastest way to try RAG – the "from scratch" version above is how it works under the hood.

When RAG works (and when it doesn't)

Great at: finding facts in your docs, grounding answers, citations ("this is from section 3"), private data.
Good at: summarization across many documents (retrieve then summarize).
Not great at: math or multi-step reasoning over the whole corpus (that's agents + tools, a different animal), or when the answer needs context from many disconnected chunks.

Chunk size and retrieval count are the two knobs that matter most. Too few or too-small chunks → missing context. Too many → the LLM gets distracted or the answer gets long. Start with ~500-char chunks and top-5 retrieval, then tune.

Bottom line

RAG is load, embed, retrieve, generate – four steps you can run entirely locally. It turns a general LLM into one that genuinely knows your documents, with grounded, citable answers and no retraining. Whether you use a library or the raw steps above, the concept is the same, and it's one of the most useful things you can build on top of a local model.