# RAG from scratch: ask questions about your own documents

> What RAG is, how it works, and a working local example – build a private question-answering system over your own files with an LLM and embeddings.

*Source: https://velstech.net/rag-from-scratch · Updated: 2026-08-30 · Category: AI · Tags: RAG, Embeddings, Local AI*

*Markdown version of [RAG from scratch: ask questions about your own documents](https://velstech.net/rag-from-scratch). [Read the full guide with interactive tools](https://velstech.net/rag-from-scratch).*
*Also as Markdown: [Hindi](https://velstech.net/rag-from-scratch.hi.md) · [Tamil](https://velstech.net/rag-from-scratch.ta.md).*

---

An LLM knows what it was trained on – but not your notes, your PDFs, your code, or
your company's docs. **RAG** (Retrieval-Augmented Generation) fixes that
by letting a model answer questions using *your* documents, without retraining.
It's how "chat with your PDF" apps work, and you can build it entirely on your own
machine with a local LLM.

This guide explains the four moving parts of RAG and walks through a working,
local-first example.

## The four parts of RAG

- Load – read your documents (PDF, markdown, text) into chunks of text.

- Embed – turn each chunk into a vector (a list of numbers that captures its meaning) using an embedding model.

- Retrieve – when a question comes in, embed it too and find the chunks whose vectors are closest to it.

- Generate – give the LLM the retrieved chunks plus the question, and let it answer using only that context.

That's the whole trick. The LLM never sees your whole document set – only the few
relevant chunks, which it uses as grounded context. That's why answers are accurate
and why you don't need to retrain anything.

## Step 1: Load and chunk

Split your documents into chunks of a few hundred characters, with a little overlap
between them. Why overlap? So that context that straddles a chunk boundary isn't lost.
A simple approach in Python:

```
import re
text = open("notes.md").read()
chunks = [text[i:i+500] for i in range(0, len(text), 400)]  # 500 chars, 100 overlap
```

Tools like [LlamaIndex](https://docs.llamaindex.ai)
and [LangChain](https://langchain.com) handle
smarter chunking out of the box, but the idea is the same.

## Step 2: Embed

An embedding model converts text into a vector – a fixed-size list of numbers – where
similar meaning lands near each other. A tiny embedding model runs fast on any CPU, so
this is cheap. Ollama has several, e.g. `nomic-embed-text`:

```
ollama pull nomic-embed-text
```

Embed every chunk and store the vectors (a simple JSON or SQLite file is fine for
personal use; vector databases like Chroma or Qdrant are for scale).

## Step 3: Retrieve

When someone asks a question, embed it with the same model, then find the chunks whose
vectors are most similar. "Similarity" is usually cosine similarity – smaller angle
between vectors means more related meaning. Pick the top 3–5 chunks:

```
import numpy as np
# assume emb(q) is the question vector, emb(chunks) is a matrix
scores = emb(chunks) @ emb(q) / (norms * norm_q)   # cosine sim
top = np.argsort(scores)[::-1][:5]                  # top 5 chunk indices
```

If the question and a chunk contain similar words or concepts, they'll rank highly –
that's how RAG finds the right passages without the LLM seeing everything.

## Step 4: Generate

Finally, prompt the LLM with the retrieved chunks as context. The magic is the
instruction: *answer using only this context*.

```
system: You are a helpful assistant. Answer the question using ONLY the
context below. If the answer isn't there, say you don't know.

Context:
--- chunk 3 ---
--- chunk 7 ---
--- chunk 11 ---

Question: What did the notes say about X?
```

With a local model (e.g. a 7B via Ollama), this runs fully offline. The
[local AI guide](https://velstech.net/how-to-get-started-local-ai) shows how to run the
model; the [prompting guide](https://velstech.net/better-prompts) explains why the "only
use the context" instruction matters so much.

## A simpler shortcut: Ollama + Open WebUI

If you don't want to write the code, [Open WebUI](https://openwebui.com)
has RAG built in: upload a document, and it chunks, embeds, and retrieves for you
automatically against your local Ollama models. It's the fastest way to try RAG –
the "from scratch" version above is how it works under the hood.

## When RAG works (and when it doesn't)

- Great at: finding facts in your docs, grounding answers, citations ("this is from section 3"), private data.

- Good at: summarization across many documents (retrieve then summarize).

- Not great at: math or multi-step reasoning over the whole corpus (that's agents + tools, a different animal), or when the answer needs context from many disconnected chunks.

Chunk size and retrieval count are the two knobs that matter most. Too few or too-small
chunks → missing context. Too many → the LLM gets distracted or the answer gets long.
Start with ~500-char chunks and top-5 retrieval, then tune.

## Bottom line

RAG is load, embed, retrieve, generate – four steps you can run entirely locally. It
turns a general LLM into one that genuinely knows your documents, with grounded,
citable answers and no retraining. Whether you use a library or the raw steps above,
the concept is the same, and it's one of the most useful things you can build on top
of a local model.

## FAQ

**What is RAG?**

RAG (Retrieval-Augmented Generation) lets an LLM answer questions using your own documents. It retrieves the few relevant chunks for a question, then has the model answer using only that context – no retraining needed.

**Can RAG run entirely locally?**

Yes. With a local LLM and a small embedding model (both via Ollama), the whole load-embed-retrieve-generate pipeline runs offline and private.

---

*VelsTech – technology explained for everyone. Original: https://velstech.net/rag-from-scratch*