Lesson 15 of 28 · Making AI useful

What is RAG (retrieval-augmented generation)?

RAG, short for retrieval-augmented generation, is a way of making an AI model look things up before it answers. The system first searches a knowledge source, such as company documents or the web, for passages relevant to your question, then gives those passages to the model with your question. The model writes its answer from that material, often citing it.

An open-book exam for AI

A language model on its own answers from memory: whatever patterns it absorbed during training. That memory stops at a training cutoff date and never included your private files, so on its own it cannot know your company's refund policy or yesterday's news, and it may invent an answer instead (a hallucination).

RAG turns the closed-book exam into an open-book one. Before answering, the system finds the relevant pages and puts them in front of the model. The model is still the writer, but now it is summarising and quoting real material rather than relying on memory alone. The approach was named in a 2020 paper by Lewis et al.

How RAG works in three steps

A RAG system has one stage that runs ahead of time and two that run every time someone asks a question:

Prepare: documents are split into small chunks, and each chunk is turned into an embedding, a list of numbers that captures its meaning, stored in a vector database or search index.
Retrieve: your question is turned into an embedding too, and the system finds the chunks whose meaning is closest to it.
Generate: those chunks are added to the prompt with an instruction such as 'answer using only these sources', and the model writes the answer.

What RAG is not

RAG is not training. The model's parameters do not change; only what it is shown at question time does. That is why new documents can be used as soon as they are indexed, while teaching a model through fine-tuning takes a separate training run.

RAG is also not a guarantee of truth. If the search step finds the wrong passage, or the right passage is missing, the model can still answer badly or fill gaps from memory. Good systems show their sources so you can check, and say so when nothing relevant was found.

Where you meet RAG in real tools

Whenever an AI tool answers from a specific set of material and shows where the answer came from, RAG or a similar technique is usually at work.

When you compare such tools, ask which sources they search, whether every answer cites them, how often the index refreshes, and whether people only get answers from documents they are allowed to open. Typical places you will meet it:

'Chat with your PDF' and document Q&A tools.
Help-desk bots that answer from a company's help articles.
Workplace assistants that search your notes, wiki or shared drive.
AI search engines that answer with numbered web citations.

Example

Worked example: what the model actually receives

Worked example

Question: How many days of annual leave do new staff get? Retrieved from the staff handbook (section 4.2): 'Employees receive 18 days of paid annual leave per year, prorated in the first year of employment.' Prompt sent to the model: Answer the question using only the source below. Cite the section. If the source does not answer it, say so. Source: [section 4.2 text] Question: [the user's question] Answer: New staff get 18 days of paid annual leave a year, prorated in their first year (Handbook, section 4.2).

A made-up company handbook, for illustration. Without the retrieved passage, the model could only guess.

Key points

RAG means retrieving relevant passages, then generating an answer from them.
It lets a model use private or recent information without retraining.
Think open-book exam: the model reads the sources before answering.
It reduces hallucinations but depends on finding the right passages.
Good RAG tools cite their sources so you can check them.

Try it on Mindber

Glossary: retrieval-augmented generation (RAG)

Find tools that answer from your documents

Glossary: vector database

Next lesson: fine-tuning, and how it differs from RAG

AI fundamentals

Frequently asked questions

Is RAG better than fine-tuning?

They solve different problems. RAG suits knowledge: facts that change, private documents and answers that need citations, because you update the documents rather than the model. Fine-tuning suits behaviour: a consistent style, format or narrow task. Many production systems use both, starting with good prompts and RAG and adding fine-tuning only if needed.

Do I need a vector database to use RAG?

Not always. Vector databases are a common way to find passages by meaning, but RAG can also use a keyword search engine, a regular database or a web search, and many systems combine methods. If you use a ready-made document chat or workplace assistant, the retrieval part is built in and you never manage it yourself.

Do I still need RAG if a model has a long context window?

For a small set of documents, you can often paste everything into the prompt, since many models accept long inputs. RAG becomes necessary when the material is too large to fit, changes often, or must respect access permissions. Sending only the relevant passages also keeps costs down, and models can overlook details buried in the middle of very long inputs.