What RAG is, and when you actually need it

Someone has told you that you need RAG. Possibly a vendor, possibly a developer, possibly a slide with a diagram on it. It is worth knowing what it actually is before you buy one, because the concept takes a paragraph to explain and the version people sell takes months to maintain.

Retrieval-augmented generation means this: before the model answers, your system searches your own documents, finds the passages that look relevant, and pastes them into the prompt along with the question. The model then answers from that text rather than from memory.

That is the whole idea. Nothing is added to the model, the model does not learn anything, and next week it will know nothing about your business that your search did not hand it. The quality of a RAG system is the quality of its search, which is why most of the work has nothing to do with AI.

The pipeline, step by step

1. Decide what is authoritative. Before any technology, pick which documents are the source of truth and retire the rest. Two versions of the same policy in the index is the defect that produces most wrong answers, and no amount of engineering downstream removes it.
2. Split the documents into passages. A chunk is the unit that gets retrieved, so it has to make sense alone. Keep headings with their sections, keep tables with their column headers, and aim for passages a person could read as an answer. Splitting on a fixed character count, which is the default in most tutorials, is how a price table ends up separated from the row that says what the prices are for.
3. Turn the passages into vectors and store them. This is where the vector database appears. If you already run Postgres, the pgvector extension is usually enough, and a separate service is one more thing to host for benefits you may never need at your size.
4. Search with both methods. Vector search finds passages that mean the same thing in different words. Keyword search finds exact strings: part numbers, product codes, article references, the name of a form. Business questions contain a lot of exact strings, so hybrid search beats pure vector search for most corpora.
5. Rerank the results. Retrieve more candidates than you need, then have a reranking model order them by how well they answer the actual question, and keep the best three or four. This step is cheap and it is where a mediocre system usually becomes a good one.
6. Generate with rules. Answer only from the supplied passages, cite which one each claim came from, and say plainly when the passages do not contain the answer. The citation matters more than it looks: it lets a reader check, and it makes wrong answers findable instead of invisible.
7. Evaluate retrieval separately from the answer. These are two different failures with two different fixes. Take fifty real questions, record which passage should have been found, and measure how often it was. If the right passage never arrives, no prompt change will save the answer.
8. Keep the index current. Re-index when documents change, and know how long the lag is. This is the most common production failure by a distance, and it is discussed below.

When RAG is the wrong answer

Your corpus is small. If everything the assistant needs fits comfortably in the model's context window, put it in the prompt and skip the pipeline entirely. A handbook of thirty pages does not need a vector database, and the version that skips it has no index to go stale.

The data is structured. "How many orders did we ship last month" is a database query. Embedding your orders table and asking a retrieval system to count things is expensive, slow and wrong in ways that are hard to spot. If the answer lives in rows and columns, give the model a tool that runs a query, or write the query yourself.

The facts change constantly. Stock levels, prices, delivery dates and account balances should be fetched from the system of record at the moment of asking. An index is a copy, and a copy of a number that changes hourly is a liability.

One document at a time. If the task is to answer questions about the contract a user just uploaded, that is not retrieval, it is reading. Send the document.

The uncomfortable version: a large share of "we need RAG" conversations end with a document clean-up, a search box and no model at all. If your team cannot find the current version of a policy today, adding a language model gives them confident answers from the wrong version instead.

The failure mode to picture

Everything works. Retrieval finds a passage, the passage is real, the citation links to a genuine internal document, and the answer is wrong, because the document it quoted was superseded in March and nobody removed the old one from the folder the indexer reads.

This failure is worse than an obvious mistake for two reasons. It is confident, and it is auditable in the wrong direction: the citation makes the answer look verified. Search engines and staff alike will treat it as checked.

The fix is unglamorous and it is mostly governance. One authoritative location per document type, a visible last-reviewed date, an index that rebuilds on change rather than on a monthly schedule, and a spot check that specifically looks for answers citing documents that should have been retired. That work is the same discipline as handling hallucination in production: the model is not the part you control, the inputs and the checks are.

What it costs to keep

The build is a few weeks. The commitment is the index, the evaluation set and the document hygiene behind both, and those do not end. Budget them as recurring lines rather than project costs, in the same way as the rest of the three-year cost picture.

Done properly, this is what makes a private AI assistant worth having: answers that come from your material, with a link to the paragraph they came from. The rest of the concept guides cover the neighbouring pieces.

Frequently asked questions

Is RAG better than fine-tuning?

They solve different problems. RAG gives a model access to information it did not have; fine-tuning adjusts how a model behaves, such as tone, format or a narrow classification task. If the complaint is "it does not know our stuff", you want retrieval. If it is "it knows, but answers in the wrong shape", you may want fine-tuning.

Do we need a vector database?

Not necessarily. If you already run Postgres, pgvector handles a corpus of tens of thousands of passages comfortably. A dedicated vector service earns its place at larger scale or with specific filtering needs, and before then it is an extra system to host, secure and pay for.

Why does our assistant quote an out-of-date document?

Because the index still contains it. Either the source folder holds the superseded version, or the index has not been rebuilt since the document changed. Fix the source first, then the refresh schedule, and add a check that looks for citations to documents that should be gone.

How do we know whether retrieval is working?

Measure it separately from the answer. Take fifty real questions, note which passage should be found for each, and check how often it appears in the retrieved set. That number tells you whether to work on search or on the prompt, and without it you will be guessing at both.