How to handle hallucination in a production system

A model that answers ninety-nine questions correctly and invents the hundredth is not a system with a small defect. It is a system whose output cannot be trusted without a check, and that changes what you can safely put it in front of.

The good news is that this is an engineering problem with known answers. You do not eliminate hallucination, you contain it, and the containment is what most of the work in a production AI system actually consists of.

Why it happens, briefly

A language model produces the most plausible continuation of the text it has been given. It has no separate store of facts to consult and no internal signal that says "I do not know this one". A confident wrong answer and a confident right answer are produced by exactly the same process, which is why the tone gives you nothing.

Two consequences follow. Asking a model to be more careful helps a little and is not a control. Fluency is not evidence of accuracy, and any process that relies on a reader noticing the difference will fail on the day the reader is busy.

Six rules that do the work

1. Never let the model be the source of a fact it could look up. Prices, policies, stock, opening hours, contract terms and customer records belong in a system of record. The model's job is to find the relevant passage and phrase the answer, with a citation the reader can open. If retrieval finds nothing relevant, the correct output is that it found nothing, not a summary of what such a document would probably say.

2. Constrain the shape of the output. For anything machine-readable, require a fixed schema: defined fields, enumerated values, and validation on the way out. A model asked to choose between eight categories fails in ways you can catch. A model asked to describe the category fails in ways you cannot.

3. Make "I do not know" a first-class answer. It needs to be an allowed output, tested for, and routed somewhere. A system with no escape hatch will fill the gap, because filling gaps is what the underlying model does.

4. Separate drafting from acting. Sending, paying, deleting, publishing and committing are irreversible. Put either a deterministic check or a person between the model and any of them. This is the single design decision that most often separates a system that fails gracefully from one that fails publicly.

5. Verify against the source, not the model. Do not ask the model whether its answer was right. Check the extracted total against the arithmetic, the quoted policy against the document, the record identifier against the database. Self-assessment measures confidence, and confidence is the thing that is broken.

6. Measure it, or you are guessing. Build a set of fifty to a hundred real cases with known answers, and re-run it whenever the prompt, the model or the retrieval changes. Log every input and output, and spot-check a sample every week. Without this, nobody finds out that quality slipped until a customer does.

For a worked example of rules one, four and five in a system that touches money, see how validation works in automated invoice processing: the check is arithmetic, the source is your supplier master data, and the model is never allowed to approve anything.

The failure mode to picture

In 2024 a Canadian tribunal ruled on a case brought against Air Canada by a passenger who had followed advice given by the airline's website chatbot about applying for a bereavement fare after travelling. The advice was wrong, the airline argued it should not be responsible for what the chatbot said, and the tribunal disagreed. The passenger was awarded damages.

The number involved was small. The principle is not: a statement your system makes to a customer is a statement your business made. That is the standard to design against, and it applies to a quote, a delivery promise, a warranty answer and a policy summary just as much as to a fare rule.

Picture the internal version too. An assistant answers a question about a discount that expired last quarter, because the retrieval index still holds the old page and nothing told it which version was current. Nobody reviews the answer, because the previous two hundred were fine. The cost is not the wrong answer. It is that you now have to find out which other answers came from the same stale document.

When a language model is the wrong tool

If your accuracy requirement is one hundred per cent and no human can be in the loop at a sensible cost, you do not want a language model. You want a form, a lookup or a rules engine. Deterministic systems fail predictably, and predictable failure is a feature when the stakes are high.

Some categories should simply stay off the table for customer-facing automation: medical, legal and tax determinations, credit and eligibility decisions, and anything where a wrong answer is not recoverable by apologising and fixing it. For the regulated ones, a wrong answer is not only expensive, it may put you in a category the EU AI Act treats far more strictly.

The version of this that a vendor will not tell you: if the honest answer to "what happens when it is wrong" is that nobody would know, the right move is not better prompting. It is a smaller scope, with a person at the point where it matters, and the option of not building it still on the table.

What this costs

Containment is not free, and it is worth pricing before you commit. The test set has to be written and maintained, the review queue takes somebody's time, the citations have to be built and the retrieval index kept current. That is ordinary engineering, and it is the reason a demo takes an afternoon and a production system does not.

It is also the part of the bill that keeps paying. A system with evaluation attached tells you when it degrades. A system without it degrades exactly as fast and informs nobody. Those two systems have very different costs in year two, which is the honest case for looking at three years rather than one.

This discipline is built into the custom systems we deliver, and the rest of the concept guides explain the underlying pieces.

Frequently asked questions

Does retrieval-augmented generation eliminate hallucination?

No, it reduces it. Grounding answers in your documents removes the most common cause, which is a model filling a gap from training data, but a model can still misread a passage, blend two sources or answer from a document that is out of date. Citations matter because they let a reader check.

Will a better model fix this?

Better models hallucinate less often, which raises the stakes rather than removing the problem: rare errors get less scrutiny. Model choice is worth a few percentage points. Design decisions about grounding, output constraints and where a human sits are worth considerably more.

How do we know whether our system is getting worse?

Keep a fixed set of real cases with known answers and re-run it on every prompt, model or retrieval change. Without a stable test set, quality changes are invisible, because the only signal is complaints, and complaints lag by weeks.

Are we liable for what our chatbot tells a customer?

Treat everything it says as a statement from your business, because that is how a court has already treated it. In practice that means limiting what it is allowed to assert, grounding those assertions in documents you control, and keeping logs of what was said to whom.