RAG chatbots, and why retrieval is the hard part

What retrieval augmented generation actually does

RAG chatbot development is the work of building an assistant that looks up your own documents before it answers. Retrieval augmented generation hands the model a passage from your knowledge base, so the answer is grounded in something you wrote rather than in what the model half remembers. Most of the effort goes into the documents and the retrieval, not into the model choice.

A language model does not know your prices, your policies or your product range. Retrieval augmented generation fixes that without retraining anything. When a question arrives, a search step finds the passages in your own material most likely to contain the answer, and those passages are placed in front of the model alongside the question. The model then answers from what it was handed.

Three separate things are happening there, and they fail separately. Your content is prepared and indexed. A question becomes a search and passages come back. The model writes an answer from those passages. When a retrieval chatbot gives a bad answer, the cause is usually the middle step, and the usual response is to rewrite the prompt, which changes nothing at all.

Why grounding beats fine-tuning for most business questions

Fine-tuning adjusts the model itself using examples. It is good at teaching format, tone and consistent labelling. It is a poor way to teach facts, because facts change and a fine-tuned model has no idea which of the things it absorbed have since stopped being true.

Retrieval keeps the facts outside the model where they can be edited. Change the price list and the next answer uses the new price. It also means a wrong answer has a fixable cause, because you can find the exact passage that produced it. A fine-tuned model that has learned an old policy gives you nothing to correct except the training run.

• Facts, prices, policies and availability: retrieval, every time
• House style, output format and consistent labelling at volume: fine-tuning can earn its cost
• Anything that changes more than once a quarter: retrieval, because retraining is not a maintenance plan
• Live values such as order status or remaining stock: neither, because that is a lookup into a system rather than a search over documents

What makes a knowledge base retrievable

Retrieval works on pieces of text rather than whole documents. Your material is split into chunks, and each chunk is judged on its own when the search runs. That single fact explains most of the difference between a knowledge base that works and one that does not.

A chunk that reads as a complete answer gets retrieved and used correctly. A chunk that says this is not permitted under the above conditions is useless, because whatever this and above referred to now sits in a different chunk. The same applies to a table separated from its header row, and to a policy that never names itself because everyone in the office knows which policy it is.

The practical test takes a minute. Read one chunk on its own, hand it to a colleague from another team, and ask what it is about. If they cannot tell, the retriever cannot either.

• Write the question as the heading and answer it directly underneath
• Name things explicitly, replacing we, it and the above with the actual subject
• One canonical answer per question, in one place, rather than four near-copies in four documents
• Date and version anything that changes, and say which market or product line it covers
• Keep tables small enough to survive being cut, and repeat the header on long ones
• Never let a screenshot be the only carrier of a fact, because a search cannot read it

Confident retrieval of the wrong document

The failure people expect is a chatbot admitting it does not know. The failure that causes actual damage is the search returning something that looks right and is not: last year's pricing, a policy written for a different region, a draft that was never approved, a page from a supplier's manual uploaded once and forgotten. The model will use it, because using what it was given is the entire design.

Nothing about that answer will look uncertain. It will be fluent, specific and formatted exactly like a correct answer, which is why the defences have to sit around the retrieval step rather than in the wording of the reply.

• Filter on metadata before searching: market, product line, language, effective date
• Remove superseded documents from the index instead of leaving them beside the current one
• Allow nothing relevant was found as a valid outcome, and route those questions to a person
• Show the source behind each answer, so a reader can check it and a reviewer can spot a pattern
• Watch for questions where two documents disagree, because that is a content problem no chatbot can resolve

Measure retrieval and answering separately

To a user, a retrieval failure and a writing failure look identical. To anyone maintaining the system they need completely different fixes, so they have to be measured apart or the wrong thing gets tuned for weeks.

Assemble a set of real questions taken from what people actually ask rather than from what you expect them to ask. For each one, note which passage should be retrieved. Then measure two numbers: how often the right passage came back at all, and, when it did, how often the answer was correct and complete. A low first number is a content or indexing problem. A high first number with a low second is a prompt or model problem.

• A question set built from real traffic, including the badly worded ones
• Retrieval scored on whether the right passage appeared, before any judgement of the answer
• Answers checked against the retrieved passage rather than against somebody's memory of the policy
• The whole set rerun whenever content, chunking, model or prompt changes

When this is the wrong thing to build

If the documentation does not exist, this is a documentation project wearing a chatbot's clothes. The system can only retrieve what somebody wrote down, and the honest sequence is to write the answers people actually need first. That work is worth doing whether or not a chatbot is ever built on top of it.

If the documentation exists but contradicts itself, retrieval surfaces the contradiction and presents one side of it as fact. Deciding which version is correct is a judgement the business has to make, and no amount of tuning substitutes for making it.

And if the questions are mostly about live values, such as where an order is or whether a slot is free, retrieval is the wrong mechanism. Those need a direct lookup into the system holding the answer, which is a different build with different failure modes.

What it costs, and what it does when retrieval comes back empty

No price is published on this site for a retrieval build, and the reason is specific rather than a general refusal. What it costs is set almost entirely by the state of your documents. A company with a current handbook, a price list matching what is actually charged, and policies in one place is a short project. A company with four versions of the refund policy spread across a shared drive, a group chat and somebody's memory is paying for the archaeology, not the retrieval.

The second thing moving the figure is how wrong an answer is allowed to be. An assistant that can say it does not know is cheaper to build than one working in a corner where a confident wrong paragraph is a real liability, because the second needs evaluation sets, thresholds and a review before every change.

It should stop rather than guess. When retrieval returns nothing above the confidence threshold, or returns documents that contradict each other, the correct behaviour is to say so and hand it to a human with the question and the retrieved passages attached. That gets decided before launch, and it is the single thing separating a system people keep using from one they quietly stop trusting in month two.

Timing depends on the same documents. The audit takes the first week and mostly consists of finding out what is currently true, and a first assistant over one clean corner of the knowledge base is usually live inside a fortnight. Haad, the co-founder who owns client solutions, does most of that early work with the people who know where the real answers are kept. If there is an engineer in-house who enjoys evaluation work, building this yourself is realistic and the libraries are public, so the only question is whether that person has a spare quarter. Either way the corpus, the embeddings and your own accounts stay yours, and Wobble is answerable for retrieval quality rather than handing over a demonstration and leaving.

Common questions

What is a RAG chatbot?

A chatbot that searches your own material for relevant passages and answers from what it finds, rather than from whatever the model absorbed during training. The retrieval step is what makes the answers specific to your business, and it is also where most of the failures happen.

Is RAG better than fine-tuning?

For facts, prices, policies and anything that changes, yes, because retrieval keeps the information outside the model where it can be edited. Fine-tuning earns its cost for consistent format, house style or labelling at volume. Plenty of working systems use retrieval for the content and no fine-tuning at all.

Why does our chatbot give confidently wrong answers?

Usually because the search handed it the wrong passage, not because the model invented anything. Old versions left in the index beside their replacements are the most common cause. Filter on metadata such as market and effective date, remove superseded documents rather than keeping them alongside current ones, and display the source so wrong retrievals become visible to a reviewer.

How should we prepare documents for a RAG chatbot?

Write so each section stands alone. Put the question in the heading, answer it directly beneath, name things explicitly instead of writing we and the above, keep one canonical answer per question, and date anything that changes. The check is to read one chunk on its own and ask whether a colleague from another team could tell what it is about.

How do you measure whether a RAG chatbot works?

Score retrieval and answering as two separate numbers. Take real questions, record which passage should be retrieved for each, then measure how often it came back and, when it did, how often the answer was right. A low first number points at the content or the index, and a high first number with a low second points at the prompt or the model.

Can a RAG chatbot answer questions about live data like order status?

Not through retrieval. Searching documents finds text somebody wrote, while an order status lives in a system and changes hourly. That needs a tool call querying the system directly. Most useful deployments combine the two, using retrieval for policy and product questions and lookups for anything current.