Why RAG Chatbots Still Hallucinate, and Where the Fix Actually Lives

A support chatbot answers 200 questions correctly in testing. It goes live on Monday. By Wednesday it has told a customer that a discontinued plan is still available, cited a help article that says the opposite, and done it with complete confidence.

Nothing broke. The retrieval worked, the model worked, and the answer was wrong anyway.

This is the gap between a RAG chatbot that demos well and one that survives production. It is worth being precise about where that gap actually sits, because most of it is not in the model.

What a RAG Chatbot Is, and What the Term Hides

RAG means retrieval augmented generation. A query goes to a retriever, the retriever pulls passages from a document store, and those passages go into the prompt alongside the question.

The architecture is simple and the failure surface is not

Four components, each with its own failure mode. Chunking decides what a retrievable unit looks like. Embedding decides what counts as similar. Retrieval decides what comes back. Generation decides what to do with it.

A chatbot with RAG bolted on is not automatically grounded. It is a system that has been handed some context and told to be helpful, which are two instructions that conflict the moment the context is insufficient.

Why teams reach for it anyway

The alternative for keeping a model current on internal documentation is retraining, and nobody wants to retrain because a pricing page changed. Retrieval moves the knowledge outside the weights, which means updating it is an editorial task rather than an ML one.

That is a genuine architectural win. It is also where the overconfidence starts, because moving knowledge outside the model does not make the model honest about not having it.

Why AI Hallucinations Survive Retrieval

An AI hallucination is output that is fluent, plausible, and not supported by the source material. The definition is easy. Preventing it in a retrieval system is not, because retrieval introduces new ways to produce one.

The retrieved passage is wrong but relevant

Semantic similarity is not factual correctness. A query about current refund terms will happily retrieve the 2023 refund policy, because it is the most semantically similar document in the store.

The model then answers correctly from an incorrect source. No component failed. The system is wrong.

The answer spans two chunks and you retrieved one

This is the most common LLM hallucination example we see in support deployments, and it is entirely a chunking artifact. A procedure runs across a chunk boundary. Retrieval returns the first half. The model completes the second half from its general knowledge, and it sounds exactly like the first half.

Nothing in the output signals where the document stopped and the invention began.

Retrieval returns nothing useful and the model answers regardless

Most default configurations have no concept of insufficient context. Passages come back ranked, the top ones go into the prompt, and generation proceeds whether or not the top result has anything to do with the question.

A retrieval system without a relevance threshold is a system that will confidently answer questions about documents you never gave it.

Grounding AI Answers Is a Policy Decision

Grounding in AI gets discussed as a property of the model. In practice it is a set of choices about what the system is permitted to do when the evidence runs out.

Set a threshold and let it refuse

The most useful thing you can build into a RAG chatbot is the ability to decline. If the top retrieved passage falls below a relevance score, the correct output is that the answer is not in the knowledge base, not a best effort.

Teams resist this because refusal rates look bad on a dashboard. They look considerably better than the alternative, which is an unmeasured wrong answer rate.

Cite at the passage level

Attach the source to every answer, and attach it to the specific passage rather than the document. Document-level citation tells the reader which of forty pages the claim supposedly came from, which is not auditable.

Passage-level citation gives you two things. Users can check, and when an answer is wrong you know precisely which chunk produced it.

Test refusal explicitly

Write twenty questions your corpus genuinely cannot answer. Count how many get a confident response. That count is your hallucination rate under adversarial conditions, and it is a more honest metric than accuracy on questions you know are covered.

RAG vs Fine Tuning vs Prompt Engineering

The comparison gets framed as a choice. For most knowledge-grounded applications it is not one, because the three do different jobs.

Fine tuning changes behavior, not facts

Fine tuning is how you change tone, output format, or task-specific behavior. It is a poor mechanism for factual currency, because every documentation change becomes a training event and the model still cannot tell you where an answer came from.

RAG vs fine tuning is the wrong question when the requirement is auditability. Fine tuning cannot cite.

Prompt engineering sets the policy

The refusal behavior, the citation requirement, and the instruction to prefer retrieved context over parametric knowledge all live in the prompt. This is where grounding policy is actually enforced.

Most production systems use all three

Retrieval for facts, prompting for policy, and fine tuning where output shape matters. Treating them as competing options tends to produce a system that does one of the three jobs well.

Evaluating a RAG Chatbot Before You Trust It

Accuracy on a happy-path test set tells you almost nothing. Build the evaluation around the failure modes instead.

Evaluate retrieval separately from generation

These fail independently and averaging them hides both. Measure whether the correct passage was retrieved at all, then separately whether the answer is faithful to what was retrieved.

A system with 90% retrieval recall and 90% faithfulness is not a 90% system.

Feed it contradictory documents on purpose

Production corpora contain contradictions. The old policy and the new one both exist, both are indexed, and both are semantically similar to the query.

What the system does with that conflict is the single most informative test you can run, and it is the one that never appears in a vendor demo.

Measure faithfulness, not fluency

An answer can be well written, on topic, and unsupported. Human review on a sample, checking each claim against the cited passage, remains the most reliable evaluation available. Automated faithfulness scoring helps at scale but should not be the only check before a customer-facing deployment.

Watch the questions people actually ask

Real query distributions look nothing like test sets. The gap between them is where your corpus is thin, and that list is more valuable than most evaluation metrics.

How Agentency Runs a RAG Chatbot

We build a RAG chatbot platform, so the fair test is to hold our own defaults against the controls above.

Retrieval is gated in two stages

Agentency runs a two-stage retrieval process before anything is generated. When the retrieval quality gate finds nothing relevant, generation is blocked. The agent says it does not know and offers a human.

That is the relevance threshold from earlier, shipped as the default rather than left as a setting someone has to go looking for.

Every answer carries its source

Answers are generated only from knowledge the customer crawled, uploaded, or pasted, and each one shows the source it came from. The built-in test chat displays that source for every test question.

Which turns the twenty unanswerable questions exercise into an afternoon, not a project.

Refusals are measured, not hidden

The dashboard tracks resolution rate and handoff rate, with trend views. A rising handoff rate reads as a gap in the corpus, and the conversation log shows exactly which questions the knowledge base failed to cover.

The fix is an edit to the document, not an argument with the model.

Live facts come from live systems

The fastest way to get a stale answer is to retrieve a fact that changes daily from a document that changes quarterly. For questions like order status or current availability, a Call Action queries the API the business already runs instead of searching the document store.

That sidesteps the correct but outdated passage problem for the data most likely to go stale. High-risk Call Actions can also require an explicit confirmation from the user before they execute.

Is a RAG chatbot the same as a chatbot with a knowledge base?

Not necessarily, and the difference matters

Keyword search over a document store attached to a generative model is not retrieval augmented generation in any meaningful sense, though plenty of products describe it that way.

The distinguishing features are semantic retrieval, an explicit relevance threshold, and grounding enforced at generation. Without those three, what you have is a search box with a model attached.

What does RAG as a service actually give you?

Managed infrastructure, and someone else’s defaults

The pipeline components are well understood, so hosted platforms remove real engineering effort around chunking, embedding, indexing, and reranking.

What to check is whether the defaults match your risk tolerance. The relevance threshold, the refusal behavior, and the citation granularity are the settings that determine whether the system hallucinates, and they are often the ones you cannot see. Platforms built for customer-facing deployment tend to be more conservative here. Agentency runs a two-stage retrieval gate and escalates rather than answering when nothing relevant comes back, which is the behavior worth looking for regardless of vendor.

How do you reduce hallucinations without hurting coverage?

Accept that the tradeoff is real

Tightening the relevance threshold reduces wrong answers and increases refusals. There is no configuration that eliminates both.

The right position depends on cost of error. For internal documentation search, a slightly loose threshold is fine because the user can verify. For anything customer-facing, financial, or regulated, tighten it and absorb the refusals.

What This Comes Down To

Retrieval augmented generation solves a real problem. It keeps a model current on material it was never trained on, and it makes answers traceable in a way fine tuning cannot.

What it does not do is make a system honest. That comes from a relevance threshold, a refusal path, passage-level citation, and an evaluation process built around contradictory and unanswerable inputs rather than the questions you already know it handles.

Get those four right and RAG works. Skip them and you have built a very fluent way to be wrong.

About the Author

Omar El Bahr is a Senior Digital Growth Specialist at Agentency, where he leads SEO, content strategy, and organic growth across international markets. He is a Forbes Communications Council contributor and has written for Entrepreneur on business communication and digital strategy.