Why Your RAG System Gives Confident Wrong Answers

Retrieval-augmented generation, or RAG, is the default pattern for getting a language model to answer questions about your own documents. The term comes from a 2020 paper by Lewis and colleagues at Facebook AI Research, and the idea is simple: before the model answers, search your documents for relevant passages and include them in the prompt.

A demo takes an afternoon. A system that answers correctly, every day, on messy real documents, takes a lot longer. When a RAG system gives a confident wrong answer, the instinct is to blame the model or tweak the prompt. In my experience, the cause is usually upstream: the right passage never made it into the prompt.

So start debugging at retrieval.

Check retrieval on its own

The single most useful thing you can do is evaluate retrieval separately from generation. Build a set of real questions and, for each one, note which document or passage contains the answer. Then run only the search step and check whether that passage appears in the top results.

If the right passage is not being retrieved, no prompt will save you. If it is retrieved and the answer is still wrong, now you have a generation problem. Mixing the two together makes both impossible to fix.

Chunking decides what can be found

Documents are split into chunks before they are embedded and indexed. That split matters more than most teams expect.

Chunks that are too small lose context. A paragraph that says "this limit does not apply to enterprise accounts" is useless if the chunk does not include which limit it means. Chunks that are too large dilute the embedding, so a passage about one specific policy looks vaguely similar to everything.

Split along the structure the document already has, like headings and sections, rather than a fixed character count. Carry the document title and section heading along with each chunk, either in the text that gets embedded or as metadata. Small changes here often fix more answers than a new model would.

Pure vector search misses exact terms

Embedding search is good at meaning. It is weaker at exact strings: product codes, error messages, policy numbers, people's names, acronyms your company invented. A question about error "E-4021" may retrieve passages about errors in general instead of the one that mentions E-4021.

Keyword search, usually some form of BM25 ranking, is strong exactly where embeddings are weak. Many production systems run both and merge the results, often with a method called reciprocal rank fusion, which combines rankings without needing the scores to be on the same scale. Then a reranking model scores the merged candidates against the question and keeps the best few. Hybrid search plus reranking is one of the most reliable upgrades available.

More context is not always better

It is tempting to pass twenty or thirty chunks to the model "just in case." Longer prompts cost more and run slower, and there is evidence they do not always help. A 2023 study by Liu and colleagues, titled "Lost in the Middle," found that models used information at the beginning and end of a long context more reliably than information in the middle.

Retrieve broadly, rerank, and send the model a smaller set of passages you are confident in.

Freshness and permissions

Two production problems rarely show up in demos.

Stale content. If a policy changed last week and the index still holds the old version, the system will confidently quote the old one. Re-index when source documents change, delete chunks from removed documents, and store a last-updated date you can show to users.

Access control. If the index holds documents with different permissions, retrieval has to respect them. Filter by what the current user is allowed to see at query time. Relying on the model to "not mention" restricted content is not access control.

Make answers checkable

Finally, have the model cite which passages it used, and show those sources to the user. Citations do not prevent wrong answers, but they let people verify, and they make failures visible to you. Instruct the model to say when the provided passages do not contain the answer, and test that it does. An honest "I could not find that" is better than a fluent guess.

Most RAG quality problems are search problems wearing a language model costume. Fix retrieval first, measure it on its own, and the generation step gets much easier.