Why RAG gives a wrong answer — and how to find the cause?
Follow a question from source documents to the final answer. Find the first stage where useful evidence disappears, changes or stops supporting the claim.
10 min · Reading & exerciseFree lesson · No registration
Distinguish missing documents, retrieval failures and unsupported generation. Choose the next experiment based on evidence from the failing stage.
Before you start
Know the evaluation criteria from lesson 01. RAG means retrieving relevant information and providing it to a model as context for an answer.
01 / Trace the pipeline
One wrong answer can have several causes.
In a typical RAG flow, the application prepares documents, searches for relevant passages, selects context and asks a model to answer using that context. RAG can provide evidence from maintained company information. It does not guarantee that every search finds the right passage or that every answer uses it correctly.
Fictional teaching example
A question about contractor access
Northstar’s invented current policy permits contractor access for 30 days after sponsor approval. A retired policy contains a 90-day limit. The employee onboarding guide describes another process. We ask: “Can a contractor receive access for 90 days without approval?” All documents and outcomes in this example are illustrative.
If the answer says yes, first capture the evidence: the query, eligible documents, retrieved passages, final context and answer with citations. Repeating the question without recording these stages makes diagnosis much harder.
Use a request identifier to connect the trace. Apply access controls to diagnostic logs as well as search results, and avoid recording credentials or unnecessary confidential content.
02 / Check the source
Search cannot recover a rule that was never indexed.
Open the original document and verify that the current rule exists, that extraction captured it and that its version and applicability are recorded. An outdated import, an unreadable scan or a dropped table cell can remove the crucial condition before a search ever runs.
Inspect the actual chunks. A passage saying “Access is permitted for 30 days” may lose meaning if “after sponsor approval” lands in an unrelated chunk. Keep enough context to understand the rule and attach source identifiers, headings and version metadata. Test chunk boundaries on representative questions rather than selecting a size by habit.
Compare two chunks
Incomplete: Contractor access is permitted for 30 days.
Usable: Contractor access is permitted for 30 days after sponsor approval. Applies to external contractors. Current policy.
The second passage carries the condition and applicability needed to judge the question. A heading can be essential evidence, not decorative text.
03 / Check retrieval
Was the supporting passage among the candidates?
Run the recorded query against the relevant index snapshot. Inspect the candidates before any later selection. Exact product identifiers often need lexical matching; paraphrased questions may benefit from semantic similarity. Hybrid retrieval combines these signals, and reranking can reorder candidates by relevance. Neither step repairs a missing or incorrectly extracted source.
Observation
Likely stage to investigate
Next check
Current policy absent from the index
Ingestion or freshness
Import status, extracted content and version.
Policy indexed but supporting chunk absent from results
Retrieval
Query wording, filters and candidate retrieval.
Correct chunk retrieved but absent from final context
Context selection
Reranking, deduplication and token budget.
Correct evidence in context, unsupported final claim
Generation
Instruction, claim support and handling of uncertainty.
Enforce document permissions before returning content to the model. “Do not reveal restricted information” in a prompt cannot replace an authorization filter. A permitted document can still be irrelevant, and a highly relevant document can still be restricted.
04 / Inspect the final context
Retrieved evidence is useful only if the model receives it.
Compare retrieved candidates with the exact context supplied for generation. A token budget can trim away the approval condition; deduplication can keep a retired version; reranking can favor a general employee guide. Preserve provenance when assembling context so a reviewer can trace every passage to its document.
Resolve source applicability explicitly. For the Northstar example, prefer the current contractor policy over a retired policy using maintained metadata and business rules. If current sources genuinely conflict, disclose the conflict or ask for clarification. Do not force a confident answer from contradictory evidence.
A useful diagnostic experiment is to supply the known supporting passage directly, under the same answering instruction. If that produces a valid answer, focus on upstream retrieval and selection. If it still fails, investigate generation. This narrows the search; it does not by itself prove production reliability.
05 / Verify the answer
Check each claim against its evidence.
Require the answer to distinguish supported facts from missing information. Check whether each important claim is supported by the cited passage, not just whether a citation is present. When the context lacks an answer, the assistant should say what is missing and identify a useful next step.
Retrieved documents are untrusted input. A passage that tells the assistant to ignore its instructions should not gain authority over the application. Use tool permissions and validation outside the model to limit the consequences of a prompt injection attempt.
After a targeted change, rerun both retrieval checks and end-to-end answer cases. Keep ordinary, ambiguous, missing-evidence and conflicting-source examples. Raising the number of retrieved chunks can improve coverage while also introducing irrelevant context; measure the actual tradeoff.
Your exercise / About 5 minutes
Find the first broken stage in the RAG lab.
Open the interactive lab and compare the question, retrieved passages and answer across its scenarios. The lab is a deterministic teaching simulation with fictional documents, so you can inspect the same failure repeatedly.
Start with a scenario that has enough evidence. Identify the passage supporting the answer.
Compare it with a scenario where useful evidence is missing or misleading.
Write down the first stage that no longer provides enough evidence and one change you would test there.
“The model is wrong” is a symptom. A useful diagnosis identifies the missing or misused evidence, the stage where it happened and a test that could confirm the cause. When the source lacks the answer, abstention is the correct outcome.
Knowledge check
Sources & further reading
The diagnostic sequence and fictional examples are teaching material. These primary sources explain retrieval architecture and handling of untrusted content.