How to Debug a RAG System That Retrieves the Wrong Context?

Quick answer

When a RAG answer is wrong, first check what the retriever actually returned.

Work through the pipeline in order: documents, chunks, retrieval, context, then the model.

Fix the first broken step, then test again before moving on.

Hey, welcome back to Roy's AI Lab!

Have you ever built a RAG application that worked perfectly with your test questions but started giving irrelevant answers when you tried more realistic queries?

Your documents are indexed. Your embeddings are generated. Your vector database returns results. The LLM produces fluent answers.

Everything looks fine until someone asks a real question.

The system retrieves an unrelated paragraph, misses the exact information you need, or confidently answers using evidence that has nothing to do with the question.

So, naturally, you start changing things.

You increase the chunk size. You try another embedding model. You rewrite the prompt. Maybe you even consider replacing the vector database.

But the results are still inconsistent.

If this sounds familiar, don't worry. This is exactly the kind of problem engineers need to learn how to diagnose systematically.

And if you're preparing for a GenAI engineer interview, there's a good chance you'll encounter a question like this:

"Your RAG application retrieves the wrong context. How would you debug it?"

Today, let's work through the problem together, understand what can go wrong at each stage, and build a debugging approach you can actually use in a project or explain in an interview.

The first principle to remember: don't start fixing a RAG system before you know which part is broken.

1. Start by reproducing the failure

Imagine you're building a company policy assistant.

An employee asks:

"How many days of parental leave am I eligible for?"

The correct answer exists in the company's HR policy document. But your application retrieves a paragraph about annual leave and produces an incorrect answer.

What's your first move?

Would you immediately change the LLM?

Not yet.

The model may have answered incorrectly because it received the wrong evidence in the first place. Even a capable model cannot reliably answer a question using a passage that doesn't contain the required information.

Before changing anything, capture six things:

The original user question.
The expected answer and its supporting source passage.
The retrieved chunks and their ranks.
Retrieval scores, where available.
The exact context passed to the LLM.
The final generated answer.

This gives you a trace of what actually happened instead of a collection of guesses.

For example, you might discover that the correct policy was indexed successfully, but the retriever returned five unrelated passages. That immediately narrows your investigation.

Practical tip: Save failed queries as test cases. Every failure you investigate today can become a regression test that prevents the same problem from returning tomorrow.

2. Check whether the correct information was indexed

Let's go back to our company policy assistant.

Suppose the correct parental leave policy exists in a PDF. You know the document is uploaded, and your ingestion pipeline completed without errors.

Does that guarantee the content was indexed correctly?

Not necessarily.

Imagine the policy contains a table specifying different leave entitlements. Your PDF extraction library reads the text in the wrong order, drops a column, or merges separate rows into one confusing paragraph.

The original PDF looks perfect. The extracted text does not.

Your embedding model then converts that corrupted text into a vector, and your vector database stores it.

The database is working. The retrieval pipeline is still broken.

Check the following:

PDF extraction: Is the relevant passage present in the extracted text?
Tables and formatting: Were rows, columns, headings, and reading order preserved?
OCR: Were scanned pages converted into readable text?
Document freshness: Are you indexing the latest version?
Duplicates: Are outdated copies competing with current documents?
Metadata: Are document IDs, dates, categories, and versions attached correctly?

For PDFs, compare the extracted text with the original page. Don't assume that a successful ingestion job means the content was extracted accurately.

Here's the question to ask yourself:

Does the correct information exist in the indexed content in a form my retriever can actually search?

If the answer is no, fix the ingestion process before tuning retrieval.

3. Investigate chunking before changing your embedding model

Now suppose the correct information exists in your extracted documents.

The next question is whether it was split into useful chunks.

Chunking isn't simply about dividing a document into pieces of a particular size. It's about preserving enough meaning for retrieval to identify the right passage.

Consider this example.

A policy document contains:

"Employees are eligible for parental leave after completing the required service period. The duration of leave depends on the employee category and applicable company policy."

Imagine splitting this into tiny chunks that separate the eligibility condition from the explanation of leave duration.

The retriever might return a chunk mentioning parental leave without returning the condition that determines eligibility.

Now consider the opposite approach: putting an entire 30-page policy into one enormous chunk.

A relevant question might match that chunk, but the retrieved context also contains a large amount of unrelated information.

Neither approach is ideal.

How should you debug chunking?

Start with these questions:

Does the relevant fact appear inside a single chunk?
Does the chunk preserve the heading or section that explains its meaning?
Are important conditions separated from the statements they qualify?
Are chunks too large for precise retrieval?
Are chunks too small to make sense without surrounding text?

Then compare several configurations using the same evaluation questions.

You might test different chunk sizes, overlap settings, or structure-aware splitting. For documents with nested sections, parent-child retrieval can help you find a precise passage and then retrieve its larger surrounding context.

But don't assume one chunk size works best for every dataset.

A collection of short FAQ entries has different needs from technical manuals, legal policies, or long research papers.

What matters isn't how many chunks you create. It's whether the right chunk contains enough information to answer the question.

4. Check whether the user's wording matches the document

Here's another situation you might encounter.

The user asks:

"How long does delivery usually take?"

Your logistics document uses the phrase:

"Estimated shipment lead time."

The meaning is similar, but the wording is different.

Semantic search can help bridge this gap. However, retrieval quality still depends on the embedding model, document content, query, and search configuration.

When the original query performs poorly, test alternatives.

Original query:

"How long does delivery usually take?"

Normalized query:

"What is the expected delivery duration?"

Domain-specific query:

"What is the estimated shipment lead time?"

Compare the retrieved results for each version.

If a rewritten query consistently retrieves better evidence, query transformation may help your application.

You can also explore multi-query retrieval when a question genuinely has multiple interpretations.

But be careful. Query rewriting can introduce assumptions that the user never expressed.

Keep the original question, preserve its intent, and evaluate rewritten queries against known examples.

Don't add query rewriting just because it sounds sophisticated. Add it when testing shows that it solves a real problem.

5. Test retrieval independently of the LLM

This is one of the most important debugging steps.

Many developers inspect only the final answer. That makes it difficult to determine whether the retriever failed or the model mishandled the retrieved information.

Instead, temporarily separate the two components.

Take your failed question and run the retriever directly. Inspect the top-k results before they reach the LLM.

Ask yourself:

Does the correct supporting passage appear anywhere in the retrieved results?

There are two possible outcomes.

Case A: The correct passage is missing

This is primarily a retrieval investigation.

Look at:

The embedding model and its suitability for your content.
How queries and documents are preprocessed.
Chunk boundaries and overlap.
Metadata filters.
The number of candidate results retrieved.
Dense versus keyword-based search.
Index freshness and document coverage.

Case B: The correct passage is present

Good. You've narrowed down the problem.

Now inspect what happens after retrieval.

The relevant passage might be ranked too low, removed during context assembly, truncated because of the token budget, or ignored by the LLM.

This distinction saves time because you no longer need to change every component at once.

Which retrieval metrics should you use?

If you're building a serious RAG application, you need more than a few successful demos.

Use a representative evaluation dataset with questions and human-reviewed relevant passages.

Metric	What it tells you
Recall@k	Did relevant evidence appear in the top-k results?
Precision@k	How much of the retrieved set was relevant?
Mean Reciprocal Rank (MRR)	How highly ranked was the first relevant result?
nDCG	Were results ordered well when relevance has different degrees?

For example, suppose the correct passage is absent from the top five results for a particular question.

Improving the final answer prompt will not make that passage magically appear in those results.

You need to investigate retrieval first.

One important detail: define what counts as relevant evidence for your task. A similarity score alone doesn't prove that a passage is useful, and retrieval metrics are only as reliable as the labels used to calculate them.

6. When should you introduce hybrid search or reranking?

Let's say your dense retriever performs reasonably well, but you've noticed a recurring pattern.

It struggles with product IDs, exact error codes, policy numbers, technical identifiers, or terminology that must match precisely.

This is where keyword search can help.

Dense vector search is useful for semantic similarity. Lexical search, such as BM25, can be valuable when exact terms matter.

Hybrid retrieval combines these approaches so the system can benefit from both semantic matching and keyword matching.

One way to combine the results is Reciprocal Rank Fusion (RRF).

RRF combines ranked lists using a formula of this form:

[ \operatorname{RRF}(d)=\sum_{r\in R}\frac{1}{k_0+\operatorname{rank}_r(d)} ]

Here, (d) is a document, (R) represents the ranked result lists, and (k_0) is a smoothing constant. Documents ranked highly across multiple lists tend to receive stronger combined scores.

You can then introduce a reranker to evaluate query-document pairs more carefully and reorder the candidate passages.

But here's the practical point.

Hybrid search and reranking aren't automatic fixes for every RAG problem.

If the correct document was never indexed, neither technique will recover missing information. If a metadata filter excludes the document, reranking won't solve the root cause.

First identify the failure pattern. Then compare the baseline against hybrid retrieval and reranking using the same test dataset.

Measure improvements in retrieval quality alongside latency and cost. Keep the additional complexity only if the results justify it.

7. Check metadata filters and access permissions

This is a particularly frustrating failure because everything can look correct at first.

Your document exists. Its embeddings are valid. A search without filters returns the expected passage.

But the production application still misses it.

Why?

A metadata filter may be excluding the document.

Imagine a knowledge base containing documents for several product versions. A query is restricted to version 3.0, but the relevant policy was accidentally indexed with a missing or incorrect version field.

The retriever cannot return it because the filter removes it before or during candidate selection.

Inspect filters for:

Tenant or customer ID.
Product or document version.
Date and document status.
Language and category.
User access permissions.
Missing or incorrectly formatted metadata.

Compare filtered and unfiltered retrieval in a controlled test environment.

However, never fix a retrieval issue by simply removing authorization filters. In a multi-tenant or enterprise application, that could expose documents to users who shouldn't have access to them.

The correct solution is to repair the metadata or filtering logic while preserving access controls.

8. Inspect the exact context sent to the model

Suppose your retriever returns the correct passage.

You might reasonably conclude that retrieval is working and move on.

But don't skip this step.

The retrieved chunks aren't necessarily identical to the final context passed to the LLM.

Your application might:

Truncate context to fit the token budget.
Remove duplicate chunks.
Reorder passages.
Merge chunks incorrectly.
Drop source IDs during formatting.
Include conflicting versions of a document.

The retriever could be correct while the context builder introduces the actual failure.

Log the assembled context in a controlled, privacy-safe debugging environment. Compare it with the original retrieved results and verify that the relevant evidence survives every transformation.

Also check whether your prompt clearly separates retrieved evidence from instructions and untrusted document content.

This is particularly important when retrieved documents contain user-generated content or instructions that shouldn't be treated as trusted system instructions.

Debug the context the model actually receives, not just the context you intended to send.

9. How do you know whether the problem is retrieval or generation?

Let's bring the previous steps together.

Imagine your RAG application gives a wrong answer even though you believe the correct information exists in the knowledge base.

Use this diagnostic guide.

What you observe	Where to investigate first
Correct evidence is absent from retrieved results	Ingestion, chunking, query processing, filters, embeddings, retrieval settings
Correct passage appears but ranks too low	Candidate retrieval, hybrid search, reranking
Retriever finds the right passage, but final context omits it	Context assembly, deduplication, ordering, token budget
Correct evidence reaches the LLM, but the answer is wrong	Prompt instructions, conflicting evidence, model behavior, generation settings
Answer sounds convincing but is unsupported	Grounding, abstention behavior, answer evaluation
Answer is correct, but citations point to the wrong source	Source ID mapping, citation construction, response formatting

There's one experiment I particularly recommend.

Take a failed question and manually provide the known correct passage directly to the LLM, using the same relevant instructions as your application.

What happens?

If the answer is still wrong: Investigate generation, prompt design, conflicting evidence, and how the model interprets the passage.
If the answer becomes correct: Investigate the retrieval and context-building path that failed to supply that evidence in the normal workflow.

This is a useful diagnostic experiment, not absolute proof. Manually supplying context can change the amount, ordering, and framing of the evidence, so compare the conditions carefully.

10. Build a small evaluation dataset before making more changes

Here's where debugging becomes much more reliable.

Instead of repeatedly testing random questions, create a small dataset of representative queries and expected supporting evidence.

You don't need thousands of examples to start.

Begin with questions that cover:

Common questions users ask frequently.
Questions that use different terminology from the source documents.
Questions involving exact identifiers or technical terms.
Questions requiring information from multiple passages.
Questions involving outdated and current document versions.
Questions whose answers aren't present in the knowledge base.
Questions where users lack permission to access the relevant document.

For each test case, record the expected evidence, the acceptable answer, and whether the system should abstain.

Run the same dataset after every significant change.

Track separate measures for:

Retrieval relevance and recall.
Final answer correctness.
Whether answers are supported by the retrieved evidence.
Citation correctness.
Latency and cost.

This separation matters because an improvement in one metric can hide a regression in another.

For example, retrieving more chunks may increase recall while also adding irrelevant context and increasing latency.

When a change improves one area but hurts another, you can make an informed trade-off rather than relying on a subjective impression that the chatbot feels better.

11. How to explain this in a GenAI engineer interview

Now imagine an interviewer asks:

"Your RAG application retrieves the wrong context. How would you debug it?"

Don't immediately respond with a list of tools: ChromaDB, FAISS, Pinecone, BM25, or rerankers.

The interviewer is usually looking for your diagnostic reasoning, not just your familiarity with libraries.

A stronger answer would sound like this:

"I would first reproduce the failure using the original query, the expected supporting passage, the retrieved results, and the final answer. Then I would inspect the ingestion output to confirm that the correct information was extracted and indexed.

Next, I would evaluate retrieval independently of generation. If the relevant passage is missing from the top-k results, I would investigate chunking, embeddings, query processing, metadata filters, and retrieval configuration.

If the passage is present, I would check its ranking and trace the context assembly process to confirm that the evidence reaches the model.

If the correct evidence is present in the final prompt but the answer remains incorrect, I would investigate generation, grounding instructions, and conflicting context. I would validate each change against a fixed evaluation dataset and track retrieval quality, answer correctness, latency, and cost separately."

Notice what makes this answer effective.

You're not guessing that the embedding model is the problem. You're explaining how you would isolate the failure, collect evidence, test a hypothesis, and measure the result.

That is the difference between knowing RAG terminology and demonstrating engineering judgment.

A practical debugging checklist

Before you finish investigating a RAG failure, make sure you've answered these questions:

You don't need to change every component to fix a RAG system.

You need to identify the component responsible for the failure and prove that your change addresses it.

Final thoughts

When a RAG application gives the wrong answer, resist the temptation to change everything at once.

Start with the evidence. Trace the query through ingestion, chunking, retrieval, ranking, context assembly, and generation.

Isolate the failure. Test one hypothesis at a time. Use a fixed evaluation set to measure whether the system actually improves.

The goal isn't to make your RAG pipeline more complicated.

It's to make it more reliable, easier to debug, and easier to explain to another engineer.

And if you're preparing for a GenAI engineering interview, remember this:

A good engineer doesn't just know which tools to use. They know how to find the real problem before choosing the tool.

Keep learning with Roy's AI Lab

Want to become more confident building and debugging production-ready GenAI applications?

Explore Roy's AI Lab for practical RAG guides, GenAI engineering concepts, and interview preparation resources. Pick a topic, work through the examples, and turn what you learn into something you can build and explain.