How Do You Evaluate a RAG System? The 4 Metrics That Actually Matter

Why "It Sounds Right" Isn't Good Enough

An interviewer once asked me: "How do you evaluate a RAG system?"

It sounds like a simple question. It isn't. It reveals whether you know how production RAG systems are actually measured, or whether you've only ever eyeballed a demo.

Here's the uncomfortable truth: a RAG bot can give a confident, well-written answer that is completely wrong. Language models are fluent by default — fluency is not the same thing as being grounded in fact. Teams that ship RAG without a real evaluation harness usually discover this the hard way: a support bot confidently quoting a refund policy that changed three months ago, or a legal search tool citing a clause that doesn't exist in the contract.

"It sounds right" is not an evaluation strategy. You need numbers, and you need to know exactly what those numbers are measuring. This post goes deep into that.

RAG Has Two Halves — and Each Fails Differently

A RAG pipeline fails in one of two places:

• Retrieval — did the system find the right chunks from the knowledge base?

• Generation — did the model actually use those chunks correctly to produce the answer?

These are independent failure surfaces. You can have perfect retrieval and a hallucinating model. You can have a perfectly faithful model working off garbage context.

Treating "the answer was wrong" as a single undifferentiated bug is the single most common mistake I see in RAG debugging. It sends people chasing prompt tweaks when the actual problem is three steps upstream in the retriever.

This is why RAG evaluation frameworks split scoring into two metrics for each half of the pipeline.

The 4 Metrics, With the Actual Math

Most explainers give you a one-line definition. Here's what's actually happening under the hood in tools like RAGAS.

Generation-side metrics

1. Faithfulness

Definition: the proportion of claims in the generated answer that can be inferred from the retrieved context.

How it's actually computed: the answer is first decomposed into individual atomic claims (usually via an LLM call: "break this answer into a list of standalone factual statements").

Each claim is then checked against the retrieved context using natural language inference (NLI) — essentially asking "does the context entail this claim?"

Example: an answer makes 5 claims. The retrieved context supports 4 of them. Faithfulness score = 0.8.

Why this matters more than it sounds: faithfulness is claim-level, not answer-level. A single fabricated sentence in an otherwise perfect answer will drag the score down — which is exactly the granularity you want, because in production, one wrong sentence is often the one that gets a user in trouble.

2. Answer Relevance

Definition: how well the generated answer addresses the actual question asked.

How it's actually computed: the model is asked to generate several hypothetical questions that the given answer would be a good response to. Each generated question is embedded and compared (cosine similarity) against the embedding of the original question.

Example: user asks "How do I reset my password?" The answer thoroughly explains account security best practices but never actually gives reset steps. The reverse-engineered questions from that answer ("What are good password practices?") will be semantically distant from the original question — low relevance score, even though nothing in the answer was factually wrong.

This is the metric most people forget to check because a relevant-sounding answer feels correct. It's the "technically true, doesn't answer the question" failure mode.

Retrieval-side metrics

3. Context Precision

Definition: of the chunks retrieved, how many are actually relevant to answering the question — and are the relevant ones ranked near the top?

How it's actually computed: for each retrieved chunk (in rank order), an LLM or classifier judges relevance (yes/no). Precision is then computed as a weighted average, where relevant chunks appearing earlier in the ranking are weighted more heavily (this is essentially Average Precision, borrowed from information retrieval).

Example: you retrieve 10 chunks. Chunks 1, 2, and 5 are relevant; the rest are noise. Because two of the three relevant chunks are ranked at the very top, precision is high — even though 7 of 10 chunks are irrelevant.

If the relevant chunks were ranked 7, 8, 9 instead, precision would be much lower for the same relevant-chunk count.

This is why reranking exists: precision doesn't just reward "did you retrieve relevant things," it rewards "did you put them where the model will actually pay attention to them."

4. Context Recall

Definition: whether the retrieved context contains all the information needed to answer the question — measured against a ground-truth reference answer.

How it's actually computed: the reference answer (your "golden" answer, usually human-written) is broken into claims, same as faithfulness. Each claim is checked against whether it can be attributed to something in the retrieved context.

Example: your golden answer for "What's our refund policy?" has 3 key facts: the time window, the exceptions, and the process to request one. Retrieval only surfaced the chunk with the time window. Context Recall = 1/3 ≈ 0.33 — even if the model writes a beautifully faithful answer using only that one fact, the answer will be incomplete.

Important: context recall is the one metric on this list that requires a labeled reference dataset. You can't compute it from live traffic alone — which has real implications for how you build your evaluation pipeline (more on this below).

A Worked Example: Debugging With Numbers

Say you run this eval on 50 questions from your support bot and get these averages:

| Metric | Score | |---|---| | Faithfulness | 0.94 | | Answer Relevance | 0.91 | | Context Precision | 0.52 | | Context Recall | 0.88 |

Read this like a diagnosis, not a report card. Faithfulness and relevance are both high — generation is doing its job well. Recall is decent, so retrieval is finding most of what it needs. But precision is low: the retriever is pulling in a lot of noise alongside the relevant chunks.

The fix here isn't a better prompt. It's a reranker, or tighter top_k, or better chunk boundaries — because the diagnosis points squarely at retrieval ranking, not generation. This is the entire point of splitting metrics: the numbers tell you where to spend your engineering time, instead of guessing.

Which Metric Do You Fix First?

| Symptom | Root cause | Fix | |---|---|---| | Retrieval misses important documents | Low Context Recall | Widen top_k, improve chunking, add hybrid search (BM25 + vectors) | | Retrieval returns too much irrelevant noise | Low Context Precision | Add a reranker, tighten chunk size, improve embedding model | | Answer hallucinates despite good retrieval | Low Faithfulness | Stricter grounding prompts, lower temperature, add citation requirements | | Answer is accurate but off-topic | Low Answer Relevance | Better prompt instructions, query rewriting/expansion |

The interviewer isn't testing whether you can list four metrics. They're testing whether you can trace a symptom back to a root cause in the pipeline — the table above is that skill, condensed.

Building an Actual Evaluation Pipeline

Here's what this looks like in code, using RAGAS (the most widely adopted framework for this):

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset

# Your evaluation dataset needs 4 things per row:
# question, answer, contexts (list), ground_truth
eval_data = Dataset.from_dict({
    "question": questions,
    "answer": generated_answers,
    "contexts": retrieved_contexts,   # list of lists
    "ground_truth": reference_answers,
})

result = evaluate(
    eval_data,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)

df = result.to_pandas()

# Flag anything below your quality bar for manual review
low_quality = df[df["faithfulness"] < 0.8]
for _, row in low_quality.iterrows():
    flag_for_review(row["question"], row["answer"])

The part people skip: building the golden dataset

Context Recall and RAGAS's ground-truth-based faithfulness checks need a labeled dataset — question, reference answer, ideally the "correct" chunks too. In practice, teams build this by:

1. Sampling real user questions from logs (not synthetic ones — synthetic questions are too clean)

2. Having a domain expert (or a careful LLM pass, human-reviewed) write reference answers

3. Keeping the set small but representative — 50–100 well-chosen examples catch more real issues than 1,000 lazy ones

4. Re-running this set on every meaningful pipeline change (new chunking strategy, new embedding model, new prompt) — this is your regression test suite for RAG

Offline eval vs. online eval

These four metrics run in two very different modes in production:

• Offline / CI evaluation: run against your golden dataset before every deploy. This is where Context Recall lives, since it needs ground truth. Treat it like a test suite — a PR that drops faithfulness by 10 points should block the merge the same way a failing unit test would.

• Online / live evaluation: faithfulness and answer relevance can run on live traffic without ground truth (they compare the answer against the retrieved context, not a reference answer). This is how you catch drift — a retriever that was fine at launch but degrades as your document set grows.

Context precision can technically run online too, but needs an LLM call per retrieved chunk, which gets expensive at scale — most teams sample a percentage of live traffic rather than scoring 100% of it.

Tool Comparison

| Tool | Best for | Notes | |---|---|---| | RAGAS | Standardized metric definitions, quick setup | The four metrics above are essentially RAGAS's core offering; widely cited in interviews for a reason |

| TruLens | Tracing + evaluation together | Good if you want to see the full trace (retrieval → generation) alongside scores, not just the final numbers |

| DeepEval | Pytest-style testing | Feels like writing unit tests — assert_faithfulness(...) — good fit if your team already thinks in test suites |

| Arize Phoenix | Production observability | Stronger on live monitoring/drift detection than on offline golden-set evaluation |

None of these are mutually exclusive — a common setup is RAGAS or DeepEval in CI, and Phoenix or LangSmith for live production tracing.

Common Mistakes That Quietly Break Evaluation

• Using an LLM-as-judge without checking judge consistency. Every one of these metrics leans on an LLM to decompose claims or judge relevance. Run your eval twice on the same data — if faithfulness swings by 0.1+ between runs, your judge model or prompt needs tightening (lower temperature on the judge call, more constrained rubric).

• Testing on synthetic questions only. Golden sets built entirely from LLM-generated questions tend to be too clean and don't reflect how real users phrase things (typos, vague phrasing, multi-part questions). Always seed from real logs.

• Ignoring cost. Faithfulness and context precision both require multiple LLM calls per evaluated example. Running this on every single production query is usually not economical — sample, don't census.

• Chasing one metric in isolation. Optimizing purely for context precision (e.g., aggressive reranking that returns only 2 chunks) can quietly tank recall. Always look at the four scores together, the way you'd look at precision and recall together in any classic IR system.

• No threshold, no action. A dashboard of scores that nobody acts on isn't an evaluation pipeline. Set a concrete bar (e.g., faithfulness < 0.8 → auto-flag for human review) and wire it into your workflow.

How I'd Structure This in an Interview

Q: "How do you evaluate a RAG system?"

A good RAG system isn't just about generating the right answer — you have to evaluate retrieval quality and generation quality separately. Four metrics cover this: Faithfulness and Answer Relevance for generation, Context Precision and Context Recall for retrieval.

Q: "Which metric would you optimize first?"

It depends on where the pipeline is failing. If retrieval misses documents, that's low recall. If retrieval returns noise, that's low precision. If the answer hallucinates despite good retrieval, that's a faithfulness problem — and the fix for each of those is completely different.

Q: "How would you actually implement this?" (the follow-up that trips people up)

I'd build a golden dataset from real user questions with human-reviewed reference answers, run RAGAS or DeepEval against it in CI on every pipeline change, and separately run faithfulness and relevance checks on a sample of live traffic to catch drift over time. Anything below threshold gets flagged for review rather than shipped silently.

That third answer is the one that signals you've actually built this, not just read about it.

Practice Question

"Walk me through how you'd debug a RAG system that keeps giving outdated answers."

If you can answer that by separating retrieval freshness (index staleness, re-sync cadence) from a faithfulness or recall problem — you're ready for this question in any interview, and more importantly, ready to actually fix it in production.

---

When evaluating your own RAG systems, which metric has exposed the biggest bottleneck? I'd genuinely love to hear the war stories the weird ones are always in context precision.

Practice Question

"Walk me through how you'd debug a RAG system that keeps giving outdated answers."

If you can answer that by separating retrieval freshness (index staleness, re-sync cadence) from a faithfulness or recall problem — you're ready for this question in any interview, and more importantly, ready to actually fix it in production.

---

When evaluating your own RAG systems, which metric has exposed the biggest bottleneck? I'd genuinely love to hear the war stories — the weird ones are always in context precision.

---

Want to master such interview Q&A?

👉 Get the full handbook here

550+ production-focused interview questions across 14 chapters covering Python, RAG, AI Agents, LangGraph, Guardrails, AWS, System Design, and more.

Designed to help you build interview confidence, think like an AI engineer, and answer beyond textbook definitions.