Similar Is Not the Same as Supported: Why Search Results Need Checking

Quick answer

Embedding similarity finds passages that are close in meaning. It doesn't check whether they apply to this user.

Before you use a passage as evidence, check the plan, the country, the date and any other conditions that change the answer.

A similarity score is not the chance that an answer is correct. Test any threshold on your own data before you rely on it.

A customer asks your support assistant a simple question. The search engine returns a passage that sounds almost exactly right. The assistant gives a confident answer based on it.

Then you find out the passage was about a different plan, in a different country, under an older policy.

The search did its job. It found text that was similar to the question. The mistake was assuming that similar meant supporting.

Why read this? Because most RAG failures look like "the model made something up," when the real problem is that a plausible passage was treated as proof. Knowing the difference is one of the most useful skills in this field.

What similarity actually measures

Embedding-based similarity compares numerical representations of text. Sentence Transformers, the popular library for this, documents cosine similarity as its default way of comparing two embeddings.

That comparison tells you how close two pieces of text are in meaning, according to the model. It doesn't tell you whether a passage:

Applies to the user who is asking.
Is current.
Is factually correct.
Supports the answer you want to give.

Those are separate checks. You need to make them yourself.

Try this first

A user asks:

"Can a premium subscriber in India cancel within 14 days?"

Here are three candidate passages. These policies are fictional, for teaching.

Passage A: "Standard subscribers in India can cancel within 14 days."

Passage B: "Premium subscribers in India can cancel within 7 days."

Passage C: "Premium subscribers in another market can cancel within 14 days."

Before reading on, answer two questions. Which passage directly addresses the requested plan and country? And which passages could tempt you into an overgeneralised answer?

Check your thinking

Passage B is the only direct match. It names the right plan and the right country. Its answer is 7 days, not 14.

Passage A matches the country and the topic, but it covers the standard plan. Using it would give a plausible but wrong answer.

Passage C matches the plan and the 14-day period mentioned in the question, but it covers a different market. It is the most tempting mistake, because it seems to confirm what the user already suspected.

The 14 days in the question doesn't override the evidence. The user's expectation is not a source. The passage that applies is the one that counts, even if it gives a less convenient answer.

Follow the checks

Here's the flow that turns a search result into a supported answer:

Question
   ↓
Retrieve similar candidates
   ↓
Check plan, country, date and other conditions
   ↓
Identify the passage that actually supports the answer
   ↓
Answer within that evidence

Similarity is the first step. It helps you find candidates. It doesn't finish the job. The applicability check is where most of the real work happens.

What if a score is 0.9?

Don't read a similarity score as "a 90% chance this answer is correct."

The meaning of a score depends on the model, the similarity function and the task. A 0.9 from one setup may mean something quite different from a 0.9 in another. The score tells you how close two texts are in that model's space. It says nothing about whether the passage is the right one to rely on.

This note deliberately gives no invented ranking scores, because any number you saw here would be made up.

An interview follow-up

"Can I use one similarity threshold for every corpus?"

Don't assume so. A threshold that looks sensible on one set of documents may be too loose or too strict on another.

Evaluate the threshold on examples that look like your real workload. Include relevant passages, irrelevant passages and near-misses. Check how the chosen model and scoring setup behave on your data. A good threshold is one you've tested, not one you borrowed.

Our RAG evaluation guide explains how to measure retrieval quality in a way that doesn't rely on a single score.

Prove you understood it

A passage has the highest similarity score in your search. But it refers to a policy that expired last year.

Should it automatically become the source for the answer?

Pause before reading on.

The answer: no. Ranking and applicability are different checks. The passage is similar to the question, but it may not be current, so it can't support a current answer. The system should either find a valid passage or say that the current evidence isn't available.

A quick checklist

Before you use a retrieved passage as evidence, ask yourself:

Does it cover the exact plan, product or user type in the question?
Does it apply to the right country, region or setting?
Is it current, and does it have a date or version I can check?
Does it actually support the answer I want to give?
Am I treating a similarity score as proof, when it's only a ranking signal?

A note on limits

The policy examples are fictional, and this note doesn't claim that any particular embedding model would rank these passages in a specific order. No embeddings were computed for this post. The point is the reasoning, not a benchmark.

Keep learning

Want to connect this to the rest of your RAG thinking?

RAG explained, in plain English: https://roy-s-ai-lab.vercel.app/blog/rag-explained-why-your-llm-doesn-t-actually-know-anything-and-how-to-fix-that
How to debug a RAG system that retrieves the wrong context: https://roy-s-ai-lab.vercel.app/blog/how-to-debug-a-rag-system-that-retrieves-the-wrong-context
Free RAG practice sets: https://roy-s-ai-lab.vercel.app/topics/rag
Full guides and scenario packs: https://roy-s-ai-lab.vercel.app/products

Try this: take a question from your own project and list every retrieved passage that looks relevant. For each one, write down whether it actually applies, and why.

Source

Sentence Transformers, "Semantic Textual Similarity": https://sbert.net/docs/usage/semantic_textual_similarity.html

The source was checked and confirms that cosine similarity is the default similarity function. The address in the original draft was incorrect and has been corrected here. The policy examples, the checklist and the exercise are original teaching material.

Ritesh Rai