Question 1

What is an embedding in RAG?

Accepted Answer

An embedding is a list of numbers that represents the meaning of a piece of text. Texts with similar meaning have embeddings that are close together. In RAG, both the documents and the user's question are embedded. The system then searches for the document chunks whose embeddings are closest to the question, and passes them to the LLM as context.

Question 2

What does top-k retrieval mean in RAG?

Accepted Answer

Top-k retrieval returns the k chunks most similar to the user's question. For example, with k set to 5, the system passes the five best-matching chunks to the model. Choosing k is a trade-off. A small k may miss the right context, while a large k adds noise and uses more of the model's context window. Test a few values on a set of known questions to pick one.

Question 3

The Question That Ends Most AI Interviews Early

Accepted Answer

The Scenario: You've shipped a RAG-based support chatbot. It works fine in testing. Two weeks after launch, it's live for 50,000 users. One night, latency spikes from 800ms to 11 seconds. Users are abandoning mid-conversation. Your vector DB is healthy. Your LLM provider status page shows no incidents. You have 15 minutes before your manager pings you on Slack. Walk me through exactly what you check, in order, and why. The Answer: Most candidates freeze here not from lack of knowledge, but because they've never been trained to think in this order. 1. Isolate the layer first. Check retrieval latency and generation latency separately. If retrieval is fast but generation is slow, it's not your vector DB. 2. Rule out the boring stuff before the exotic stuff. Cold cache after a deploy? Autoscaler lagging behind a traffic spike? A cron job or batch ingestion job hammering the same DB? These cause most "mystery" spikes. 3. Check for silent throttling. LLM providers often rate-limit per-token or per-org without a status page incident. Look at response headers, not just the status page. 4. Check for one noisy neighbor. Is one tenant, one query pattern, or one runaway retry loop eating shared capacity? A single misbehaving client can look like a global outage. 5. Check what changed in the last 24 hours. New prompt version, new embedding model, new chunk size, a config push, most incidents trace back to something that shipped recently. 6. Only then, suspect the vector DB or LLM provider itself. In most real incidents, the bug is on your side of the fence. That order, isolate, rule out the boring, check for throttling, check for noisy neighbors, check recent changes, then suspect the vendor, is what interviewers are actually listening for. Not the right answer. The right sequence.

Question 4

An interviewer asked: When would you choose RAG over fine tuning?

Accepted Answer

I paused for a second. A lot of people think it's an either/or decision. It isn't. RAG and fine-tuning solve different problems. When to Use RAG Use RAG when your knowledge changes frequently. Think about internal documentation, company policies, product catalogs, or news. Retraining a model every time the data changes is expensive and impractical. RAG keeps the model grounded in the latest information while providing source attribution and reducing hallucinations. That means: No retraining every time new documents are added. Responses stay aligned with the latest knowledge. Source citations improve trust and explainability. Better factual accuracy for dynamic information. When to Use Fine-Tuning When the goal is to teach the model a specific writing style or behavior. When you need consistent domain-specific reasoning. When minimizing latency is more important than retrieving external knowledge. What I'd Use in Production The interviewer then asked: What would you use in a production AI application? The answer was straightforward: I would combine both. Fine-tune the model for behavior, tone, and task-specific patterns. Use RAG to inject the latest factual knowledge at inference time. The interviewer wasn't testing whether I knew the definitions of RAG and fine-tuning. They were testing whether I understood the engineering trade-offs behind each approach. 💬 If you were building an enterprise AI assistant today, would you rely only on fine-tuning, only on RAG, or use both?