Prompt Caching Is Not Answer Caching: What Your AI App Is Really Reusing

Quick answer

• Prompt caching reuses work the model has already done on the start of a prompt. It doesn't reuse the answer.
• A match needs the same prefix, character for character. Similar meaning isn't enough.
• Measure cache hits and real cost on your own traffic before you claim any saving.

You read that prompt caching can cut your bill by a large share. Excited, you enable it, and the dashboard shows cache hits. Problem solved?

Not quite. A cache hit tells you something specific, and it's easy to misread. It doesn't mean your app returned a saved answer. It means the model skipped some work it had already done on a matching start of your prompt, then wrote a fresh response.

This post explains the difference, why small layout changes can quietly kill your savings, and how to check the numbers properly.

Why read this? Because teams often claim savings they haven't measured, and interviewers like to test whether you can tell two similar-sounding ideas apart.

Two different things

People often mix up two ideas.

Prompt caching reuses processing. When a request starts with the same instructions, examples or reference material as an earlier one, the model can reuse the internal state it built for that shared start. It still reads the new question and writes a new response.

Answer caching stores a finished reply. Your application saves the response and returns it later, when a request matches a rule you wrote.

These are different mechanisms, and they have different risks. Answer caching can serve an out-of-date reply. Prompt caching doesn't skip generating the response, so the answer stays fresh, but it can only save work on the shared start.

Follow the difference

PROMPT CACHING

Shared instructions ──┐
Shared reference ─────┼── Reused processing
New question ─────────┘           ↓
                          New answer is written


ANSWER CACHING

Request
   ↓
Your app finds a saved answer that matches its rules
   ↓
Returns that saved answer

A prompt-cache hit doesn't mean the answer came from storage. The new answer is still generated.

Five things to understand

Shared input can be reused. Repeated instructions, examples and reference material can form a reusable prefix. The longer and more stable the shared start, the more there is to reuse.

The match is exact. The caching described in OpenAI's guide depends on the actual prefix matching, not on similar meaning. Two prompts that say the same thing in different words don't share a cache.

Layout matters. Changing content near the beginning can shrink or break the shared prefix. Put stable content first, and request-specific content last.

Rules depend on the provider and model. Supported models, minimum sizes, how long entries are kept, and pricing all vary. Check the documentation for the exact model and settings you use. Minimum sizes and rates have changed between model generations, so don't rely on an old figure.

Savings have to be measured. Caching being available doesn't prove you're saving money. Look at cache usage, latency and billing on real traffic.

A worked example: PolicyPal

PolicyPal is a fictional company-policy assistant. Each request sends three parts: stable operating instructions, a stable policy reference section, and the current user question.

Two requests share the first two parts. That shared start may be eligible for reuse.

Now suppose the team adds a timestamp at the very top of every request: "Current time: 14:32:05." Every request now begins differently, so the shared prefix changes right at the start. The reusable part may shrink to almost nothing.

The fix isn't to delete the timestamp. The timestamp matters for some answers. The fix is to move it, so the stable content comes first and the timestamp sits near the question at the end.

Don't remove necessary information just to improve cache hits. Reorder it first.

Try this first

Two prompts give the same instructions, but one uses different wording. Does that give you a cache hit?

Write your answer before reading on.

The answer: no, not automatically. The mechanism depends on the actual shared prefix. Different wording means a different prefix, even when the meaning is identical. If you want reuse, keep the shared text byte-for-byte the same, and use the same wording every time.

Pause and think

Your dashboard shows a high cache-hit rate. Someone says, "So most of our answers are now coming from the cache, and we don't need to worry about freshness."

Is that right?

Pause before reading on.

The answer: no. A high cache-hit rate describes reused processing on the prompt's start. It doesn't mean answers are stored and reused. Every response is still generated, so freshness still depends on your sources and your design. Keep answer caching and prompt caching in separate boxes in your head, and in your design review.

What should you measure?

Compare representative requests before and after a change. Record:

• The structure of the prompt, and which parts are stable.
• How long the repeated prefix is.
• The cached-input usage the provider reports.
• End-to-end latency.
• Input usage, and the charges that apply.
• Whether the application still behaves correctly.

Be careful with headline numbers. A saving estimated from one successful request tells you very little. Use enough real traffic to see a pattern, and check that quality hasn't dropped.

A quick checklist

Before you claim a prompt-caching saving, ask yourself:

• Is my shared prefix actually identical from one request to the next?
• Have I put stable content first, and changing content last?
• Does my shared prefix meet the model's minimum size?
• Am I measuring cache usage and cost on real traffic?
• Am I keeping prompt caching separate from answer caching in my design?

Interview follow-up

"Does prompt caching mean we can reuse previous answers?"

A clear answer: "No. Prompt caching reuses the work done on an identical start of a prompt. The model still generates each new response. If I wanted to reuse answers, that would be a separate design, with its own rules for freshness and correctness. I'd measure cache usage and cost before claiming any saving, and I'd keep the shared content stable at the start of the prompt."

A note on limits

The provider's rules change over time, including minimum sizes, retention and pricing. Check the current documentation for your model before you design around any number. The example here is fictional, and this post doesn't include a benchmark.

Keep learning

Want to connect this to the rest of your optimisation thinking?

• Faster calls do not always mean a faster app: https://roy-s-ai-lab.vercel.app/blog/faster-calls-do-not-always-mean-a-faster-application
• Design the problem before choosing the stack: https://roy-s-ai-lab.vercel.app/blog/design-the-problem-before-choosing-the-stack
• Free RAG practice sets: https://roy-s-ai-lab.vercel.app/topics/rag
• Full guides and scenario packs: https://roy-s-ai-lab.vercel.app/products

Try this: write down the three parts of one of your prompts. Mark which parts never change. Check whether they come first.

Sources

• OpenAI, "Prompt caching": https://developers.openai.com/api/docs/guides/prompt-caching
• OpenAI Cookbook, "Prompt Caching 201": https://developers.openai.com/cookbook/examples/prompt_caching_201

The first source was checked for the prefix-matching, layout and model-dependence points. The cookbook page was referenced in the OpenAI cookbook repository, but its exact address wasn't opened during this check, so please confirm it before you share the post. The PolicyPal example and checklist are original teaching material.

Ritesh Rai