Cache tokens:your model’sdéjà vu.
When a prompt begins the same way as one the model saw recently, that matching beginning can be reused. Less repeated work. Lower latency. A happier invoice.
- AUTOon eligible requests
- EXACTprefix matches only
- 1,024+tokens on OpenAI
4,352 tokens skipped the expensive part
09:41:02 prefix hash matched
09:41:02 attention state restored
09:41:02 only fresh tail processed →
The model doesn’t remember your answer. It reuses the warm-up.
Prompt caching stores reusable intermediate attention state for a matching input prefix. The model still generates a fresh output. Think mise en place, not reheated leftovers.
“I’ve already read your 40-page instruction manual. Just tell me what today’s question is.”
Build a prompt the cache can love.
Pick a workload, then commit the cardinal sin: move the changing part to the front. Watch one tiny choice nuke the reusable prefix.
Stable rules, tools, and repository context
A match is a zipper. Break one tooth, lose everything after it.
Cache matching moves left to right. It can reuse the exact prefix before the first difference—never skip a changed bit and resume later.
Great: the first difference is near the tail, where variable data belongs.
Now multiply that déjà vu.
The first request does the full read. Similar requests reuse the shared beginning for less. That is the whole trick.
Fixed example: a 6,000-token shared beginning. Cached reads cost 25% of a normal read. Every request also has a fresh 400-token tail.
in this simplified 12-request example
Excludes output tokens, cache-write premiums, minimum cacheable lengths, and provider-specific pricing. This is intuition math, not a quote.
Look for big, boring beginnings.
The ideal workload repeats a long, identical setup and changes only a small tail. In AI engineering, that pattern is everywhere.
Coding agents
Reuse the system prompt, tool definitions, repo map, and coding conventions. Append today’s task at the end.
static setup → “fix flaky test”Support copilots
Keep policies, product facts, tone rules, and tool schemas stable. Add the customer conversation last.
stable playbook → customer issueBatch extraction
Reuse a large extraction schema and worked examples. Put each new document after that fixed instruction set.
schema + examples → new document- Long system prompts
- Stable tool definitions
- Repeated few-shot examples
- Shared reference documents
- Multi-turn conversations with a stable beginning
- Short one-off prompts
- Everything changes every request
- Personalization injected at the top
- Randomly reordered tools or examples
- Traffic too sparse for the cache to stay warm
The anatomy of a cache hit.
- 1ROUTE
Fingerprint the beginning
The service hashes an initial prompt prefix and uses it to route similar requests toward the same cache neighborhood.
- 2LOOK UP
Find the longest exact prefix
The cache checks for matching prompt state. If several prefixes match, the longest useful one wins.
- 3RESUME
Restore state, process the tail
Cached attention state is loaded. The model processes only the uncached suffix, then generates a brand-new response.
Five rules for cache-friendly prompts.
- 01
Put reusable content first.
System instructions, tool schemas, examples, and shared context lead the prompt.
- 02
Put user-specific content last.
Names, timestamps, live data, current tasks, and conversations belong at the tail.
- 03
Keep serialization deterministic.
Don’t shuffle tools, object keys, examples, or whitespace for fun. Exact serialized stability matters.
- 04
Group related traffic when your API supports it.
A stable prompt cache key can improve routing for requests that share the same long prefix.
- 05
Measure actual hits.
Log the cached-token count. A beautifully structured prompt with zero hits is just typography.
What it looks like in an API response.
You usually don’t “fetch” a prompt cache yourself. Send a stable prefix, then inspect usage metadata to see what was reused.
const response = await client.responses.create({
model: "your-cache-capable-model",
// Keep this stable for requests with the same prefix.
prompt_cache_key: "coding-agent:repo-42:v3",
input: [
{ role: "system", content: SYSTEM_PROMPT },
{ role: "developer", content: TOOL_SCHEMAS },
{ role: "developer", content: REPO_CONTEXT },
// The changing part goes last.
{ role: "user", content: currentTask }
]
});{
"usage": {
"input_tokens": 4680,
"input_tokens_details": {
"cached_tokens": 4352
},
"output_tokens": 612,
"total_tokens": 5292
}
}That’s the receipt. The response was still generated fresh; 4,352 input tokens came from the prompt cache.
Three common wrong ideas.
“It caches the answer.”
Nope. Prompt caching reuses work done while reading the input. Output generation still happens again.
CACHE
“Semantically similar is close enough.”
Not here. Similar meaning is irrelevant if the prefix differs. This is exact-match territory.
“More cached tokens means lower quality.”
The computation is reused, not approximated. Same prefix, same state—just less repeated prefill work.
Where does the cache stop?
Request B changes one character inside the tool schema, then returns to content identical to Request A.