Prompt caching:reuse the matchinginput prefix.
Prompt caching:reuse the repeatedbeginning.
When eligible requests share an exact input prefix, the service can reuse intermediate work for that prefix. The uncached suffix is processed normally, and the output is generated fresh.When several AI requests begin with the same long instructions, the service may reuse the work from that shared beginning. It still writes a new answer each time.
- AUTOAUTOMATICon eligible requestswhen the service allows it
- EXACTSAME STARTprefix matches onlythe beginning must match
- 1,024+LONG INPUTStokens on OpenAIminimums vary by provider
4,352 input tokens reused cached work
09:41:02 prefix hash matched
09:41:02 attention state restored
09:41:02 only fresh tail processed →
Prompt caching reuses input processing, not generated output.The repeated reading can be reused. The answer cannot.
For an eligible exact prefix, the service restores reusable intermediate state and processes only the uncached suffix before generation.The service can skip repeating work for the identical beginning, then read the new ending and write a fresh response.
Put long shared instructions first. Put the request-specific material at the end.
Put stable content before changing content.Put shared reading before today’s question.
Choose a workload, then move its changing block to the front. Because matching is prefix-based, that one ordering change removes the reusable prefix.Choose an example, then move today’s changing material to the front. The service must reread everything after the first difference.
Stable rules, tools, and repository contextThe shared instructions and project background stay the same
Reuse stops at the first difference.
Cache matching moves from the start of the request toward the end. It can reuse the exact prefix before the first difference, but it cannot skip a change and resume later.
Most of the prefix remains reusable because the first difference is near the end.
Repeated prefixes make the savings accumulate.
The first request processes the full input. Later matching requests can receive the provider's cached-input rate for the shared prefix.
Fixed example: a 6,000-token shared beginning. Cached reads cost 25% of a normal read. Every request also has a fresh 400-token tail.
in this simplified 12-request example
Excludes output tokens, cache-write premiums, minimum cacheable lengths, and provider-specific pricing. This is intuition math, not a provider quote.
Look for long, repeated beginnings.
The ideal workload repeats a long, identical setup and changes only a small tail. In AI engineering, that pattern is everywhere.
Coding agents
Reuse the system prompt, tool definitions, repo map, and coding conventions. Append today’s task at the end.
static setup → “fix flaky test”Support copilots
Keep policies, product facts, tone rules, and tool schemas stable. Add the customer conversation last.
stable playbook → customer issueBatch extraction
Reuse a large extraction schema and worked examples. Put each new document after that fixed instruction set.
schema + examples → new document- Long system prompts
- Stable tool definitions
- Repeated few-shot examples
- Shared reference documents
- Multi-turn conversations with a stable beginning
- Short one-off prompts
- Everything changes every request
- Personalization injected at the top
- Randomly reordered tools or examples
- Traffic too sparse for the cache to stay warm
The anatomy of a cache hit.What happens when the beginning matches.
- 1ROUTELABEL
Fingerprint the beginningGive the beginning a matching label
The service hashes an initial prompt prefix and uses it to route similar requests toward the same cache neighborhood.The service turns the start of the request into a short label. Requests with the same label can be sent toward the same reusable work.
- 2LOOK UP
Find the longest exact prefixFind the longest identical beginning
The cache checks for matching prompt state. If several prefixes match, the longest useful one wins.The service compares the request from left to right. If several earlier beginnings match, it reuses the longest one.
- 3RESUMECONTINUE
Restore state, process the tailReuse the shared reading, then read the new ending
Cached attention state is loaded. The model processes only the uncached suffix, then generates a brand-new response.The service reuses the work from the shared beginning, reads the new ending, and then writes a completely fresh answer.
Five rules for cache-friendly prompts.
- 01
Put reusable content first.
System instructions, tool schemas, examples, and shared context lead the prompt.
- 02
Put user-specific content last.
Names, timestamps, live data, current tasks, and conversations belong at the tail.
- 03
Keep serialization deterministic.
Keep tool order, object keys, examples, and formatting stable. The serialized prefix must match exactly.
- 04
Group related traffic when your API supports it.
A stable prompt cache key can improve routing for requests that share the same long prefix.
- 05
Measure actual hits.
Log cached-token counts and hit rates. Prompt structure only matters if requests actually reuse it.
What it looks like in an API response.
You usually don’t “fetch” a prompt cache yourself. Send a stable prefix, then inspect usage metadata to see what was reused.
const response = await client.responses.create({
model: "your-cache-capable-model",
// Keep this stable for requests with the same prefix.
prompt_cache_key: "coding-agent:repo-42:v3",
input: [
{ role: "system", content: SYSTEM_PROMPT },
{ role: "developer", content: TOOL_SCHEMAS },
{ role: "developer", content: REPO_CONTEXT },
// The changing part goes last.
{ role: "user", content: currentTask }
]
});{
"usage": {
"input_tokens": 4680,
"input_tokens_details": {
"cached_tokens": 4352
},
"output_tokens": 612,
"total_tokens": 5292
}
}The usage data confirms the cache hit. The response was still generated fresh; 4,352 input tokens came from the prompt cache.
Three common misconceptions.
“It caches the answer.”
Prompt caching reuses work done while processing the input. Output generation still runs again.
ONLY
“Semantically similar is close enough.”
Similar meaning is not enough. Reuse applies only to the exact matching prefix.
“More cached tokens means lower quality.”
The computation is reused, not approximated. Same prefix, same state—just less repeated prefill work.
Where does the cache stop?
Request B changes one character inside the tool schema, then returns to content identical to Request A.