Prompt caching is the single most underused optimization in the AI stack.
The economics
A cached token costs roughly 10% of a fresh input token and lands in a fraction of the time. If your app has any kind of shared context — a system prompt, a retrieved document, a tool spec — you're paying for that content on every request unless you cache it.
Where caching wins big
- Long system prompts (>1k tokens). Basically every serious assistant has one of these.
- Shared few-shot examples. Stable for the life of the deployment.
- Long tool / schema definitions. Also stable.
- Retrieved documents reused across turns. Variable by session, but cache TTLs usually cover the common case.
Where it doesn't
- User-specific content. Don't try.
- Anything that changes per request.
- Low-volume endpoints where the cache never warms.
The numbers
A customer-support bot I worked on dropped input costs 72% by caching just three things: the system prompt, the tool schema, and the top-three retrieved KB articles (which had a hit rate of 60% across sessions).
The changes took a day to implement. The savings run indefinitely.
If you only have time for one cost optimization this quarter, this is it.