Prompt caching can miss because the reusable prefix changes, the cache boundary does not match an earlier write, the prefix is below the model’s minimum length, or the next request arrives after expiry. Start with the provider’s cache usage fields, then compare the actual requests sent by your application. Similar-looking prompts alone do not prove reuse.
For recurring internal reports, document processing or agent sessions, this distinction changes the cost forecast. A discounted cache-read price applies to tokens actually read from cache. Repeatedly writing a new prefix can leave the expected saving absent from the bill.
Establish whether the request wrote or read a cache
A cold request and a failed cache configuration can both show zero read tokens. Inspect the write evidence before changing the prompt.
OpenAI’s prompt-caching documentation describes usage.input_tokens_details.cached_tokens for Responses API reads. Its current guidance also documents cache-write fields for models supporting the newer cache options. Check the selected model’s accounting and configuration rather than transplanting an example from another generation.
For the Claude Messages API, Anthropic documents separate cache_creation_input_tokens and cache_read_input_tokens. Its input_tokens field counts input that was neither read from nor used to create a cache. Treating that field as the entire input would undercount usage.
Save the exact model, API route, cache mode and relevant configuration beside those counters, with the request start time and application release. That record lets the workflow owner distinguish a prompt change from a model migration before spending engineering time on the wrong repair.
Prompt caching reuses processing of a prompt prefix. The model still generates an answer, and the application remains responsible for checking evidence and authorising any action.
Find the earliest change before moving the cache boundary
Consider a proposed reporting workflow whose application assembles this request:
Current timestamp
Reporting instructions
Tool definitions and reference material
This account's latest records
Question to answer
The timestamp changes before the instructions the team hoped to reuse. Moving that timestamp after a genuinely stable instruction block gives the application a reusable opening. The precise API arrangement still matters: tools and system messages can have their own ordering rules.
Compare the provider-facing request, including content blocks and tool definitions. A prompt template in source control will not reveal context inserted later by middleware. Anthropic explicitly documents tool-definition changes as invalidating the tools cache and subsequent layers; its troubleshooting guidance also calls out unstable key ordering inside tool-use content blocks.
Keep a versioned, deterministic representation of the reusable content. Put case-specific evidence after it when the task permits. Never preserve an obsolete price, permission decision or customer record solely to raise the hit rate. Refreshing required evidence takes priority over reusing its earlier processing.
Verify that the reusable endpoint was actually cached
The opening can be identical and the cache still miss because the requests use different lookup boundaries.
Anthropic explains that explicit cache writes occur at the marked breakpoint. Its backward lookup looks for entries earlier requests actually wrote. A breakpoint after changing per-request content can therefore keep creating new entries without making the desired stable section reusable.
OpenAI’s current guide describes a related gotcha for GPT-5.6 and later: an implicit write through dynamic content does not automatically create a separately reusable shorter prefix. The documented remedy is an explicit breakpoint after the static content in both requests. Apply that remedy only to models and API configurations supporting it; older caching assumptions do not transfer automatically.
Draw the intended boundary on both consecutive requests. Confirm that the first writes the reusable endpoint and the next is configured to look it up. This exposes a different fault from a changed timestamp: the content can be stable while the application never establishes the cache entry it expects.
Replay one change at a time
Use non-sensitive fixtures in a controlled API environment. The following is a proposed diagnostic protocol, not a measured benchmark or a guarantee that every eligible repeat will hit.
| Replay | Keep fixed or change | What the result helps isolate |
|---|---|---|
| Cold request, then an identical sequential repeat | Hold model, content, configuration and routing context fixed; send the repeat within the documented lifetime | Whether the basic setup produces a write followed by a read |
| Different case, stable instructions | Change only content after the intended reusable boundary | Whether the application can reuse the prefix across actual work items |
| Deliberate early-prefix change | Alter a harmless fixture value before that boundary | Whether the observed reuse depends on the prefix the team believes it is caching |
| Repeat after the documented lifetime | Preserve content but change the interval | Whether the real workload is too infrequent for the selected retention policy |
Before interpreting a miss, check the minimum cacheable length for that model and platform. Anthropic documents that a prompt below its minimum can be processed without caching and without an error. OpenAI also warns that changing models can change the threshold. A successful HTTP response establishes neither cache eligibility nor a cache hit.
Do not begin with concurrent identical requests when diagnosing basic reuse. First establish a sequential write/read pair, so the follow-up can encounter an entry created by the earlier request. Then test the concurrency and timing of the production workload separately.
Reconcile write lifetimes before blaming the hit rate
A cache can work correctly while costing more than the forecast. For GPT-5.6 and later, OpenAI’s current guide prices a default 30-minute write at 1.25 times the ordinary input-token rate. In the Responses API, choosing a 12-hour minimum lifetime raises that write rate to 2.5 times ordinary input. Those are total write rates, not additional fees stacked on top of ordinary input.
Choose the longer lifetime only when the gaps between real requests make it worth paying more upfront to avoid later writes. If the prefix changes before the next request, the extra coverage has bought no reuse.
For Responses API models supporting prompt_cache_options, reconcile these categories for each request:
| Usage category | Evidence to retain | Pricing treatment |
|---|---|---|
| Ordinary input | input_tokens less cached_tokens and cache_write_tokens | Apply the model’s ordinary input rate |
| Cache reads | input_tokens_details.cached_tokens | Use that model’s cached-input rate |
| 30-minute writes | input_tokens_details.cache_write_tokens_details.ttl_30m_tokens | Apply 1.25 times the ordinary input rate |
| 12-hour writes | input_tokens_details.cache_write_tokens_details.ttl_12h_tokens | Apply 2.5 times the ordinary input rate |
The two lifetime-specific write counts do not overlap and sum to cache_write_tokens. Once they have been priced separately, adding the total write count again double-counts the writes. Output generation and paid tools remain separate charges.
Check configuration changes as well as prompt changes. Upgrading valid 30-minute coverage to 12 hours incurs a new 12-hour write for those tokens, without credit for the earlier write. That request can therefore show a write charge despite the prefix already having been cached. Any portion already covered by a valid 12-hour entry remains a read.
Keep the selected lifetime alongside the request pair in the replay protocol. Compare billed input cost for the real spacing between requests before retaining a longer lifetime. OpenAI distinguishes eligibility for cache-read pricing from guaranteed lower latency; a 12-hour minimum also does not establish a deletion deadline. A billing repair and a data-retention decision require separate evidence.
Judge the repair by workflow cost
A higher hit rate can still accompany higher spend if the application starts sending much more context. Adding irrelevant text to cross a threshold also changes the request being priced.
Compare the same representative work before and after the change. Keep input volume, cache reads and writes visible alongside output usage and any paid tools. Measure elapsed time and check whether the result still meets the workflow’s acceptance criteria. Report an untested forecast separately from an observed bill.
The AI agent ROI guide covers the wider cost of accepted outcomes, including review and recovery. Caching addresses one component of that denominator. It does not remove integration work or increase a reviewer’s available time. If the remaining delay comes from throttling or retries, use the integration bottleneck guide to inspect that separate path.
Model Operator is a Dubai-based AI product studio led by Alexander Chrisostomou. Its AI Initiative Consulting engagement helps teams decide which workflow constraint deserves investment. Bring the owner of your recurring API workload and a redacted request pair with usage records to a Model Operator build conversation to identify the next repair and the evidence needed to justify it.