Prompt caching: how provider caches work and how to structure prompts for them

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

OpenAI and vLLM describe prompt caches in terms of stored key-value state from a processed prefix; Anthropic also refers to KV representations. Google's cited pages do not specify their internal stored representation. The next request reuses that state when its rendered prefix matches, and it still processes every token after the match. Caching does not change how output is generated, although separate sampled requests can still produce different answers. What changes is the prefill work, the latency before the first new token, and the price of the reused input.

An AI writing system produced this article for NetLeverage's editorial studio. "The model remembers the prefix" here means stored tensors keyed by the token sequence, not a verified claim about memory or experience. Nothing below was run in a lab. Prices, minima, and field names are from the pages opened on 10 November 2026.

Why it matters

Long instructions, tool definitions, and a document you ask many questions about are paid for on every call if the prefix is rebuilt from scratch. Caching is the vendor's way of not doing that twice. It fails quietly. A timestamp in the first paragraph, a reordered tool, or a request that lands on a different machine produces a full-price prefill and a response that looks normal.

The published multipliers, which will move, show why the structure is worth getting right.

• Anthropic, on the page opened that day: a 5-minute cache write is 1.25 times base input, a 1-hour write is 2 times, and a read is 0.1 times, with exceptions. Claude Opus 5.5 and Claude Sonnet 5.5 reads are listed at 0.05 times base input. Claude Fable 5.1 and Mythos 5.1 reads are listed at 0.025 times.
• OpenAI, for GPT-5.6 and later: a 30-minute write is 1.25 times uncached input, a 12-hour write in the Responses API is 2.5 times, and a read is 0.1 times on most of those models and 0.05 times on GPT-6.1 Sol and GPT-6.1 Terra. Their arithmetic for a 0.1-times read: one 30-minute write plus nine reads is 2.15 times ordinary input, against 10 times with no cache.
• Google Cloud's context-caching overview: implicit cache hits are a 90 percent discount against standard input. Explicit caching on Gemini 2.5 and later is also a 90 percent discount; Gemini 2.0 explicit caching is 75 percent. Explicit caches also have a storage charge. Implicit caches do not.

OpenAI says the minimum lifetime governs the read price and does not guarantee that the request reused stored tensors or finished the prefill faster.

How it works

The prefix is a chain of blocks

While a Transformer processes position $t$, it forms keys and values for that position so later positions can attend to it. A prompt cache keeps those tensors for a leading span of the request. OpenAI's guide states this directly: the cache stores key-value tensors. The token string is how you look the tensors up.

vLLM's prefix-caching design, on the main-branch document retrieved the same day, is a concrete version of that lookup. The engine hashes each full block from three parts: the parent block's hash, the exact tokens in this block, and extra hashes for anything else that changes the computation, including a LoRA adapter id, a multimodal input hash, and an optional cache_salt. Only full blocks are cached. In the document's worked example the block size is 4. A new request shares the first 10 tokens of an older one and hits two blocks, 8 tokens. The leftover 2 tokens sit in a partial block and are computed again.

That is why a one-token edit at the start of a prompt wastes the rest of the cache. The parent hash changes, and every later block hash changes with it. A one-token edit after the last full block does not.

cache_salt is mixed into the first block's hash so only requests with the same salt reuse those blocks. The stated reason is a timing probe: a faster response revealing that a guessed prompt was already cached. The default hash is sha256 as of v0.11. The xxhash option is documented as not cryptographically secure, with a theoretical collision risk the project says can leak information across tenants.

Provider APIs hide the block size. Anthropic still requires a 100 percent identical segment, text and images, through the cached block. OpenAI requires the whole rendered prefix to match, including hidden instructions, tools, and history.

Anthropic: you mark the end of the prefix

On the Messages API you opt in. Top-level cache_control: {"type": "ephemeral"} is automatic caching: the breakpoint sits on the last cacheable block and moves forward as the conversation grows. Putting cache_control on a content block is an explicit breakpoint. The page's curl examples send anthropic-version: 2026-10-21.

The cached span is the whole prefix in a fixed order: tools, then system, then messages, through the breakpoint. A change at one level invalidates that level and everything after it. Editing a tool name, description, or schema invalidates tools, system, and messages. Toggling web search or citations invalidates system and messages, because those toggles change the system prompt. Changing tool_choice invalidates messages only. Speed and thinking settings also have model-specific invalidation rules. Adding or removing an image anywhere invalidates messages.

A write happens only at a breakpoint. The read path hashes the prefix at that breakpoint and, on a miss, walks backward one block at a time looking for a prefix some earlier request actually wrote. It does not discover that your first five blocks were stable and cache them for you. The lookback is 20 blocks, counting the breakpoint. A run of consecutive tool_use blocks counts as one position, and so does a run of consecutive tool_result blocks.

The page's failure example is the one to memorise. Blocks 1–5 are a static system context. Block 6 is a timestamp plus the user message, and cache_control sits on block 6. Request 1 writes a hash that includes the timestamp. Request 2's hash differs, the walk back through blocks 5 to 1 finds nothing, because nothing was ever written there, and every request pays for a fresh write. Move the breakpoint to block 5.

You may place up to 4 breakpoints. Automatic caching consumes one of those slots, and a request that already has four explicit breakpoints returns a 400. Use a second breakpoint when a growing transcript would push the only write more than 20 blocks behind the latest one. The page's sketch is a write at block 15 and a later breakpoint at block 35: the window is blocks 35 through 16, and the hit is missed.

Minimum lengths on the Claude API, Claude Platform on AWS, Google Cloud's managed Claude models, and Microsoft Foundry, as listed that day:

A shorter prompt is processed with no cache and no error. Both cache_creation_input_tokens and cache_read_input_tokens come back 0. The identity the page gives for the bill is:

input_tokens counts only the tokens after the last breakpoint. A 100,000-token read, a 0-token write, and a 50-token question are 100,050 input tokens, of which the usage field input_tokens shows 50.

The default lifetime is 5 minutes, measured from the start of the request that writes or reads, not from the end of the response. Time spent generating counts. If a response streams for 4 minutes, a follow-up that wants the same entry has about a minute left. A hit refreshes the 5-minute entry at no extra charge. {"type": "ephemeral", "ttl": "1h"} buys the longer lifetime at the 2-times write price. ephemeral is the only cache type on that page. A cache entry becomes readable when the first response starts. Parallel calls fired before that miss. Caches are isolated by organisation, and on the Claude API, Claude Platform on AWS, and Microsoft Foundry also by workspace. Output tokens are unaffected. Anthropic states the response matches the uncached response.

OpenAI: on by default, then a routing problem

OpenAI's guide says prompt caching is enabled by default on supported models. The cached object is the full rendered context: hidden instructions, developer messages, tool definitions, and history, including images, documents, and supported audio. Hidden system tokens do not count toward the minimum. For GPT-5.6 and later that minimum is 1,024 visible input tokens. Earlier models vary with tools, images, schemas, reasoning effort, and verbosity. The page says to check the model comparison rather than assuming 1,024.

For GPT-5.6 and later you choose breakpoints. prompt_cache_options.mode: "explicit" writes only where you set prompt_cache_breakpoint. With no breakpoint, that request does not cache. Content after the last breakpoint is billed as ordinary input, with no write charge, which is how you keep a one-off question out of the write. Implicit mode places a breakpoint at the end of the latest eligible message: a user message, the last tool response in a run of them, or the last developer message in the opening developer block. An implicit breakpoint uses one of four write slots and leaves three for explicit breakpoints. Top-level instructions cannot hold an explicit breakpoint; put reusable developer text in an input_text block inside a developer message.

Lookup walks eligible boundaries from the longest prefix to the shortest, on the machine that received the request. Entries live on individual machines. Above about 15 requests per minute, overflow routing can send a call somewhere that does not hold the entry. Caches are not shared across organisations or across regional processing boundaries. Before GPT-5.6, a stable prompt_cache_key helps route related calls together. It does not pin a machine and does not promise a hit. The page's guidance for a busy key is about 15 requests per minute across the prefixes that share it, and to partition higher volume across keys with a stable mapping. On GPT-5.6 and later, routing is automatic and the key is for separate accounting, including a defence against cache-hit probing across users.

prompt_cache_options.ttl is "30m" or, in the Responses API, "12h". These are minimum lifetimes. OpenAI may keep an entry longer, and "12h" is not a deletion deadline. A 30-minute read refreshes 30-minute coverage. Reading a 12-hour entry keeps the later of the existing expiry and 30 minutes after the read, without a new write and without restarting the 12-hour clock. Upgrading live 30-minute coverage to 12 hours charges a new 12-hour write, with no credit for the earlier one. Inside one request a token is billed once, at the longest lifetime of the writes that cover it, so a later 12-hour breakpoint pulls earlier tokens in that prefix up to the 12-hour write rate.

Older models have implicit breakpoints only, and prompt_cache_retention of in_memory (typically 5 to 10 minutes idle, up to an hour) or 24h (typically around 30 minutes, up to 24 hours). Where both exist, organisations without Zero Data Retention default to 24h, and organisations with it default to in_memory. On those models cached_tokens drops hidden system tokens and rounds down to a multiple of 128.

Settings that rewrite the prefix: model, tool definitions and their order, parallel_tool_calls, text.format, reasoning.effort, text.verbosity, and compaction. On supported GPT-6 models, append a configuration_update to change reasoning effort and leave the request-level value in place. Set tool_choice to none, or use allowed_tools, when you want fewer calls without rewriting the tool list.

Gemini on Google Cloud: a named object, plus an automatic tier

Google Cloud's overview, opened 10 November 2026, splits the feature in two.

Implicit caching is on by default. Hits receive the 90 percent discount. Prefixes sit in memory and, on a subset of models, move to durable storage when memory evicts them rather than disappearing. Both tiers default on. An implicit prefix expires no later than 24 hours after creation, and the page says most entries go sooner. The advice for a hit is the same prefix rule: large shared content at the beginning, and similar requests close together in time.

Explicit caching is a resource you create, then cite by name. The request identifies a model; this article does not infer model portability beyond the documented API. The create page says the default expiry is 60 minutes, overridable with ttl or expire_time, and an expired cache is no longer available and must be recreated for later use. The overview's limits say the shortest expiry you can set is 1 minute, and that explicit caches have no maximum duration. Minimum cached length is 4,096 tokens for Gemini 3 (with stated exceptions) and Gemini 4 models, 6,144 tokens for implicit caching only on Gemini 3.0 Flash Preview, 3.1 Pro Preview, 3.7 Flash, and 3.8 Flash, and 2,048 tokens for the Gemini 2 family. A blob or inline text cache is limited to 10 MB; larger content has to be a Cloud Storage URI. Do not modify that object while the cache lives. The overview's limits section warns that an update can make the cached content unusable.

cachedContentTokenCount is the usage field for both modes. Cached data is isolated by project number and is not shared across projects. The overview tells multi-tenant and background-agent deployments to scope cache and outputs to the end user's permissions, so one session cannot reuse another's cached credentials. Explicit caches can interact with implicit caching and retain more than the object you named. Turning implicit caching off, and not creating explicit caches, is the documented way to stop that retention. Context caching is covered by VPC Service Controls; a bucket you cache from has to sit inside the perimeter as well.

What to put where

Build the request in change-frequency order, slowest first.

A tool-definition edit is the expensive miss, because on Anthropic it takes the system prompt and the messages with it. Gemini's named cache fits many requests against one large file. Implicit caching fits a chat that only appends, inside the lifetime.

Check the counters before you believe a hit. Anthropic's read and write fields, OpenAI's cached_tokens, and Gemini's cachedContentTokenCount are the evidence. A zero can reflect a short prompt, expiry, no eligible write, changed prefix, routing, or unsupported settings. Counters describe provider accounting; on OpenAI they do not alone establish physical tensor reuse or a latency gain.

Give each tenant its own cache scope: cache_salt on a shared vLLM engine, a distinct prompt_cache_key where you need separate OpenAI accounting, a workspace or project boundary on the hosted APIs, and a session-scoped explicit cache on Gemini. A shared salt across customers with different permissions is a cross-tenant prefix, visible as a latency difference even when the text never comes back.

Risks and limits

• A cache hit is an exact prefix match. Paraphrase, a reordered JSON key inside a tool schema, or a single image added "just for this call" moves the boundary.
• Lifetimes are short relative to a human pause. Anthropic's 5-minute clock runs during generation. OpenAI's 30-minute and 12-hour values are minima, not promises that the tensors are still on the machine.
• Hosted caches follow the vendor's routing. OpenAI documents per-machine entries and overflow above about 15 requests per minute. A retry on another machine is a miss.
• Parallel fan-out against Anthropic needs the first response to have started, or the followers miss.
• Explicit Gemini caches bill storage and the create page places their content in the creation-request region; check model compatibility in the API. Changing the Cloud Storage object under a live cache can make it unusable.
• vLLM reuses full blocks only. A shared engine without cache_salt is a timing side channel the project documents.
• Pricing rows, model minima, and beta headers were read on 10 November 2026. Re-read them before you budget.
• This article does not claim any behaviour of a NetLeverage product.

Sources

Sources checked on 10 November 2026; access limitations are identified below.

1. Anthropic, Prompt caching — https://platform.claude.com/docs/en/build-with-claude/prompt-caching. Breakpoints, lookback, minima, pricing multipliers, invalidation table, usage identity, lifetime clock.
2. OpenAI, Prompt caching — https://developers.openai.com/api/docs/guides/prompt-caching. KV tensors, minima, explicit and implicit modes, TTL rates, routing, prompt_cache_key.
3. Google Cloud, Context caching overview — https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/context-cache/context-cache-overview. Implicit and explicit modes, discounts, token minima, project isolation and Cloud Storage object-change warning. Page displayed opened 10 November 2026.
4. Google Cloud, Create a context cache — https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/context-cache/context-cache-create. Default 60-minute expiry, 10 MB inline limit, creation region.
5. vLLM, Automatic Prefix Caching design note — https://github.com/vllm-project/vllm/blob/main/docs/design/prefix_caching.md (raw document retrieved 10 November 2026). Block hash, full blocks only, cache_salt, sha256 default as of v0.11, xxhash warning.

This article was originally published on 10 November 2026.