Prompt Caching Explained: Slash Your LLM API Bills

The Verdict: Is Prompt Caching Worth It?

TL;DR: Prompt caching stores the computed state of the stable start of your prompt. You pay a write fee to store it once, then a fraction of the normal input price on each hit. Anthropic bills a 5-minute cache write at 1.25 times the base input price and reads at 0.1 times the base price, with exceptions for some models. OpenAI discounts cached input by up to 95 percent and keeps caching on by default. Google caches implicitly on Gemini 2.5 and newer and applies the savings without publishing a rate. If each request resends 1,000 tokens or more of stable text (system prompt, tool definitions, reference docs), caching is the cheapest lever on your API bill. If your prompts are short or change on each call, skip it.

Caching fits chatbots with long system prompts, agents that resend tool schemas each turn, and RAG pipelines that attach the same document set to many queries. I treat it as a cost lever first and a latency lever second: the bill shrinks faster than the response time does.

How Prefix Caching Works

An LLM request has two phases. In the prefill phase, the model reads your full prompt and builds a key-value cache of its attention state. In the decode phase, it generates tokens one by one while reusing that cache. Prefill is expensive; decode is incremental. Prefix caching removes the prefill cost for tokens the provider has already processed.

It works like memoization for prompts. Requests that share the same opening tokens hit a stored entry: the provider keys the computed state to that prefix and skips prefill for the shared portion. Both the bill and the latency drop, because the provider reuses work it did once.

Also Read:

Vector Databases: pgvector vs Pinecone vs Qdrant

A hit needs a byte-identical prefix: the same model, the same settings, the same tool definitions, in the same order. Change one character near the start and the match fails; the provider recomputes from that point onward. The provider compares bytes. Intent does not enter the match.

Provider Showdown: Anthropic, OpenAI, and Google

The three big providers take different routes to the same idea. Anthropic gives you explicit control through breakpoints you place. OpenAI and Google flip caching on by default and keep the details under the hood. The table below compares what each provider publishes in its official docs, as verified the same day this article was written.

Also Read:

LLM Structured Outputs: OpenAI vs Anthropic vs Google

A note on the numbers. OpenAI says “up to 95 percent”, which is a ceiling, not a flat rate. Anthropic says reads cost 0.1 times the base input price and adds that per-model exceptions exist; the docs page does not enumerate each exception. Google publishes no percentage at all. A blog that quotes a flat 95 percent or a flat 75 percent for Gemini without qualification trades in stale marketing, not documentation.

The same three providers appear in the structured-outputs comparison across the same three providers: if you pin response schemas, that comparison pairs well with this one.

Minimums, TTLs, and Breakpoints: The Gotchas That Kill Hit Rates

Caching has entry requirements, and misses trace back to one of them.

Also Read:

Foreman Review: Reliability Gates for Scheduled AI Agents

Minimum prefix length. Anthropic varies by model: Claude Sonnet 5.5 and Claude Opus 5.5 start at 512 tokens; Claude Opus 4.8, Claude Sonnet 5, Claude Sonnet 4.6, and Claude Sonnet 4.5 start at 1,024; Claude Haiku 4.5 needs 4,096. OpenAI sets 1,024 tokens for GPT-5.6 and later. Google sets 2,048 for Gemini 2.5 Flash and Gemini 2.5 Pro, and 4,096 for Gemini 3.1 Pro Preview, 3.6 and 3.8 Flash, and Gemini 4 Argon. I would not bother caching a prompt under 1,000 tokens. The write fee outweighs the discounted reads.

Cache lifetime. Anthropic keeps entries for a minimum of 5 minutes with the standard setting or 1 hour with the extended setting. OpenAI keeps them for 30 minutes or more after the last write or reuse. Sporadic traffic still hits: a request every few minutes refreshes the entry, while a request every few hours starts cold. Match the TTL to your traffic pattern before projecting savings.

Breakpoint placement. Anthropic supports explicit breakpoints at the end of stable sections and checks at most 20 positions per breakpoint (the lookback window). Place breakpoints at natural seams: the end of the system prompt and the end of the tool definitions. Each breakpoint raises the odds that at least the earlier portions hit.

Also Read:

RAG vs Fine-Tuning: When to Use Which

Prefix purity. Timestamps, request IDs, and per-user greetings at the start of the prompt cause permanent misses. Move each volatile token to the end. One dynamic header can invalidate an otherwise identical 8,000-token prefix.

Write cost. Anthropic charges more to write the cache than to run the prompt once: 1.25 times the base input price for the 5-minute setting, 2 times for the 1-hour setting. Reserve caching for prefixes you resend constantly. Occasional reuse rarely clears the write fee. The math turns positive around five to ten reuses, depending on the multiplier and your hit rate.

Designing Prompts for Cache Hits

The pattern is simple: stable material first, volatile material last. Order the request as system prompt, then tool definitions, then reference documents, then the user input. Keep that order on each request. Rearranging sections breaks the byte match even when the content is identical.

On Anthropic, add explicit breakpoints at the end of each stable section. Four breakpoints is the practical ceiling, so spend them on the longest stable chunks. On OpenAI, use prompt_cache_key when you route similar traffic through different keys; identical prefixes routed to the same key share the warm cache. Send a warm-up request on deploy so the first real user does not pay for the first write.

Keep tool schemas stable. Agents resend tool definitions on each turn, and a schema edit (a renamed parameter, a reworded description) invalidates the cached prefix from that point on. Context engineering for agents covers the other half of this problem: caching fixes the billing side, while a disciplined context budget fixes the quality side.

Place retrieved documents after the stable prefix if you run RAG. The system prompt and tool definitions stay cacheable while the retrieved chunks vary per query. Your vector database results sit in that volatile tail, and caching is worth trying before reaching for fine-tuning, since it changes nothing about model weights.

Measuring the Win: Usage Fields and Dashboards

Each provider exposes cache activity in its usage response. Anthropic reports cache_read_input_tokens and cache_creation_input_tokens. Google reports usage.total_cached_tokens. OpenAI ships a Prompt Caching dashboard plus cache diagnostics in the API. I check these fields before changing anything else: they report the hit rate, which is the number that decides whether caching helps.

The savings formula fits on one line. Take the tokens in your stable prefix. Multiply by the number of requests in a billing period and the base input price: that is the cost without caching. Then compute the cost with caching: one write at the write multiplier, plus hits at the read multiplier. The gap between the two lines is your savings. If the hit rate sits under 50 percent, fix the prompt structure before sending more traffic.

Caching loses to simpler moves in two cases. Sub-threshold prompts never write an entry, so they never save. Low-traffic endpoints expire between requests, so each call pays the write fee for no benefit. Raise the prefix length or raise the traffic first, then revisit.

Pair caching with a smaller model when the task allows it. Caching cuts the price of the tokens you resend; a smaller model cuts the price of every token. Coding assistants that resend context on each turn benefit from the same stack.

The Bottom Line

Prompt caching is the rare optimization with no quality tradeoff: the model returns the same output, you pay less, and responses start faster. The requirements are mechanical: a long stable prefix, and clean ordering with the volatile tokens last. Traffic needs to arrive before the TTL expires. Get those conditions right and the discounts apply themselves.

Do one thing this week. Pick the endpoint with the highest token volume, read its usage fields, and compute the hit rate. Then reorder the prompt so the stable sections come first, add breakpoints where the provider supports them, and compare the usage fields again. Many teams find one endpoint that pays for the experiment on the first day.

Start with the official docs: Anthropic’s prompt caching guide, OpenAI’s prompt caching guide, and Google’s context caching guide. Check Anthropic’s pricing and OpenAI’s pricing for the current per-token rates before you project savings. Model catalogs shift; treat the docs pages as the source of truth.