Prompt Caches Need Ownership Boundaries

Prompt caching saves real money only after the team decides who owns the cached context, when it expires, and what must never cross the boundary. Treat the cache as part of the agent runtime contract, not as a hidden discount switch on a long prompt.

Cache hit rate is the wrong first metric

The first prompt caching dashboard most teams want is simple: input tokens, cached tokens, saved dollars, and latency. That is useful, but it is not the first production question. The first question is whether the system reused the right context for the right request.

OpenAI describes prompt caching as a way to reduce latency and cost by reusing prompt prefixes the model has recently seen (OpenAI prompt caching). Anthropic exposes both automatic caching and explicit cache breakpoints, including cache_control on content blocks for finer control (Anthropic prompt caching). Google Vertex AI presents context caching as a managed resource that can be created, used, inspected, updated, and deleted (Vertex AI context caching). Amazon Bedrock describes prompt caching around checkpoints and separate accounting for cache writes and cache reads (Amazon Bedrock prompt caching).

Those product details differ, but the architecture problem is the same. A production prompt is not one blob of text. It is a bundle of authority: system instructions, developer policy, tool schemas, retrieval results, user memory, tenant permissions, safety rules, and sometimes expensive examples. Reusing that bundle without boundaries can turn a cost optimization into a correctness bug.

The bad failure mode is not a lower cache hit rate. The bad failure mode is a high hit rate on stale policy, the wrong tenant scope, an old tool schema, or a prompt prefix that no longer matches the release running behind it.

The cache boundary should have an owner

A cache boundary is the line between context that can be reused and context that must be rebuilt for each request. In a normal web service, teams already know how to reason about this. You do not cache a personalized account page under a shared key. You do not reuse an authorization result after the permission model changed. You do not keep a payment quote forever because it was expensive to compute.

LLM applications deserve the same discipline. The cached prefix is not just text. It is executable influence over the model.

A practical ownership model looks like this:

Context part	Usual owner	Cache posture
Product behavior instructions	Application team	Cache with release version
Tool schemas and tool policy	Platform or agent runtime team	Cache with schema hash
Examples and style guidance	Product or applied AI team	Cache with prompt version
Retrieval results	Search or knowledge team	Usually live, sometimes cache by corpus snapshot
User memory and tenant policy	Identity, privacy, or app team	Live by default
Incident overrides and safety blocks	Operations or safety team	Live by default

This table is intentionally organizational, not just technical. If nobody owns a prompt segment, nobody owns invalidation. If nobody owns invalidation, the cache is gambling with production behavior.

Separate stable context from live context

The simplest pattern is to split the model call into stable context and live context. Stable context changes on deploys, schema updates, evaluation updates, or style changes. Live context changes on every user, tenant, retrieval query, permission check, current time, or operational override.

flowchart LR
    A["Stable context"] --> K["Cache key"]
    B["Tool schemas"] --> K
    C["Prompt version"] --> K
    K --> P["Cached prefix"]
    U["User request"] --> L["Live context"]
    R["Retrieval result"] --> L
    M["User memory"] --> L
    P --> X["Model call"]
    L --> X
    X --> T["Trace"]
    K --> T
    I["Invalidation event"] --> K

This split keeps the optimization where it belongs. Cache the stable prefix if the provider supports it. Rebuild the live suffix for the current request. The model still sees a coherent prompt, but the runtime can reason about which parts were reused and which parts were fresh.

The retrieval boundary is especially important. OpenAI File Search is documented as a tool that searches a knowledge base of uploaded files (OpenAI File Search). That is the right mental model. Retrieval is a runtime decision, not static prompt decoration. If a support policy, customer record, code document, or incident note is part of the answer, the system should know whether that evidence came from the current request path or from a cached prefix.

There are exceptions. A static API manual snapshot, pinned to a corpus version, might be cacheable. A tenant-specific policy answer, filtered through current permissions, usually is not. The difference should be expressed in the cache contract, not buried in a prompt builder helper.

Invalidation is part of the API

Most prompt caching bugs come from lazy invalidation. The prompt string changes, or it does not. The provider cache hits, or it does not. The application team shrugs because caching is treated as an implementation detail.

A better contract makes invalidation explicit:

{
  "cache_contract": "agent_context.v1",
  "owner": "agent-runtime",
  "stable_prefix": {
    "prompt_version": "support-agent.42",
    "tool_schema_hash": "sha256:7f8c...",
    "policy_bundle": "safety-2026-07-25",
    "examples_version": "refunds.v6"
  },
  "live_suffix": {
    "tenant_id": "tenant_123",
    "principal_id": "user_456",
    "retrieval_snapshot": "request-scoped",
    "memory_snapshot": "request-scoped"
  },
  "invalidate_when": [
    "prompt_version_changes",
    "tool_schema_hash_changes",
    "policy_bundle_changes",
    "provider_model_changes"
  ]
}

The point is not this exact JSON. The point is that the cache key should encode the facts that change model behavior. A new tool parameter should invalidate the stable prefix. A policy bundle update should invalidate it. A model migration should invalidate it, even if the text stayed mostly the same, because the cached context was tuned and evaluated against another execution surface.

This is also where cost accounting becomes honest. Amazon Bedrock calls out that tokens written to cache and tokens read from cache can be accounted for differently depending on the model (Amazon Bedrock prompt caching). That means a cache strategy has a write path, a read path, and a break-even point. Production teams should measure all three, but only after correctness and isolation are defined.

Do not hide cache behavior from traces

If prompt caching affects latency, cost, and context freshness, it belongs in the trace. A model call trace should not only record the model name and token count. It should record whether a cache was attempted, whether it hit, which stable-prefix version was used, which invalidation dimensions formed the key, and which live context was appended after the cached segment.

A minimal trace record can be small:

{
  "span": "llm.call",
  "model": "production-chat-model",
  "cache": {
    "attempted": true,
    "hit": true,
    "contract": "agent_context.v1",
    "prompt_version": "support-agent.42",
    "tool_schema_hash": "sha256:7f8c...",
    "policy_bundle": "safety-2026-07-25"
  },
  "live_context": {
    "retrieval_docs": 6,
    "memory_items": 3,
    "tenant_filter_applied": true
  }
}

This trace shape helps during incident review. If an answer used the wrong refund policy, the team can ask whether retrieval returned stale evidence, whether the policy bundle was old, whether the cached prefix survived a deployment, or whether the model ignored correct context. Without these fields, everyone argues from screenshots and token totals.

It also helps evaluation. Cache-aware tests should run at least three cases: a cold request, a warm request with the same stable prefix, and a request after invalidation. If the answer changes only because the cache was warm, the system has a hidden state bug. If invalidation does not force the expected miss or prefix rebuild, the deployment pipeline is unsafe.

Multi-tenant agents need stricter defaults

Prompt caching becomes riskier when agents serve multiple tenants, use long-lived memory, or call tools with different authority levels. A shared cached prefix can be fine for global behavior instructions. It is not fine for tenant-specific permissions, customer contracts, private memory, or current security posture.

The safe default is boring:

1. Cache global, release-versioned instructions.
2. Cache tool schemas only with a schema hash.
3. Keep user memory outside the cached prefix unless it is encrypted, isolated, and scoped by tenant and principal.
4. Keep retrieval results live unless they are tied to a documented corpus snapshot.
5. Put cache version, hit status, and invalidation reason in the trace.
6. Include cache warm and cache invalidated cases in evals.

This is not anti-cache advice. It is pro-cache advice for systems that need to survive real traffic. The teams that get the most value from prompt caching are usually the teams with the strictest boundaries, because their cache hit rate is meaningful rather than accidental.

One useful rollout pattern is to start with a single workflow and one stable prefix. Log the cache key without secrets, run evals against cold and warm calls, then add an invalidation test to the deployment pipeline. After that, expand to additional workflows only when each one has a named owner and a documented live-context boundary. This keeps the platform team from shipping a clever global prompt cache that product teams cannot reason about. It also gives finance a cleaner cost story, because savings are tied to workflows with known correctness checks rather than blended across unrelated traffic.

The operational control should be equally simple. A team should be able to disable caching for one agent, one model route, or one tenant class while leaving the rest of the runtime alone. If disabling the cache requires editing prompts or changing application code, the cache is too deeply hidden.

A production checklist

Before turning prompt caching on for an agent or RAG workflow, I would ask for this checklist in the design review:

• Which prompt segments are stable across users, tenants, and requests?
• Who owns each stable segment?
• Which fields are included in the cache key?
• Which deployments or policy changes force invalidation?
• Are retrieval results cached, live, or pinned to a corpus snapshot?
• Can traces prove whether a response used a cached prefix?
• Do evals cover cold, warm, and invalidated paths?
• Does cost reporting separate cache writes, cache reads, and normal input tokens?
• Can operations disable caching for one workflow without redeploying the whole application?

If the team cannot answer those questions, prompt caching is still a useful experiment. It is not ready to be a production assumption.

FAQ

Should every long system prompt be cached?

No. Cache the parts that are stable, versioned, and safe to reuse. Long prompts often contain a mix of stable behavior, policy, examples, user memory, and request-specific evidence. Length alone is not a cache contract.

Should retrieval results go into the cached prefix?

Usually no. Retrieval is a live runtime decision unless the corpus snapshot, permission scope, and freshness rules are pinned. Treat retrieved evidence as live context by default, especially for customer data, policy answers, and incident analysis.

How should teams test prompt caching?

Run the same workflow cold, warm, and after an invalidation event. The warm path should preserve behavior while improving latency or cost. The invalidated path should rebuild the stable prefix when prompt version, tool schema, policy bundle, or model version changes.

Is prompt caching safe for multi-tenant agents?

It can be safe if the cached prefix contains only global, release-versioned context and the cache key cannot mix tenant-specific authority. User memory, permissioned retrieval, and tenant policy should remain live unless the isolation model is explicit and tested.

References

1. OpenAI prompt caching guide
2. Anthropic prompt caching documentation
3. Google Cloud Vertex AI context caching overview
4. Amazon Bedrock prompt caching guide
5. OpenAI File Search guide