Optimizing cost and latency with Amazon Bedrock prompt caching

extract

Artificial Intelligence Optimizing cost and latency with Amazon Bedrock prompt caching Prompt caching in Amazon Bedrock can reduce your input token costs by up to 90 percent when you repeatedly send the same context to foundation models, based on Amazon Bedrock prompt caching pricing. Without caching, a 10,000-token contract sent alongside 50 user questions means 500,000 input tokens billed at full price for content the model has already processed. You can mitigate this issue by shortening prompts, reducing context windows, or implementing application-level caching. Each option involves a trade-off: - Shortened prompts reduce token count but might also reduce context quality. - Smaller context windows lower cost at the expense of the model’s ability to reason over complete information. - Response caching handles identical queries well, yet provides no benefit when the same context is paired with different questions. Prompt caching in Amazon Bedrock helps reduce this challenge at the infrastructure level. When you cache parts of your conversation context (system prompts, documents, tool definitions), Amazon Bedrock reads the cached tokens on subsequent requests instead of reprocessing them. This can reduce time-to-first-token (TTFT) and lower costs for cached input tokens by up to 90 percent on cache hits, without changing your model or prompt quality. This post walks through six practical prompt caching scenarios using the Converse API in Amazon Bedrock, progressing from basic to advanced patterns: - Message content caching: Cache long documents for multi-question analysis. - System prompt caching: Cache persona definitions and instructions across conversations. - Tool definition caching: Cache tool schemas for agentic workflows. - Mixed TTL caching: Assign different cache lifetimes to different content tiers. - Tenant isolation: Implement per-tenant cache separation in multi-tenant applications. - LangChain integration: Use prompt caching with the LangChain framework. How prompt caching works Prompt caching stores a snapshot of partially processed input so that subsequent requests with the same prefix skip redundant computation. This section covers the request flow, supported models, and pricing. When you include a cachePoint marker in your request, Amazon Bedrock evaluates whether the content preceding that marker matches an existing cache entry. If it does (a cache hit), the model can skip reprocessing those tokens and begin generation from the cached state. If no match exists (a cache miss), the model processes the full content and writes the result to cache for potential future requests. This diagram shows the flow: With this flow in mind, four key concepts determine how caching behaves in practice: - Cache scope: Cache entries are scoped to individual AWS accounts and AWS Regions. - Token thresholds: Each cache checkpoint must meet a minimum token threshold to activate. For example, Anthropic Claude Sonnet 4.5 and Sonnet 4.6 require at least 1,024 tokens per checkpoint, while Opus models require at least 4,096. - Time-to-live (TTL): Cache entries expire based on the TTL specified in the request. The default is 5 minutes, with select models supporting up to 1 hour. - Model-agnostic syntax: The Converse API cachePoint syntax is identical across supported model families, including Anthropic Claude and Amazon Nova. For the latest model support information, see the Amazon Bedrock Prompt Caching documentation. Pricing Prompt caching introduces two token categories in addition to standard input and output tokens: For workloads with repeated context, the savings reach approximately 75 percent on input token costs. For example, if you send a 10,000-token document with 10 different questions, the first request incurs a cache write cost. The remaining nine requests each read from cache at 90 percent reduced cost, resulting in a net savings of approximately 75 percent on input token costs for that document context. This assumes all subsequent requests occur within the TTL window. Requests after expiration trigger a new cache write, reducing the net savings. See Amazon Bedrock pricing for detailed pricing information. Prerequisites Before getting started with the scenarios, make sure you have the following: - An AWS account with Amazon Bedrock access in a supported AWS Region (such as us-west-2 ). - Model access enabled for the target model. The examples in this post use Anthropic Claude Sonnet 4.5 (global.anthropic.claude-sonnet-4-5-20250929-v1:0 ). See Manage model access for instructions. For the latest model and Region availability, see Supported models by AWS Region in Amazon Bedrock. This is a cross-Region inference profile. Requests automatically route across Regions, which can occasionally increase cache write frequency. - Python 3.10 or later with the following dependencies installed: Note: Boto3 1.43.0 or later is required for the ttl parameter in cachePoint used in Scenario 4 (Mixed TTL). - AWS credentials configured through the default profile or environment variables. See Configure the AWS Command Line Interface (AWS CLI) for setup instructions. Scenario 1: Message content caching A common use case for prompt caching is caching long documents or reference content that you query repeatedly. For example, in a Retrieval Augmented Generation (RAG) application, you ask multiple questions about the same document, or a coding assistant references a large codebase. In this scenario, you place a cachePoint marker between the static document and the dynamic question. Amazon Bedrock caches the document on the first call and might reuse it on subsequent calls. How message content caching works Place a cachePoint content block after the static content and before the dynamic question. Amazon Bedrock caches everything before the checkpoint and reuses it on subsequent requests: The following code puts this pattern into practice with a complete working example. Implementation First, set up the Amazon Bedrock runtime client and define a sample document. In a production application, this document can be a PDF, a knowledge base article, or other content exceeding the 1,024-token threshold: Next, define the caching function. The key elements are the cachePoint block placed between the static document and the dynamic question: Run two requests to observe the caching behavior. The initial call populates the cache, and a subsequent call with a different question reuses it: Reading cache metrics The response usage object includes two cache-specific fields: In our testing with Anthropic Claude Sonnet 4.5 and a document exceeding 1,024 tokens, the initial response shows a cache write: A subsequent request with the same document prefix produces a cache read: Notice that cacheReadInputTokens now reflects the 1,898 tokens read from cache. The entire document prefix was reused without reprocessing. Only 28 tokens (the question itself) were processed as standard input. These cached tokens are billed at the reduced cache-read rate (90 percent lower than standard input). Simplified cache management Claude models on Amazon Bedrock support simplified cache management. You can place a single cachePoint, and Amazon Bedrock automatically checks for cache hits on prefixes up to approximately 20 content blocks before that marker. You do not need to manually place multiple cache checkpoints to get cache hits on earlier portions of your conversation. For more granular control, you can place multiple cachePoint markers after each section of content: This approach supports partial cache hits. If only the first two sections match a previous request, the model reuses the cache for those sections and processes the remaining content. Streaming variant The same caching syntax works with converse_stream. The key difference is that cache metrics arrive in the metadata event at the end of the stream rather than in the immediate response: TTFT benchmark To quantify the latency improvement, you can measure TTFT with and without caching: Prompt caching can reduce TTFT, with the benefit growing as the cached prefix size increases. For smaller documents (approximately 2,000–5,000 tokens), the improvement may not be statistically significant across a small number of iterations. The exact improvement varies based on document size, model, and current load. For large cached prefixes (over 10,000 tokens), the reduction in TTFT becomes more pronounced. Scenario 2: System prompt caching Many applications use detailed system prompts that define the model’s persona, guidelines, and domain expertise. These system prompts can span thousands of tokens and remain constant across user interactions. With system prompt caching, you pay the full processing cost once and reuse the cached system prompt for every subsequent message. The cache point goes inside the system parameter, separate from user messages. Configuration The Converse API system parameter accepts an array of content blocks. Place a cachePoint after the system text: The next example shows this in a full request with a detailed persona prompt. Implementation The following example defines a comprehensive system prompt: an Expert Space Science Advisor persona with detailed response guidelines. The prompt exceeds the 2,048-token threshold required for caching: The caching function places the cachePoint in the system parameter, keeping user messages separate: The user message changes between requests, but the system prompt remains identical. Amazon Bedrock is designed to cache the system prompt prefix and reuse it, regardless of what the user asks. When to use system prompt caching System prompt caching is ideal for: - Persona-based assistants with detailed role descriptions and behavioral guidelines. - Agentic workflows with extensive instructions that stay constant across turns. - Customer service bots with complex company policies and response protocols. - Domain-specific assistants with embedded knowledge bases in the system prompt. Scenario 3: Tool definition caching Agentic applications often define dozens of tools with comprehensive JSON schemas. These tool definitions can collectively contain thousands of tokens and rarely change between requests. Tool definition caching prevents reprocessing these schemas on every turn. Setup Append a cachePoint as the last element in the tools array within toolConfig: The example below demonstrates this with a set of tool schemas that collectively exceed the token threshold. Implementation Here we define a set of space-themed tools with comprehensive schemas. In a production application, these might be API integrations, database queries, or external service calls: We build the Converse API tool format and append the cachePoint: Tool definition caching is useful for agentic workflows where the same set of tools is invoked across many conversation turns. By caching the tool schemas once, you avoid reprocessing thousands of tokens of schema definitions on every turn. Scenario 4: Mixed TTL caching Cached content doesn’t have a single lifecycle. Core reference material (domain knowledge, product catalogs, compliance rules) rarely changes and benefits from longer cache durations. Session-specific context (recent conversation turns, user preferences) changes more frequently and benefits from shorter expiration. With mixed TTL caching, you can assign different expiration times to different content tiers within a single request. Approach Each cachePoint can include a ttl field. There is one ordering constraint: longer TTL checkpoints must appear before shorter ones in the request: Implementation In this scenario, the content is split into two tiers: a core reference section cached for 1 hour and a session context section cached for 5 minutes: Understanding cacheDetails The Converse API response includes a cacheDetails field that shows the per-TTL token breakdown. This shows whether both TTL tiers are working correctly. When both sections independently exceed the model’s minimum token threshold, you will see one entry per TTL: If only one section exceeds the threshold, the response will show a single TTL entry. Each content tier must independently meet the model’s minimum token requirement to get separate cache entries for each TTL. TTL ordering constraint Cache checkpoints must be ordered from longest to shortest TTL within a single request. The API returns an error if a shorter TTL appears before a longer one: - ✅ Valid: 1h then5m . - ❌ Invalid: 5m then1h . This constraint applies across cache checkpoint locations (messages, system prompt, and tool definitions). When to use mixed TTL The following table maps content types to recommended TTLs based on how frequently the content changes: With mixed TTL in place, the next scenario addresses a common multi-tenant challenge: preventing one tenant’s cached content from being read by another. Scenario 5: Tenant isolation In multi-tenant applications, you must prevent one tenant’s cached content from being read by another tenant. Prompt caching in Amazon Bedrock scopes entries by account and Region, but within the same account and Region, cache entries might be shared across requests. The SHA-256 hash prefix pattern provides tenant isolation with only approximately 16 tokens of overhead, without requiring separate AWS accounts. Pattern Prepend a SHA-256 hash of the tenant_id to the cached content. Because the hash changes the content prefix, Amazon Bedrock creates a separate cache entry for each tenant: Implementation The following function applies the SHA-256 tenant prefix to the cached content: You can check the isolation behavior across tenants: The expected behavior across the four requests: This pattern provides three advantages: - No server-side configuration: You achieve isolation purely through content prefixing. - Independent cache entries per tenant: Each tenant’s cached content stays separate. - Minimal overhead: The SHA-256 hash adds only 64 characters (approximately 16 tokens) to the prompt. Scenario 6: LangChain integration For teams using the LangChain framework, prompt caching integrates with the ChatBedrockConverse class. LangChain provides a create_cache_point() method that generates the correct cachePoint content block without requiring you to manage the raw API format. Usage Use ChatBedrockConverse.create_cache_point() within message content arrays or ChatPromptTemplate definitions: Implementation: Direct message construction The following example caches a document and asks questions with prompt caching enabled: Implementation: LCEL chain with cache point For reusable chain patterns, you can integrate cache points directly into a ChatPromptTemplate: Inspecting cache metrics in LangChain The usage_metadata on the response object includes input_token_details with cache-specific fields: - cache_creation : tokens written to cache (first request) - cache_read : tokens read from cache (subsequent requests) Comparing the Converse API and InvokeModel API The scenarios in this post use the Converse API, which provides a model-agnostic cachePoint syntax. If your application uses the InvokeModel API, be aware that the caching syntax differs by model family: Use the Converse API for new applications because the cachePoint syntax works identically across the supported model families. This means you can switch between Anthropic Claude and Amazon Nova without changing your caching code. Best practices Based on the patterns demonstrated in this post, we recommend the following best practices for production deployments: - Profile your prompts: Identify which components are static (system prompts, tool schemas, reference documents) and which are dynamic (user questions, session context). Cache the static components. - Meet the token threshold: Each cache checkpoint must exceed the model’s minimum token requirement. Amazon Bedrock processes content below the threshold normally without caching. - Choose appropriate TTLs: Use 1-hour TTLs for rarely-changing content (domain knowledge, tool definitions) and 5-minute TTLs for session-specific context. Remember that longer TTLs must appear before shorter ones. - Monitor cache metrics: Track cacheWriteInputTokens andcacheReadInputTokens in your application logs. A low cache-hit ratio might indicate that your content is changing too frequently or that your TTL is too short. - Apply responsible AI controls: For production deployments, use Amazon Bedrock Guardrails to add content filtering and grounding validation alongside your caching patterns. - Implement tenant isolation for multi-tenant systems: Use the SHA-256 hash prefix pattern to prevent cross-tenant cache sharing when serving multiple customers from the same AWS account. - Use simplified cache management: For Anthropic Claude models, a single cachePoint can cover multiple preceding content blocks. You don’t always need to place checkpoints after every section. - Combine caching locations: You can cache system prompts, message content, and tool definitions simultaneously in a single request. This stacks the savings across the three locations. Clean up The examples in this post use on-demand Amazon Bedrock inference and don’t create persistent AWS resources. No cleanup is required beyond stopping any running Jupyter notebook kernels. Conclusion This post walked through six practical prompt caching scenarios using the Amazon Bedrock Converse API, progressing from basic document caching to advanced patterns like mixed TTL and tenant isolation. Each scenario addresses a specific production challenge: - Cache content to reduce costs when you ask multiple questions about the same document. - Reuse the prompt and avoid reprocessing detailed persona definitions on every conversation turn. - Optimize agentic workflows by efficiently caching tool schemas once. - Tune mixed TTLs for fine-grained control over cache lifetimes for content with different update frequencies. - Isolate tenant caches for safe multi-tenant deployments with per-tenant cache separation. - Integrate with LangChain to bring prompt caching to your framework with minimal code changes. The cachePoint syntax is model-agnostic across the Converse API. The same code works with Anthropic Claude, Amazon Nova, and other supported models that support prompt caching on Amazon Bedrock. You can adopt prompt caching incrementally, starting with the highest-impact scenario for your workload and expanding to additional patterns as needed. For next steps, profile your existing Amazon Bedrock applications to identify opportunities for prompt caching. Start with the scenario that matches your workload (document analysis, persona-based assistants, or agentic tool use) and measure the TTFT and cost improvements. Each scenario includes working code that you can run immediately against Amazon Bedrock. The complete set of notebooks and scripts referenced throughout this post is available in the amazon-bedrock-samples GitHub repository. For more details on prompt caching configuration and supported models, see the Amazon Bedrock Prompt Caching documentation. To get started, visit Amazon Bedrock or open the Amazon Bedrock console to enable model access and begin using prompt caching in your applications.