Skip to content

Conversation

@anakin87
Copy link
Member

@anakin87 anakin87 commented Jan 29, 2026

Related Issues

Proposed Changes:

  • allow setting cache points in messages
    ChatMessage.from_user("long message..."), meta={"cachePoint": {"type": "default"}}
    • aligns with Bedrock docs
    • The documentation is ambiguous about multiple cache points per message. Implementing this would be hard/hacky within the current ChatMessage abstraction, so this PR only supports one cache point per message.
    • we also discussed in Add prompt caching for AmazonBedrockGenerator and AmazonBedrockChatGenerator #2776 (comment) about adding a global parameter to the Chat Generator to cache all user/system messages. However, Bedrock allows a maximum of 4 cache points per request, so per-message cache points are more transparent and allow finer control.
  • add a tools_cachepoint_config init parameter to the ChatGenerator to enable caching for tools
    • Bedrock may support more granular caching for tools, but it is hard to implement while respecting Haystack API, so I adopted this simpler solution.

How did you test it?

CI, new unit and integration tests.

Checklist

@github-actions github-actions bot added integration:amazon-bedrock type:documentation Improvements or additions to documentation labels Jan 29, 2026
@anakin87 anakin87 changed the title Bedrock prompt caching feat: Bedrock - support prompt caching Jan 29, 2026
@anakin87 anakin87 marked this pull request as ready for review January 29, 2026 18:26
@anakin87 anakin87 requested a review from a team as a code owner January 29, 2026 18:26
@anakin87 anakin87 requested review from julian-risch and removed request for a team January 29, 2026 18:26
@anakin87 anakin87 self-assigned this Jan 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

integration:amazon-bedrock type:documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants