🪨 feat: Add Bedrock Prompt Caching Support

[dustinhealy]

Summary

Closes #7109

Adds a new parameter, Use Prompt Caching, for Bedrock endpoints, enabling cache points to be automatically embedded in messages sent to supported models

Note: actual insertion logic relies on LibreChat-AI/agents#28 (as well as a bug-fix around tool calling in LibreChat-AI/agents#30). This PR now mainly concerns the addition of the Use Prompt Caching toggle itself as a parameter. Previous implementation details which have been moved and refactored for the agents repo (or are no longer necessary as a result of the refactor) are left with strikethroughs for the sake of historical tracking.

Change Type

• New feature (non-breaking change which adds functionality)

Details

• Introduced addBedrockCacheControl to insert cache points when Use Prompt Caching is enabled (api/app/clients/prompts/addBedrockCacheControl.js).
• Integrated logic into the Bedrock agent client so cache control is applied as appropriate (api/server/controllers/agents/client.js).
• Updated exports to make addBedrockCacheControl available (api/app/clients/prompts/index.js).
• Added promptCache (shown in UI as Use Prompt Caching) to Bedrock parameter settings and schema for configuration and validation (packages/data-provider/src/parameterSettings.ts, packages/data-provider/src/bedrock.ts).

Testing

• Enabled and disabled Use Prompt Caching via parameter panel and verified the presence or absence of cache points in messages sent to Bedrock endpoints.
• Using LangFuse, verified traces contained expected cache read and cache write usage metadata when prompt caching turned on for an agent using the Bedrock endpoint.

Test Configuration

Agent using AWS Bedrock endpoint with model: us.anthropic.claude-sonnet-4-20250514-v1:0 with Use Prompt Caching toggled on

Three message chain with high-token initial message (1000+ token) in order to trigger cache reads and cache writes.

Initial Input

User:
Use the Calculator tool, what is 5123043/39201? This is a long input test to see if prompt caching works. Lorem ipsum Lorem ipsum Lorem ipsum... <Continues for around ~1300 more tokens>

AWS Bedrock Prompt Caching (Tool Use) Test Agent:
I'll calculate 5123043/39201 for you using the calculator tool.
<Ran calculator>
The result of 5123043/39201 is approximately 130.687.

Usage Metadata:

{
    "inputTokens": 1794,
    "outputTokens": 26,
    "serverToolUsage": {},
    "totalTokens": 1820
}

---

Follow-up (triggers write to cache)

User:
This is a follow-up to check for cache_write

AWS Bedrock Prompt Caching (Tool Use) Test Agent:
I understand you're testing the cache functionality with this follow-up message. The previous calculation of 5123043/39201 gave us the result of approximately 130.687.

Is there anything else you'd like me to calculate or help you with?

Usage Metadata:

{
    "cacheReadInputTokenCount": 0,
    "cacheReadInputTokens": 0,
    "cacheWriteInputTokenCount": 1844,
    "cacheWriteInputTokens": 1844,
    "inputTokens": 4,
    "outputTokens": 58,
    "serverToolUsage": {},
    "totalTokens": 1906
}

---

Final follow-up (triggers read from cache)

User:
This is a follow-up to check for cache_read

AWS Bedrock Prompt Caching (Tool Use) Test Agent:
I see you're continuing to test the cache functionality. The calculation we performed earlier (5123043/39201 ≈ 130.687) should be available from the cached context along with all the Lorem ipsum text from your original message.

Is there anything specific you'd like me to help you with or any other calculations you need?

Usage Metadata:

{
    "cacheReadInputTokenCount": 1735,
    "cacheReadInputTokens": 1735,
    "cacheWriteInputTokenCount": 182,
    "cacheWriteInputTokens": 182,
    "inputTokens": 4,
    "outputTokens": 75,
    "serverToolUsage": {},
    "totalTokens": 1996
}

Checklist

• My code adheres to this project's style guidelines
• I have performed a self-review of my own code
• I have commented in any complex areas of my code
• I have made pertinent documentation changes
• My changes do not introduce new warnings
• I have written tests demonstrating that my changes are effective or that my feature works
• Local unit tests pass with my changes
• A pull request for updating the documentation has been submitted.

[KiGamji]

@dustinhealy Hi there! Do you have any due dates for this? Thanks in advance

[dustinhealy]

@dustinhealy Hi there! Do you have any due dates for this? Thanks in advance

Hi KiGamji, still need to write tests but planning to get this ready for review within the next couple days.

[KiGamji]

Hopefully we'll see this merged in the next week 👍

[danny-avila]

Hopefully we'll see this merged in the next week 👍

It requires an upstream change for it to fully work:

langchain-ai/langchainjs#8524

So we have to wait when that PR is tested/merged

[mikhailmokhov]

Hopefully we'll see this merged in the next week 👍

It requires an upstream change for it to fully work:

langchain-ai/langchainjs#8524

So we have to wait when that PR is tested/merged

Note that the mentioned PR is adding cache point support only to @langchain/aws package, make sure you're using it. Also, cachePoint must be a separate content block in the content array, see example: https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html#prompt-caching-get-started. You have it as a text block property.

[KiGamji]

@danny-avila @dustinhealy Hi there! Hope you consider this in any near time, as the mentioned PR was finally merged, and we would really like to see Bedrock with Prompt Caching! 👍

[danny-avila]

@danny-avila @dustinhealy Hi there! Hope you consider this in any near time, as the mentioned PR was finally merged, and we would really like to see Bedrock with Prompt Caching! 👍

still waiting on package release for @langchain/aws as the latest version 0.1.13 at the time of writing doesn't include prompt caching support:

https://www.npmjs.com/package/@langchain/aws

[danny-avila]

Lastly, rebase branch to latest (to resolve conflicts)

[Pfaufisch]

@langchain/aws 0.1.14 has been released and includes prompt caching support: langchain-ai/langchainjs#8524 (comment)

[KiGamji]

@danny-avila @dustinhealy tagging for visibility too. We would like to see this, leading to huge cost savings! Thanks in advance for your work!

[KiGamji]

any reason for this not being merged?

[danny-avila]

any reason for this not being merged?

not fully tested, will be part of next release a week from now. needs to be rebased with latest upcoming changes to dev branch

[ldavis9000aws]

@dustinhealy @danny-avila So this will be in the next release, right? :)

[Defiect]

@danny-avila Are there specific tests currently blocking this from being in the next release?

Really looking forward to this feature, as I'm sure many others are. Anthropic has been severely tightening the belt on usage limits via subscriptions; general user sentiment seems to indicate that users are increasingly switching to Bedrock to access their models

[KiGamji]

yeah, we are waiting for this too, as much as new Bedrock keys format support, and fixes for the PROXY env!

[AetherMagee]

Are there any blockers for this besides needing a rebase?

[dustinhealy]

Are there any blockers for this besides needing a rebase?

The agents repo still needs a version bump for @langchain/aws in order for this feature to actually work + we still have to do some testing on the Bedrock side of things to make sure the prompt caching points are working as intended. I will rebase later today after I finish my current PR just to resolve the merge conflicts. Thanks all for your interest, we know this is highly anticipated feature, so we appreciate your patience in the meantime!

[Defiect]

Are there any blockers for this besides needing a rebase?

The agents repo still needs a version bump for @langchain/aws in order for this feature to actually work + we still have to do some testing on the Bedrock side of things to make sure the prompt caching points are working as intended. I will rebase later today after I finish my current PR just to resolve the merge conflicts. Thanks all for your interest, we know this is highly anticipated feature, so we appreciate your patience in the meantime!

Awesome, thank you. Your effort is greatly appreciated!

[matthill33]

Are there any blockers for this besides needing a rebase?

The agents repo still needs a version bump for @langchain/aws in order for this feature to actually work + we still have to do some testing on the Bedrock side of things to make sure the prompt caching points are working as intended. I will rebase later today after I finish my current PR just to resolve the merge conflicts. Thanks all for your interest, we know this is highly anticipated feature, so we appreciate your patience in the meantime!

Awesome, thank you. Your effort is greatly appreciated!

I'm also very much looking forward to this feature being implemented, thank you @dustinhealy for your work on it!

[KiGamji]

@danny-avila Any chance to get this merged until October? I appreciate your time and effort in maintaining this huge project, but this PR is already two months old and it would need more effort for @dustinhealy to rebase and maintain it each time, unless it gets merged. Also, this is one of the mostly anticipated features by activity, along with conversation cost tracking (#9190, #8531), conversation pinning (#7550), PassKeys & E2EE, so I really hope you will give it some attention in the neartime. Thanks in advance!

[dvejsada]

@danny-avila Any chance to get this merged until October? I appreciate your time and effort in maintaining this huge project, but this PR is already two months old and it would need more effort for @dustinhealy to rebase and maintain it each time, unless it gets merged. Also, this is one of the mostly anticipated features by activity, along with conversation cost tracking (#9190, #8531), conversation pinning (#7550), PassKeys & E2EE, so I really hope you will give it some attention in the neartime. Thanks in advance!

Very anticipated indeed!;)

[Ziyann]

Would it be possible to also add this as an option for custom endpoints?
LiteLLM, OpenRouter and some others also need it, with the same logic, except instead of cachePoint: { type: 'default' }, they require cache_control: { type: 'ephemeral' }.

[ldavis9000aws]

Ok...I'm sure there are more important things the project is focused on, but just a reminder:

[KiGamji]

With all the respect to Danny, I absolutely agree with this :/
It has been 3 months...

[KiGamji]

#9798 seems to be dangling too

[KiGamji]

With all due respect, dear Danny and the LibreChat Team, I believe this is not a serious approach for user-made contributions to such a project...

[KiGamji]

A huge number of people and their users are waiting while being fed false promises.

[ldavis9000aws]

Hey @dustinhealy - thanks for updating the feature branch. Hopefully we'll start to get some traction on this.

[danny-avila]

Sorry for the wait on this. It's currently working but not for tool calling. @dustinhealy is looking into it

[matthill33]

Sorry for the wait on this. It's currently working but not for tool calling. @dustinhealy is looking into it

Thank you for your work on this!

[KiGamji]

It would be really amazing if you also review this, Danny: #8871
Appreciate it!

[KiGamji]