feat(conversation): add prompt cache + usage metrics to test caching by sicoyle · Pull Request #4154 · dapr/components-contrib

sicoyle · 2026-01-06T22:47:15Z

Description

This PR breaks out work from a larger contrib PR into a smaller, more focused change: #4129

It introduces response caching support via metadata passed to LLM providers. This serves as a workaround for LangChain’s WithPromptCaching(true) option, which currently sets a boolean value that fails Dapr conformance tests because OpenAI-based providers expect a duration string rather than a boolean.

To properly validate this workaround, usage metrics support was added, which required additional data type translations within LangChain.

The PR also renames CacheTTL to ResponseCacheTTL to more accurately reflect its behavior. Backward compatibility is maintained by continuing to support the original JSON tag.

Finally, the LangChain dependency was updated to support the newly required options that I am using.

Reconfirmed things work as expected:

✗  go test -v -tags=conftests -count=1 --failfast ./tests/conformance -run="TestConversationConformance/openai"
=== RUN   TestConversationConformance
=== RUN   TestConversationConformance/openai.openai
=== RUN   TestConversationConformance/openai.openai/init
=== RUN   TestConversationConformance/openai.openai/converse
=== RUN   TestConversationConformance/openai.openai/converse/get_a_non-empty_response_without_errors
=== RUN   TestConversationConformance/openai.openai/converse/test_user_message_type
=== RUN   TestConversationConformance/openai.openai/converse/test_system_message_type
=== RUN   TestConversationConformance/openai.openai/converse/test_assistant_message_type
=== RUN   TestConversationConformance/openai.openai/converse/test_developer_message_type
=== RUN   TestConversationConformance/openai.openai/converse/test_tool_message_type_-_confirming_active_tool_calling_capability_(empty_tool_choice)
=== RUN   TestConversationConformance/openai.openai/converse/test_conversation_history_with_tool_calls
=== RUN   TestConversationConformance/openai.openai/converse/test_prompt_cache_retention
    conversation.go:577: Request 1 Response Content: "2 + 2 equals 4."
    conversation.go:578: Request 1 Response Length: 15 characters
    conversation.go:583: Request 1 Usage - Total: 1038, Prompt: 1030, Completion: 8
    conversation.go:586: Request 1 Prompt Details - Cached: 1024, Audio: 0
    conversation.go:635: Request 2 Usage - Total: 1038, Prompt: 1030, Completion: 8
    conversation.go:638: Request 2 Prompt Details - Cached: 1024, Audio: 0
    conversation.go:640: Cached tokens on second request: 1024
=== RUN   TestConversationConformance/openai.azure
    conversation_test.go:83: Skipping Azure OpenAI conformance test: AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_TYPE, and AZURE_OPENAI_API_VERSION environment variables must be set
--- PASS: TestConversationConformance (6.41s)
    --- PASS: TestConversationConformance/openai.openai (6.40s)
        --- PASS: TestConversationConformance/openai.openai/init (0.00s)
        --- PASS: TestConversationConformance/openai.openai/converse (6.40s)
            --- PASS: TestConversationConformance/openai.openai/converse/get_a_non-empty_response_without_errors (0.62s)
            --- PASS: TestConversationConformance/openai.openai/converse/test_user_message_type (0.68s)
            --- PASS: TestConversationConformance/openai.openai/converse/test_system_message_type (0.87s)
            --- PASS: TestConversationConformance/openai.openai/converse/test_assistant_message_type (0.56s)
            --- PASS: TestConversationConformance/openai.openai/converse/test_developer_message_type (1.11s)
            --- PASS: TestConversationConformance/openai.openai/converse/test_tool_message_type_-_confirming_active_tool_calling_capability_(empty_tool_choice) (0.98s)
            --- PASS: TestConversationConformance/openai.openai/converse/test_conversation_history_with_tool_calls (0.77s)
            --- PASS: TestConversationConformance/openai.openai/converse/test_prompt_cache_retention (0.82s)
    --- SKIP: TestConversationConformance/openai.azure (0.00s)
PASS
ok      github.com/dapr/components-contrib/tests/conformance    7.439s

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

Code compiles correctly
Created/updated tests
Extended the documentation
- Created the dapr/docs PR:

Note: We expect contributors to open a corresponding documentation PR in the dapr/docs repository. As the implementer, you are the best person to document your work! Implementation PRs will not be merged until the documentation PR is opened and ready for review.

… works Signed-off-by: Samantha Coyle <sam@diagrid.io>

Signed-off-by: Samantha Coyle <sam@diagrid.io>

conversation/aws/bedrock/bedrock.go

conversation/echo/echo.go

conversation/langchaingokit/model.go

conversation/langchaingokit/translate.go

JoshVanL · 2026-01-07T13:55:31Z

conversation/langchaingokit/translate.go

+// NOTE: These are all translations due to langchaingo data types.
+
+// extractInt64FromGenInfo extracts an int64 value from genInfo map to extract usage data from langchaingo's GenerationInfo map in the choices response.
+func extractInt64FromGenInfo(genInfo map[string]any, key string) int64 {


Should we not return an error here when the value is not a number? Do we not expect other number types (uint)?

I initially defined this as int64 only in my original PR. However, after splitting this effort and what I'm assuming is a subsequent version bump, my conformance tests were no longer returning usage metrics, which required loosening the handling here. This change resolved the issue.

TLDR: Yes, I can add support for uint as a defensive measure and return an error for any other types.

conversation/langchaingokit/translate_test.go

JoshVanL · 2026-01-07T13:57:30Z

conversation/converse.go

+	Metadata map[string]string `json:"metadata"`
+	Model    *string           `json:"model"`
+
+	PromptCacheRetention time.Duration `json:"promptCacheRetention"`


Should this be optional?

Suggested change

PromptCacheRetention time.Duration `json:"promptCacheRetention"`

PromptCacheRetention *time.Duration `json:"promptCacheRetention"`

conversation/metadata.go

Signed-off-by: Samantha Coyle <sam@diagrid.io>

conversation/converse.go

Signed-off-by: Samantha Coyle <sam@diagrid.io>

sicoyle · 2026-01-13T15:16:48Z

ready for review again please @JoshVanL @cicoyle

Signed-off-by: Samantha Coyle <sam@diagrid.io>

…onents-contrib into feat-convo-api-prompt-cache

Signed-off-by: Samantha Coyle <sam@diagrid.io>

…onents-contrib into feat-convo-api-prompt-cache

sicoyle · 2026-01-14T20:41:50Z

im rerunning the failed cert tests as they are unrelated. This PR is ready for review please

conversation/langchaingokit/model.go

feat(conversation): enable prompt cache + usage metrics to show cache…

45d9d32

… works Signed-off-by: Samantha Coyle <sam@diagrid.io>

sicoyle requested review from a team as code owners January 6, 2026 22:47

style: rename for clarity

77147f8

Signed-off-by: Samantha Coyle <sam@diagrid.io>

sicoyle mentioned this pull request Jan 6, 2026

feat(conversation): add prompt cache + usage metrics to test cache dapr/dapr#9265

Closed

7 tasks

style: appease linter

6e4ea77

Signed-off-by: Samantha Coyle <sam@diagrid.io>

JoshVanL requested changes Jan 7, 2026

View reviewed changes

sicoyle mentioned this pull request Jan 7, 2026

Add Features to Conversation API for Dapr Agents v1.0 dapr/dapr#8784

Closed

10 tasks

sicoyle added 4 commits January 7, 2026 16:32

style: updates per feedback

7e17b59

Signed-off-by: Samantha Coyle <sam@diagrid.io>

style: appease linter

7ea5824

Signed-off-by: Samantha Coyle <sam@diagrid.io>

Merge branch 'main' into feat-convo-api-prompt-cache

44d16dc

Merge branch 'main' into feat-convo-api-prompt-cache

2a6fb13

JoshVanL requested changes Jan 12, 2026

View reviewed changes

conversation/converse.go Outdated Show resolved Hide resolved

sicoyle added 5 commits January 12, 2026 14:38

fix: clean up field i missed

89d0187

Signed-off-by: Samantha Coyle <sam@diagrid.io>

fix: address conflicts iwth main

876ef36

Signed-off-by: Samantha Coyle <sam@diagrid.io>

fix: int64 -> uint64 + linter fix

c67bcdf

Signed-off-by: Samantha Coyle <sam@diagrid.io>

fix: capture if nonneg num

420029b

Signed-off-by: Samantha Coyle <sam@diagrid.io>

Merge branch 'main' into feat-convo-api-prompt-cache

8327cd6

sicoyle added 5 commits January 13, 2026 12:21

fix(test): update unit tests

f202da3

Signed-off-by: Samantha Coyle <sam@diagrid.io>

Merge branch 'feat-convo-api-prompt-cache' of github.com:sicoyle/comp…

22476d7

…onents-contrib into feat-convo-api-prompt-cache

Merge branch 'main' into feat-convo-api-prompt-cache

5f02deb

fix(test): update unit test

80e19cf

Signed-off-by: Samantha Coyle <sam@diagrid.io>

Merge branch 'feat-convo-api-prompt-cache' of github.com:sicoyle/comp…

b806056

…onents-contrib into feat-convo-api-prompt-cache

JoshVanL approved these changes Jan 15, 2026

View reviewed changes

cicoyle approved these changes Jan 15, 2026

View reviewed changes

conversation/langchaingokit/model.go Show resolved Hide resolved

Merge branch 'main' into feat-convo-api-prompt-cache

84029cf

cicoyle merged commit f9bc128 into dapr:main Jan 15, 2026
91 of 93 checks passed

cicoyle added this to the v1.17 milestone Jan 16, 2026

sicoyle mentioned this pull request Jan 20, 2026

Conversation API new features added to docs dapr/docs#5011

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(conversation): add prompt cache + usage metrics to test caching#4154

feat(conversation): add prompt cache + usage metrics to test caching#4154
cicoyle merged 18 commits intodapr:mainfrom
sicoyle:feat-convo-api-prompt-cache

sicoyle commented Jan 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JoshVanL Jan 7, 2026

Uh oh!

sicoyle Jan 7, 2026

Uh oh!

Uh oh!

JoshVanL Jan 7, 2026

Uh oh!

Uh oh!

Uh oh!

sicoyle commented Jan 13, 2026

Uh oh!

sicoyle commented Jan 14, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	PromptCacheRetention time.Duration `json:"promptCacheRetention"`
	PromptCacheRetention *time.Duration `json:"promptCacheRetention"`

Conversation

sicoyle commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issue reference

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JoshVanL Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

sicoyle Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JoshVanL Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sicoyle commented Jan 13, 2026

Uh oh!

sicoyle commented Jan 14, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sicoyle commented Jan 6, 2026 •

edited

Loading