Skip to content

feat: add cached token metrics support for Amazon Bedrock #531

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

oaltagar
Copy link

Description

Add cached token metrics support for Amazon Bedrock to track cache read/write input tokens.

Changes Made

  • Add optional cacheReadInputTokens and cacheWriteInputTokens fields to Usage TypedDict
  • Update EventLoopMetrics to accumulate cached token metrics
  • Add OpenTelemetry instrumentation for cached token telemetry
  • Enhance metrics summary display to show cached token information
  • Maintain 100% backward compatibility with existing Usage objects

Testing

  • ✅ All 762 unit tests pass
  • ✅ Comprehensive real-world testing with actual Bedrock API calls
  • ✅ Backward compatibility verified
  • ✅ Type checking and linting pass

Implementation Details

  • Modified src/strands/types/event_loop.py to add optional cached token fields using Required/Optional annotations
  • Updated src/strands/telemetry/metrics.py to handle cached token accumulation and telemetry
  • Added cached token constants in src/strands/telemetry/metrics_constants.py
  • Access pattern: result.metrics.accumulated_usage works seamlessly with cached tokens

Resolves

Closes #529

Checklist

  • Code follows project style guidelines
  • Self-review completed
  • Tests pass locally
  • Documentation updated (if needed)
  • Conventional commit format used

- Add optional cacheReadInputTokens and cacheWriteInputTokens fields to Usage TypedDict
- Update EventLoopMetrics to accumulate cached token metrics
- Add OpenTelemetry instrumentation for cached token telemetry
- Enhance metrics summary display to show cached token information
- Maintain 100% backward compatibility with existing Usage objects
- Add comprehensive test coverage for cached token functionality

Resolves strands-agents#529
@oaltagar
Copy link
Author

Hi maintainers,

I noticed that PR #392 addresses similar cached token metrics functionality. After reviewing both implementations, I wanted to provide transparency about the overlap and seek guidance on the best path forward.

Status Check:

Implementation Comparison:

Similarities (95% overlap):

  • Both add cacheReadInputTokens and cacheWriteInputTokens to Usage TypedDict
  • Both integrate with EventLoopMetrics for accumulation
  • Both add OpenTelemetry instrumentation
  • Both maintain backward compatibility
  • Both update metrics display formatting

Key Differences:

  • Scope: PR feat: expose cache metrics #392 implements across all model providers, while this PR focuses on Bedrock with a foundation for others
  • Type Safety: This PR uses Required/NotRequired annotations for more robust optional field handling
  • Error Handling: This PR includes more defensive None checking and type-safe accumulation
  • Testing: PR feat: expose cache metrics #392 updates existing tests across providers, while this PR includes comprehensive standalone real-API testing

Next Steps:

I'm happy to:

The core functionality is essentially the same, so I'm flexible on whichever approach best serves the project. My primary goal is getting cached token metrics available for users.

Please advise on the preferred path forward. Thanks for your time and consideration!

Note: I apologize for not catching the existing PR #392 during my initial research. In the future, I'll do more thorough PR searches before starting implementation work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE] Add cached token metrics support to Usage class
1 participant