Skip to content

[FEATURE]: LLM Observability: Take into account Cached Tokens for OpenAI #13869

Closed
@ppetrovnexo

Description

@ppetrovnexo

Package Name

ddtrace

Package Version(s)

3.10.0rc1

Describe the goal of the feature

Enhance ddtrace’s OpenAI integration to:()
• Extract the cached_tokens value from API responses.
• Include cached_tokens in relevant metrics and traces.
• Adjust cost calculations to account for the discounted rate of cached tokens.()

Additional context

OpenAI’s prompt caching is available for models like gpt-4o, gpt-4o-mini, o1-preview, and o1-mini with API versions such as 2024-10-01-preview. Incorporating cached_tokens into ddtrace would provide more accurate observability and cost tracking for applications utilizing these models.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions