Python: Add telemetry for server latency tracing

[castlenthesky]

otel has established semantic conventions for tracing server latency.

Propose setting up latency metrics for the following:

• gen_ai.server.request.duration
• gen_ai.server.time_per_output_token
• gen_ai.server.time_to_first_token

These metrics allow users to diagnose latency and identify opportunities for pipeline enhancements. This feature will also elevate MS Agent Framework for production use-cases instead of a fun prototyping tool.

[castlenthesky]

I'd like to actively contribute to this project so please feel free to give me direct feedback about opportunities to more constructively contribute.

https://github.com/castlenthesky/agent-framework/tree/feature-telemetry-provider_latency

[eavanvalkenburg]

we have added the gen_ai.client.operation.duration metrics, because Agent Framework is not a server in itself, so I would assume something like Foundry or OpenAI would emit those kinds of metrics, not Agent Framework, which is a client.

[TaoChenOSU]

Agree with @eavanvalkenburg. We probably don't want to track those since we are a client SDK. Those metrics seem more appropriate for the software running on servers that serve models.

[castlenthesky]

*edits for spelling and clarity

I should have started off by saying how excited I am for this new framework.

I'm still learning about setting up telemetry and instrumentation. Reading more into the standards and conventions, I agree with your point that these metrics are not appropriately named for what they are intended to represent. I did further research, looking also at the OpenAI client's instrumentation, and their client takes a similar stance to the one you are recommending. I see where you are coming from and it's completely valid.

My thought was that when instrumenting an application using the Agent Framework it would be helpful for the framework to have appropriate telemetry to answer the following questions:

1. How long was my request in transit to and from the LLM provider before I began seeing a response?

• A client-side version of the gen_ai.server.time_to_first_token or time to first token (TTFT) metric
• Say gen_ai.client.operation.time_to_first_token?
• This allows ops to measure (and ultimately optimize) overall lag/latency from the LLM providers' APIs (provisioning, message queue, etc...)

2. How many tokens per second were generated DURING GENERATION

• A client side version of the gen_ai.server.time_per_output_token metric
• Say gen_ai.client.operation.time_per_output_token?
• This allows ops to measure (and ultimately optimize) LLM providers based on their speed/cost

3. How long did my request take to complete in total?

• The current gen_ai.client.operation.duration you have already implemented

Effectively answering these questions feels like a low-lift & high-value add for this new framework.

The additional piece here is that many builders are not running inference locally and likely don't have access to the server's telemetry to measure directly. Having a completion/response request sitting in a queue is fundamentally different that waiting for the generation request to finish and knowing how long my request is in either/both of these stages is valuable from an LLM ops optimization standpoint.

Are you open to considering the possibility of having the setup_observability() optionally enable a set of client-side metrics in addition to the gen_ai.client.operation.duration?

I really do believe these are critical metrics for a framework that is meant to have enterprise features. I'm happy to modify my PR if you think there is value here.

[TaoChenOSU]

There are a couple things to consider when dealing with tokens:

1. As a client SDK, we don't receive tokens. We receive the full response in non-streaming mode and chunks in streaming mode. If we do instrument time_to_first_token (only in streaming mode), it won't be an accurate representation of latency of the LLM provider concerning token generation on the client side. A more valuable metric would be time_to_first_chunk or similar.
2. It'd still be difficult to interpret time_to_first_chunk because now models may think (reasoning) before the actual response.
3. Similarly, for time_per_output_token, we don't receive tokens.

I do think we should enable client side metrics by default with setup_observability()

[castlenthesky]

1. I think you're capturing the spirit of what I'm getting at. If the correct term is chunk I propose gen_ai.client.operation.time_to_first_chunk, and agree that this only applies in streaming mode.
2. I think a user setting up telemetry will understand the nuance of thinking vs text chunking and could handle their own spans for differentiating the two.

My initial PR does not enable additional metrics by default, I figured they could be optional, but happy to align to whatever strategy you all decide here.

Again, willing to re-work the PR for better alignment if the team believes there's value here.

[TaoChenOSU]

1. I think you're capturing the spirit of what I'm getting at. If the correct term is chunk I propose gen_ai.client.operation.time_to_first_chunk, and agree that this only applies in streaming mode.
2. I think a user setting up telemetry will understand the nuance of thinking vs text chunking and could handle their own spans for differentiating the two.

My initial PR does not enable additional metrics by default, I figured they could be optional, but happy to align to whatever strategy you all decide here.

Again, willing to re-work the PR for better alignment if the team believes there's value here.

time_to_first_chunk should also only be a metric at the LLM level and is probably not suitable for agents.

[castlenthesky]

@TaoChenOSU, great callout. Please let me make sure I understand correctly:

I believe you're proposing instrumenting the LLM provider clients instead of the agent classes, is this correct?

If so, my initial branch added instrumentation to _clients.py via a _instrumented_stream() method on the BaseChatClient class. This felt like the right place to add the telemetry as this class appears to be the central class each of the providers' classes inherit from and by adding the telemetry on this class we can define once and any request to the current or future providers would be automatically instrumented.

Is there somewhere more appropriate or a different approach you had in mind for these metrics?

[eavanvalkenburg]

we have a decorator on each client, use_observability also defined in observability.py that get's applied to all chat client implementation.

[castlenthesky]

Understood. I'll rework my fork with changes for the following:

• gen_ai.client.operation.time_to_first_chunk
• gen_ai.client.operation.time_per_output_chunk

I will add these metrics to the use_observability decorator defined in observability.py.

[TaoChenOSU]

It'd be great if you can propose these two new metrics to the OTel GenAI Semantic Conventions so that they become part of the standard: https://github.com/open-telemetry/semantic-conventions

I can also propose there if you'd like.