Orleans 8.2 and up creates OpenTelemetry traces with thousands of spans

[Costo]

Hi!
We recently updated projects from Orleans 8.0 to 8.2 and we noticed that our Otel traces are mixed up. We have traces with thousands of spans. It seems that persistent pulling agents always reuse the same TraceId/OperationId as you can see in the screenshots below. The only significant change I see between 8.0 and 8.2 in the persistent pulling agent is the new Grain Timers. Could it be related?

I created a reproduction project using the Simple Streaming Sample:
https://github.com/Costo/orleans-otel-issue/tree/main

Thanks!

Screenshot of Aspire Dashboard:

Screenshot of Application Insights in Azure Portal:

[ReubenBond]

The issue here is that PersistentStreamPullingAgent's message pump runs for an indefinite period of time. It's kicked off on a timer and loops waiting for messages. I am preparing a fix to sever the tie between the timer callback and the message pump loop.

Many thanks for creating a repro, @Costo, it was very helpful.