Baggage not being propagated in zio

[pallamidessi]

Hello 👋

We hit an issue were the baggage was not propagated across the network when an ot-baggage prefixed header is present in the incoming request using zio. This work well for the other services in our backend not using zio

The propagation style (both injection and extraction) is set to datadog and when turning the agent debug log, we can see the baggage being successful extracted but never inject in the subsequent upstream requests. Any pointer to how to debug the issue ?

We're running the version 1.30.1 of the agent, so this fix should be included already: #6442
Happy to provide more context and to learn where to start looking in the dd-trace-java codebase !

[amarziali]

Hello,

thanks for having opened an issue.

The mentioned PR only apply for context propagation across threads within the same application. Distributed tracing (network propagation) is not (yet) supported.

May you provide the scenario? which kind of network channel? Is it envolving zio-http or other?

I'm flagging this as a feature request

Thanks

[pallamidessi]

@amarziali Thanks a lot for the quick reply !

Yes, it is network propagation we are looking for :)
The App is using sttp with as a Backend zio-http running scala 3.3.1:

• ZioVersion = "2.0.18"
• ZioHttpVersion = "3.0.0-RC2"
• sttpVersion = "3.8.16"

Running on top of AWS corretto 17

May you provide the scenario?

Yes :) We are enable isolated deployment in our backend, to allow multiple version of a service to run and control the routing through HTTP headers. In essence, re-implementing Lyft setup here (great series of blog post btw): https://eng.lyft.com/scaling-productivity-on-microservices-at-lyft-part-3-extending-our-envoy-mesh-with-staging-fdaafafca82f

Instead of having to update all of our services and client library, we're relying of the propagation of the baggage as HTTP header since we already have the dd agent setup everywhere

It's niche use-case but worked very well for our others (in the hundreds) Play! and nodejs services, not just the zio ones

[amarziali]

@pallamidessi I did a quick test using this example: https://github.com/zio/zio-telemetry/blob/series/2.x/docs/opentelemetry-example.md

Then I started both applications (ProxyApp and BackendApp) with the following VM options using the latest available datadog tracer:

-javaagent:./dd-java-agent.jar -Ddd.trace.otel.enabled=true -Ddd.integration.zio.experimental.enabled=true

The distributed tracing is operated by the opentelemetry layer while the datadog tracer is interoperating with opentelemetry in order to get the spans that are created.

The flamegraph I could collect is showing that the traces are linked together as it would be expected:

I did not test baggage propagation but this should also interop with otel since by default, ot-baggage- prefixed header should be injected and extracted

I this example fully matching the use case that are you looking for?

Cheers

Andrea

[pallamidessi]

Thanks a lot @amarziali ! I'm gonna give it a shot and report back 👍