Propagate LLM Observability context across service boundaries

[ncybul]

What Does This Do

Carries LLM Observability context (trace id, ML app, session ID, agent attribution, parent span id, and the sampling verdict) across process boundaries, so an LLM trace stays connected when work crosses a queue or a service call.

Approach

Write the _dd.p.llmobs_* tags from inside the existing tracing propagator rather than as a separate concern, so every boundary auto-instrumentation already covers (SQS, HTTP, gRPC, Kafka) is handled at once.

agent-llmobs registers an LLMObsPropagationSource at startup via LLMObsInternal.setPropagationSource. At injection the codec calls LLMObsInternal.propagationValuesFor(spanContext) and threads the result through PropagationTags.headerValue(headerType, lastParentIdOverride, llmObsValues). When LLM Observability is disabled nothing registers a source and the codecs skip the lookup.

Tag	Source
_dd.p.llmobs_ml_app	innermost active LLMObs span's ml_app
_dd.p.llmobs_sid	effective session_id
_dd.p.llmobs_pagent_span_id	nearest agent-kind ancestor's span id
_dd.p.llmobs_pagent_name	that ancestor's name
_dd.p.llmobs_parent_id	innermost active LLMObs span's span id
_dd.p.llmobs_trace_id	the LLMObs trace id
_dd.p.llmobs_sr	LLMObs sample rate
_dd.p.llmobs_sd	LLMObs sampling decision

llmobs_trace_id is seeded from the APM trace id on a Java-only trace but carried explicitly, since an id that arrived from another language is not the APM one. It is stored as 32-character hex and sent as decimal, because released dd-trace-py parses the tag with int(x). LLMObsTraceId converts both ways, mirroring _trace_id_to_wire / _normalize_wire_trace_id_to_hex.

Values are resolved from the ambient LLMObsContext at injection time and passed as an argument rather than stored, because they belong to the span being injected while PropagationTags can be shared by every span in a local trace — holding them there would let concurrent injections overwrite each other's LLMObs context. Injection is gated on trace-id consistency (the same gate DDLLMObsSpan applies in-process). Tags are written in order of importance, so the join keys survive if the header budget is exceeded and agent attribution degrades first.

Receive side

Two possible parents: the ambient in-process LLMObsContext, and the _dd.p.llmobs_* values on the extracted span context. DDLLMObsSpan prefers the in-process one, but only when it belongs to the same trace — otherwise it falls back to the extracted values. An explicit value passed by the caller wins over both.

Falling back on trace mismatch, not only on absence, is what makes a queue consumer correct: a scope leaked from a previously handled message would otherwise shadow the attribution that arrived with the current one.

"Extracted values" means what arrived on the inbound headers. The LLMObsTagValues a PTags holds are only the decoded inbound ones — a local span's tags are never written back into it — which matters because CoreTracer hands the extracted context's PropagationTags to the local root. Two behaviours follow:

• A service that forwards a request without opening an LLMObs span of its own keeps passing its caller's context along: the source returns null and the codec writes the inbound values.
• A peer LLMObs span opened after a sibling has finished reads the inbound values, not the sibling's.

Other Changes

ml_app precedence

The service default now applies last, in DDLLMObsSpan, instead of eagerly in the span factory — substituting it at the factory left no way to distinguish "caller named no application" from an explicit value, making every inheritance step unreachable.

Order is explicit > in-process LLMObs parent > propagated > DD_LLMOBS_ML_APP > DD_SERVICE, matching dd-trace-py.

The only behaviour change: a nested span naming no ml_app now inherits its enclosing span's value rather than the default. Propagation requires this — injection reads the innermost active span's ml_app, and on the receiving side only the first LLMObs span reads the propagated value.

Validating application-supplied values

Unlike other _dd.p.* tags these come from the application, so they are checked before reaching the wire. A value carrying a character the receiving codec mis-parses fails the whole tagset with decoding_error and takes _dd.p.tid with it, leaving the two services disagreeing about the upper 64 bits of the trace id.

Rejected: anything outside printable ASCII, , (the x-datadog-tags separator), " and \ (AWS messaging concatenates these headers into a _datadog JSON attribute without escaping), and anything the tracestate conversion rewrites irreversibly — ; and ~, which both become _. That last check asks TagValue's own conversion table rather than restating it:

static boolean survivesW3CRoundTrip(char c) {
  return convertW3CtoDD(convertDDtoW3C(c)) == c;
}

= is deliberately kept — it round-trips as ~, and base64-ish session ids carry it routinely.

Rejecting rather than sanitizing, matching dd-trace-py: a _-mangled ml_app can silently collide two applications into one bucket, which is worse than an absent tag falling back to the receiver's own default.

The unescaped JSON concatenation in MessageAttributeInjector / TextMapInjectAdapter is a property of those carriers, not of LLMObs — any product's _dd.p.* value containing a quote hits it. Escaping at the injectors is the broader fix, left to agent-bootstrap.

Also changed: the shared AWS messaging attribute parser

DatadogAttributeParser — the _datadog attribute parser used by SQS, SNS, EventBridge and Step Functions — never read x-datadog-tags, so every _dd.p.* tag was silently dropped crossing those boundaries. Inject was always correct. Pre-existing bug, not introduced here, but needed for LLMObs to cross a queue.

It forwards only the _dd.p.llmobs_* tags. Forwarding everything would change behaviour for other products downstream of a queue — _dd.p.ts would shift ASM consumer sampling, _dd.p.tid would change the dd.trace_id format in logs.

Property lookups are now anchored to quoted property names rather than a bare substring search. This PR is the first thing to put application-supplied strings into x-datadog-tags, so a value containing another header's name could otherwise shadow the real one: with an ml_app of x-datadog-sampling-priority:9, the sampling priority read back as ",".

Follow-ups (not in this PR)

• Restoring extracted context into the ambient LLMObsContext. Not covered: the fully-automatic path where an inbound trace is followed by an auto-instrumented call with no manual LLMObs span — OpenAiDecorator reads LLMObsContext.current(), still null there. Someone has to own attaching and closing a scope around the consumer's work, and that owner differs per integration.
• A length cap on application-supplied values. ml_app and session_id are charset-checked but not length-checked; an oversized value drops the whole header via inject_max_size. dd-trace-py behaves identically, so worth revisiting cross-language.
• Step Functions. Writes trace context into the request input body rather than message attributes, so it needs its own extract path. A gap rather than a regression.
• The same per-injection race in dd-trace-py. Java avoids it by threading the values through headerValue instead of staging them on shared trace state. dd-trace-py still stages: LLMObs._inject_llmobs_context writes into span_context._meta, which is shared trace-wide, so two threads injecting from different LLMObs scopes on one trace can overwrite each other. Worth a cross-language ticket rather than a Java-side fix.

Testing

• LLMObsContextPropagationSourceTest — none of these call any LLMObs propagation API: injection writes the tags; nothing added with no active span; no leak from one injection into a later one on the same trace; producer → worker round trip; the trace id goes out as the decimal the wire carries; a worker adopts an LLMObs trace id that differs from the APM trace; a pass-through service with no LLMObs span forwards its caller's context; the sampling verdict is written on inject and honoured by the consumer; a peer span doesn't inherit a finished span's injected context; a worker with no upstream inherits nothing.
• LlmObsContextPropagationForkedTest — the same behaviour end to end through the openai-java instrumentation.
• DDLLMObsSpanMlAppTest — pins the precedence chain. Verified load-bearing by neutralising both fallbacks.
• LLMObsTraceIdTest — hex ↔ decimal both ways: round trip, the ambiguous all-digit case, values that are neither, and a value wider than 128 bits padded rather than truncated.
• DatadogPropagationTagsTest / W3CPropagationTagsTest — the tags through both codecs: round trip, per-value rejection of the characters above, and decoding on read (app~v1 reads back as app=v1).
• DatadogAttributeParserTest — x-datadog-tags reaches the extractor, only _dd.p.llmobs_* is forwarded, and a property whose name appears inside an earlier value still resolves.
• LLMObsSpanMapperTest — the serialized span reports an adopted LLMObs trace id that diverges from the APM trace id, and falls back to the APM one when nothing was adopted.
• LLMObsContextTest — attaching a context propagates all mechanisms together.

Each fix was checked to be load-bearing by reverting it in isolation and confirming the intended test — and only that test — fails.

End to End Testing

Before

HTTP calls are split across more than one LLMObs trace (agent trace and tool trace) whereas the APM trace is connected.

SQS messages are split across more than one LLMObs trace (agent trace and tool trace). The APM trace is connected.

After

Six scenarios on 8f44e06993, each a real pair of OS processes reporting through a Datadog Agent. No application class in any of them calls an LLMObs propagation API — they open an LLMObs span and then make an ordinary HTTP request or sendMessage.

#	Boundary	Caller → callee	What it covers	Trace
1	HTTP	Java → Java	a request scope the server framework owns	LLMObs · APM
2	SQS	Java → Java	an iterator inside the SQS client, activating a consume span per message	LLMObs · APM
3	HTTP	dd-trace-py → dd-trace-java	the read path — LLMObsTraceId.fromWire	LLMObs · APM
4	HTTP	dd-trace-java → dd-trace-py	the write path — LLMObsTraceId.toWire	LLMObs · APM
5	HTTP, tracestate only	Java → Java	the 256-char dd= budget — tags shed in the documented order	LLMObs · APM
6	HTTP, concurrent fan-out	Java → Java	6 sibling LLMObs spans on one trace injecting at once	LLMObs · APM

In every one, the downstream span lands in the caller's LLMObs trace carrying the caller's session_id, parent_id and agent_attribution, with nothing set locally. A control request with no upstream context reports parent_id: undefined, no session and no attribution.

Scenarios 3 and 4 are the ones that can catch a wire-format bug. Java-to-Java agrees on the APM trace id by construction, so the LLMObs trace id could be wrong the same way on both sides and the join would still look right. A second tracer removes that alignment, and the two directions cover the two halves of the hex ↔ decimal conversion.

Scenario 3, decoded from the raw intake payload — Python's own two ids already differ:

planner-agent (agent)   language: python
  llmobs trace_id   6ab41f6e00000000fea4f6b7295bd5e9
  apm_trace_id      6ab41f6e000000002a908918c73a5567   <- differs
  span_id           12571653173807582101

fulfill_order (tool)    language: jvm
  llmobs trace_id   6ab41f6e00000000fea4f6b7295bd5e9   <- adopted from the wire
  parent_id         12571653173807582101               <- Python's agent span
  session_id        session-0
  agent_attribution {pagent_name: planner-agent, pagent_span_id: 12571653173807582101}

Scenario 4 confirms the other half: java client trace_id 5522694557363108731 arrives as py llmobs_trace_id 6ab41f9b000000004ca48d707e9c7b7b, and 0x4ca48d707e9c7b7b == 5522694557363108731. Had toWire emitted hex, Python would have dropped it silently.

Scenario 5 exercises the degradation ladder for real: tracestate fits 237 of the 335 characters x-datadog-tags carries, so W3CPTagsCodec sheds the two llmobs_pagent_* tags on inject and the consumer sees correct trace_id, parent_id, session_id and ml_app with agent_attribution: None — the order PTagsCodec documents.

Scenario 6 covers the concurrency case. A dispatcher agent span submits six workers to an executor, so the scope crosses the async boundary and all six land on one local trace; each worker opens its own agent span with its own name and session, and a CyclicBarrier releases them into their POSTs together. This is the shape a design that stored the tags on the span context gets wrong, since that state is shared by the whole local trace.

The discriminator has to be application data: a crossed injection swaps a whole coherent bundle, so a contaminated span's parent_id, session_id and agent_attribution would still agree with each other while all belonging to the wrong sibling, and trace_id matches either way because the siblings share it. Each worker therefore puts its identity in the request body, and check_fanout.py compares that against the propagated context, reading the raw intake payloads off the test agent. 24 of 24 downstream spans matched, across four rounds of six.

Claude session: 15543c2c-2abe-408e-b16e-05ddbe972287
Resume: claude --resume 15543c2c-2abe-408e-b16e-05ddbe972287

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.88 s	14.69 s	[+0.5%; +2.1%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.72 s	13.80 s	[-1.3%; +0.0%] (no difference)
startup:petclinic:appsec:Agent	17.00 s	16.88 s	[-0.3%; +1.8%] (no difference)
startup:petclinic:iast:Agent	17.02 s	17.01 s	[-0.9%; +1.1%] (no difference)
startup:petclinic:profiling:Agent	16.77 s	16.10 s	[-0.0%; +8.4%] (no difference)
startup:petclinic:sca:Agent	16.93 s	16.68 s	[+0.5%; +2.5%] (maybe worse)
startup:petclinic:tracing:Agent	16.18 s	15.68 s	[-1.0%; +7.3%] (no difference)

Commit: 529329ce · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[ncybul]

@codex

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 77546ab8f7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[dougqh]

Given that LLMObs is already GA, I unfortunately have to agree we cannot break compatibility.
That said the long names do significantly increase the risk that as we add products, we'll hit the header limit which could adversely impact all products.
I think in the future we should be more careful to vet such changes with the cross-language guild ahead of time.

[mhdatie]

We are already planning on switching to use baggage to propagate LLMObs-specific information across distributed contexts which will allow us to deprecate these tags in a future major release

@ncybul Do you know when this work is planned? Our concern is that support for tags would drag for many versions, so the sooner the better. Also, can this plan eventually go through #guild-cross-lang-platform for review? - you can sign up for a talk here.

[ncybul]

We are already planning on switching to use baggage to propagate LLMObs-specific information across distributed contexts which will allow us to deprecate these tags in a future major release

@ncybul Do you know when this work is planned? Our concern is that support for tags would drag for many versions. Also, can this plan eventually go through #guild-cross-lang-platform for review? - you can sign up for a talk here.

@mhdatie We've already started the work for this in other languages. I will be following up after this PR to introduce the change in Java, but we'll likely need to wait until a major release to deprecate the old format.

And yes, sounds good, a talk may be a good idea so everyone is on the same page. cc @Yun-Kim

[mhlidd]

LGTM for the initial implementation. A few notes:

1. Instrumentations aren't using LLMOBs traces at this stage yet - is that a planned follow-up PR? e.g. The OpenAI Integration auto-instrumentation falls back to the APM trace id instead of the LLMOBs trace id.
2. Would it be possible to have fallback cases in case the LLMOBs headers do cause the header to exceed limits? e.g. drop LLMOBs header to preserve APM headers, or something along those lines?

[ncybul]

LGTM for the initial implementation. A few notes:

1. Instrumentations aren't using LLMOBs traces at this stage yet - is that a planned follow-up PR? e.g. The OpenAI Integration auto-instrumentation falls back to the APM trace id instead of the LLMOBs trace id.
2. Would it be possible to have fallback cases in case the LLMOBs headers do cause the header to exceed limits? e.g. drop LLMOBs header to preserve APM headers, or something along those lines?

@mhlidd I listed the OpenAI auto-instrumentation path as a follow-up, but yes eventually we will support that. And we currently do have a strategy in this PR to drop agent attribution tags (parent agent name and parent agent ID) to help mitigate the risk of the tags exceeding the size cap. I think additional tags could be dropped as well; however, I'm leaning towards addressing that in a follow-up as this PR is already quite large.

[ValentinZakharov]

LGTM

[ncybul]

@codex

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1a7a34a1a4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[dd-octo-sts]

/merge

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-10-01 21:23:51 UTC ℹ️ Start processing command /merge

---

2026-10-01 21:23:56 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

---

2026-10-01 22:36:38 UTC ℹ️ MergeQueue: This merge request was merged