Repository navigation
feat(llmobs): emit gen_ai.* APM tags when LLM Observability is disabled - #10437
Conversation
The backend reads `gen_ai.*` tags on an APM span as user-set and builds a duplicate LLMObs span out of them, which is why #10261 was reverted. Write `_dd.llmobs.artificial_gen_ai_tags` alongside, matching dd-trace-py, so it can tell the two apart. Port of DataDog/dd-trace-py#20411. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Overall package sizeSelf size: 9.24 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.5.1 | 127.66 kB | 513.61 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!🧪 All tests passed 🔄 Datadog retried 2 tests - 2 passed on retry 🎯 Code Coverage (details) 🔗 Commit SHA: 142932a | Docs | View more details | Give us feedback! |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10437 +/- ##
==========================================
+ Coverage 98.77% 98.81% +0.03%
==========================================
Files 1047 1048 +1
Lines 166488 167498 +1010
Branches 74 74
==========================================
+ Hits 164446 165509 +1063
+ Misses 2042 1989 -53
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
BenchmarksBenchmark execution time: 2026-09-25 19:00:57 Comparing candidate commit 142932a in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2307 metrics, 6 unstable metrics.
|
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 97a5210796
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
3b6712b to
44c40de
Compare
LLMObs plugins used to disable themselves entirely when LLMObs was off, so the `gen_ai.*` APM attributes were only ever written for LLMObs-enabled applications. Keep the plugins subscribed instead: their handlers skip the LLMObs payload and emit only the APM attributes. The tagging moves out of `LLMObsSpanProcessor` into a shared `gen-ai-tags.js` so the plugins can reuse it, and the marker tag moves with it. Integrations resolve token usage and late model corrections through `getGenAiApmEndTags`. Covers anthropic, claude-agent-sdk, bedrock, google-genai, langchain, langgraph, openai, vertexai and both Vercel AI telemetry paths, each with a unit spec driving its diagnostic channels directly. Carries the enabled-path changes this path is built on: the Anthropic and LangChain usage helpers it reuses, the guards its Bedrock handler shares, and the Bedrock header-caching skip that keeps a response with no token counts from reporting zeros. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
44c40de to
1c3abd5
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
- Read streamed Bedrock usage through the shared provider-aware extractor instead of scanning for `amazon-bedrock-invocationMetrics`. Anthropic reports its counts on the message body and Amazon on plain body fields, so a streamed `invokeModel` on those models reported no usage at all. - Stop tagging `gen_ai.application.name`. ml_app is an LLM Observability concept, and with LLMObs off the only value left is the service name the span already carries. Matches dd-trace-py's reduced path. - Drop the in-process session fallback, whose only writer is the tagger and so is dead on this path. A propagated session is still inherited. - Apply the model and provider defaults when an integration promotes a span to a model-backed kind at the end of the operation, so a promotion can no longer leave an llm span with no model. - Cover the Model Context Protocol plugins, which inherit this path. - Drive the google-genai chunk channel in its spec rather than fabricating the field the chunk handler sets. - Document the tags in docs/API.md and both TypeScript surfaces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#10436 was squash-merged, so master now carries the gen_ai tagging inline in `LLMObsSpanProcessor`. Resolved in favour of this branch's refactor, which moves that same logic into `gen-ai-tags.js` for the plugins to share. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drop the reference section and keep the `llmobs` option paragraph as it was, noting only that the option also turns off the basic `gen_ai.*` tags. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8abfc53fc0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Two gaps left by the plugins staying subscribed when LLMObs is off: - The response plugin subscribes to `dd-trace:openai:realtime:audio` purely as a capability signal: the instrumentation retains a turn's audio only while something listens there. Staying subscribed with LLMObs off made it buffer megabytes per turn for a payload the reduced path never builds. Gate the subscription on LLMObs itself. - The turn's token usage is read only in `setLLMObsTags`, which the reduced path skips, so realtime spans reported no `gen_ai.usage.*`. Forward it through `getGenAiApmEndTags`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: da767d91d3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… path The query span learns its session only when the stream resolves, so it arrives through the end hook rather than the register options. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fb01fcb1de
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- The disabled Bedrock path retained every streamed frame until the response completed so the shared extractor could read the provider's token counts. Keep only the frames whose bytes carry one of the fields that extractor reads, decided with a byte search over the raw frame, so generated text is dropped as it arrives. - The Realtime audio capability subscription was decided once, in the constructor. `LLMObs.enable()` and `disable()` flip `DD_LLMOBS_ENABLED` at runtime without reconfiguring plugins, so enabling later left Realtime events without audio. Re-evaluate it on configure and per turn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5375d09833
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Realtime is the one integration where staying subscribed costs more than the tags are worth: the instrumentation retains a turn's audio only while these plugins are subscribed, and only the LLM Observability payload reads those bytes. Wiring it into the reduced path meant either buffering megabytes for no consumer or reaching into the plugin's subscription lifecycle, and the latter broke the enabled path's audio capture. Restore `realtime.js` to its original behavior and give `LLMObsPlugin` a `emitsGenAiApmTags` opt-out, which realtime sets to `false`: with LLM Observability off it is disabled outright, exactly as before these tags existed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The byte search matched bare words, so a streamed response whose generated text mentioned usage or TokenCount kept those frames until the response completed. Quote the markers: a quote inside a JSON string arrives escaped, so prose cannot produce the byte sequence a field name does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ncybul
left a comment
ncybul
left a comment
There was a problem hiding this comment.
Looking good so far. I was trying to think of ways we could simplify this logic (for example, creating an llmobs payload even when llmobs is disabled and then using that to extract tokens), but that has its own tradeoffs to consider. Luckily, it seems like most of the logic here is tests and extracting accurate token counts which should keep this relatively low risk.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7b2c66c338
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The reduced gen_ai path decided which streamed frames to retain with a byte search for four hand-written field names, then re-read them through the LLMObs stream extractor at `:complete:`. The list was maintained by hand against a switch in another file, so a renamed or unlisted field would have dropped every frame and emitted a span with no token counts and no error. Fold the counts into a running total as each frame arrives instead, the way the anthropic plugin and Bedrock's own Converse path already do. `mergeStreamedUsage` holds the per-provider field table and is shared with the stream extractor, so both paths read usage from one definition and nothing is retained past four numbers. That also fixes an overwrite in the extractor: Amazon reports its counts on a frame that carries text, and a later text-only frame reset them to undefined. `buildUsage` now returns undefined when a Converse response reports no counts, and `extractTokens` leaves an unreported count undefined rather than coercing it to zero, so the APM attributes omit what nothing measured. The LLMObs metrics keep reporting zero through `zeroFilled`. Also drops the dead `marked` bookkeeping in `setGenAiApmUsageMetrics`, which could not run unmarked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The spec subscribed its plugin instances to the production prefixes, so its publishes shared diagnostic channels with the real tracer and with any other spec in the same process. Subclass both plugins with a test-only prefix, the way the ai and langgraph gen_ai specs already do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1b5a7d6f67
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Anthropic streams report the final output count on the closing `message_delta`, at the top level rather than under `message`, so reading only `message.usage` kept the provisional count from `message_start` whenever the trailing invocation metrics were absent. Read both locations. A non-streamed `invokeModel` whose response carried no correlatable token-count headers reported no usage at all, although AI21, Amazon and Meta bodies expose the counts and the enabled path already reads them. Fall back to the same extractor when the headers reported nothing, leaving the body unread otherwise. Both reported by codex on #10437. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`_llmobsEnabled` reads the live config flag, which `llmobs.enable()` and `llmobs.disable()` mutate while operations are in flight. An operation that started on the reduced path and finished after an enable asked the tagger to tag a span its start never registered, warning on every call, and skipped `getGenAiApmEndTags`, losing the model and token counts the response carries. The reverse order left the LLMObs parent store unrestored. Latch the mode on the operation context at first read and have every hook for that operation consult it, so start, end, asyncEnd and the streamed-chunk subscribers all take the branch the start took. `configure` keeps reading the live flag: subscription is a process-level decision, not a per-operation one. Reported by codex on #10437. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8195a3cbc3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A non-streamed `invokeModel` can report some counts in its headers and the rest in its body, so gating the body read on the headers reporting nothing dropped the remainder. Read both and let `extractTokens` merge them field by field, as it already does per field. `resolveCount` also picked its source with `||`, which discarded a body-reported zero and let a header override the body it documents as authoritative. Select on presence instead, so a measured zero stays a measurement. The plugin-base toggle spec drove a start with LLMObs enabled, which enters the LLMObs storage; with no matching end the fake span leaked into every later test in the process and reparented the SDK spans. Start it without register options so no span is registered and the storage is untouched. Reported by codex on #10437. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 142932a811
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
wconti27
left a comment
wconti27
left a comment
There was a problem hiding this comment.
Change LGTM from an IDM perspective
3 participants
What does this PR do?
#10436 emits the scalar
gen_ai.*APM attributes fromLLMObsSpanProcessor, which only runs when LLM Observability is enabled. This extends them to applications running with LLMObs off.LLMObs plugins used to disable themselves entirely in that case. They now stay subscribed: the handlers skip the LLMObs payload and emit only the APM attributes. The tagging moves out of
LLMObsSpanProcessorinto a sharedgen-ai-tags.jsso the plugins can reuse it, and the marker tag moves with it. Integrations resolve token usage and late model corrections through a newgetGenAiApmEndTagshook.Covers anthropic, claude-agent-sdk, bedrock, genai, langchain, langgraph, openai, vertexai and Vercel AI.
Motivation
Model, provider and token usage are worth having in APM whether or not an application sends the LLMObs track. Without this, every
gen_ai.*tag depends on LLMObs being enabled.Additional Notes
Also fixes the langchain provider lookup:
isLLMIntegrationEnabledread a plugin manager captured when the module first loaded, so a tracer rebuilt afterwards left it pointing at a manager with no plugins and the LangChain model span was never demoted toworkflow, double-counting tokens against the provider span. It now reads off the tracer that owns the plugin. Pre-existing, but this PR's specs change module load order and make it fire deterministically.Testing
tested for both streaming and non streaming calls.
when LLMObs is disabled, I have verified that integration has traces which are queryable
🤖 Generated with Claude Code