feat(llmobs): accept an agent version manually by mz1119 · Pull Request #10536 · DataDog/dd-trace-js

mz1119 · 2026-09-28T20:47:18Z

What does this PR do?

Adds an agent version, reported as an agent_version tag on the agent span and its child spans in the same process.

llmobs.trace({ kind: 'agent', name: 'travel_desk', version: '2.1.0' }, () => {})
llmobs.annotationContext({ agent: { version: '2.1.0' } }, () => {})
llmobs.annotate({ agent: { version: '2.1.0' } })

Strings only. Each span copies its nearest agent's version when it starts. A nested agent without its own version, and its children, carry the outer agent's version, matching dd-trace-java. A version changed after a child span starts is not guaranteed to reach it.

Motivation

The agent panel attributes tokens and cost to a version through the agent's child spans. Matches dd-trace-py (DataDog/dd-trace-py#20693) and dd-trace-java (DataDog/dd-trace-java#12291).

Additional Notes

No cross-service propagation.

🤖 Generated with Claude Code

dd-octo-sts · 2026-09-28T20:48:05Z

Overall package size

Self size: 9.46 MB
Deduped: 10.21 MB
No deduping: 10.21 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.5.1 | 127.66 kB | 531.94 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

datadog-official · 2026-09-28T20:50:31Z

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 3 Pipeline jobs failed

LLMObs | claude-agent-sdk — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

All Green | all-green

View more details · View in GitHub Actions

AppSec | AppSec / next (node-oldest, 13.2.0)

View more details · View in GitHub Actions

⚠️ Warnings

🧪 2 Tests failed in 1 job

LLMObs | claude-agent-sdk — ❌ 2 tests failed

View more details · View in GitHub Actions

❌ Plugin with @anthropic-ai/claude-agent-sdk >=0.2.113 (0.3.289) captures a preset with appended instructions from with @anthropic-ai/claude-agent-sdk >=0.2.113 (0.3.289)

Timeout of 15000ms exceeded. For async tests and hooks, ensure &#34;done()&#34; is called; if returning a Promise, ensure it resolves. (/home/runner/work/dd-trace-js/dd-trace-js/packages/dd-trace/test/llmobs/plugins/claude-agent-sdk/index.spec.js)

Error: Timeout of 15000ms exceeded. For async tests and hooks, ensure &#34;done()&#34; is called; if returning a Promise, ensure it resolves. (/home/runner/work/dd-trace-js/dd-trace-js/packages/dd-trace/test/llmobs/plugins/claude-agent-sdk/index.spec.js)
    at listOnTimeout (node:internal/timers:605:17)
    at process.processTimers (node:internal/timers:541:7)

❌ Plugin with @anthropic-ai/claude-agent-sdk >=0.2.113 (0.3.289) captures a preset without appended instructions from with @anthropic-ai/claude-agent-sdk >=0.2.113 (0.3.289)

Timeout of 15000ms exceeded. For async tests and hooks, ensure &#34;done()&#34; is called; if returning a Promise, ensure it resolves. (/home/runner/work/dd-trace-js/dd-trace-js/packages/dd-trace/test/llmobs/plugins/claude-agent-sdk/index.spec.js)

Error: Timeout of 15000ms exceeded. For async tests and hooks, ensure &#34;done()&#34; is called; if returning a Promise, ensure it resolves. (/home/runner/work/dd-trace-js/dd-trace-js/packages/dd-trace/test/llmobs/plugins/claude-agent-sdk/index.spec.js)
    at listOnTimeout (node:internal/timers:605:17)
    at process.processTimers (node:internal/timers:541:7)

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog retried 2 tests - 2 passed on retry

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 84.39% (-1.95%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 76c20c4 | Docs | View more details | Give us feedback!}

codecov · 2026-09-28T20:51:17Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.42%. Comparing base (80f6ca5) to head (76c20c4).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10536      +/-   ##
==========================================
- Coverage   86.38%   84.42%   -1.96%     
==========================================
  Files        1043     1048       +5     
  Lines      167050   167336     +286     
  Branches       74       59      -15     
==========================================
- Hits       144298   141279    -3019     
- Misses      22752    26057    +3305

Flag	Coverage Δ
ai-guard	`63.58% <36.66%> (-0.04%)`	⬇️
apm-capabilities	`54.69% <36.66%> (-0.01%)`	⬇️
apm-integrations	`80.76% <36.66%> (-0.04%)`	⬇️
appsec	`?`
debugger	`69.63% <36.66%> (-0.05%)`	⬇️
instrumentation	`55.30% <36.66%> (-0.02%)`	⬇️
llmobs	`80.10% <100.00%> (+<0.01%)`	⬆️
master-coverage	`84.42% <100.00%> (?)`
openfeature	`67.10% <36.66%> (-0.05%)`	⬇️
platform	`68.34% <36.66%> (-0.03%)`	⬇️
profiling	`65.75% <36.66%> (-0.05%)`	⬇️
serverless	`65.42% <36.66%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pr-commenter · 2026-09-28T21:05:05Z

Benchmarks

Benchmark execution time: 2026-10-09 18:44:17

Comparing candidate commit bc291ca in PR branch max.zhang/llmobs-agent-version with baseline commit 80f6ca5 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2394 metrics, 9 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:debugger-line-probe-without-snapshot-26

unstable cpu_user_time [-1203.492ms; +1806.401ms] or [-12.170%; +18.267%]
unstable execution_time [-1234.150ms; +1841.399ms] or [-11.180%; +16.682%]
unstable throughput [-183.799op/s; +125.946op/s] or [-17.694%; +12.124%]

scenario:encoders-0.4-immediate-flush-20

unstable max_rss_usage [-2.135MB; +7.000MB] or [-2.517%; +8.253%]

scenario:openfeature-scale-full-24

unstable max_rss_usage [-13.150MB; +48.270MB] or [-4.551%; +16.704%]

scenario:openfeature-typical-full-20

unstable max_rss_usage [-7.639MB; +11.594MB] or [-5.068%; +7.692%]

scenario:plugin-claude-agent-sdk-compact-stream-scan-24

unstable cpu_usage_percentage [-6.611%; +4.244%]
unstable execution_time [-382.044ms; +624.899ms] or [-5.122%; +8.379%]
unstable throughput [-81001.264op/s; +49236.215op/s] or [-7.517%; +4.569%]

chatgpt-codex-connector · 2026-10-05T20:20:14Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-08T22:51:22.820415Z	`c8e518e`	New commits
🔒 Security Review	✅ Completed 2026-10-08T22:51:43.905814Z	`c8e518e`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d834a4ebf7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e7c745bf1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-06T19:40:49Z

+    const agentVersion = mlObsTags?.[SPAN_KIND] === 'agent'
+      ? mlObsTags[AGENT_VERSION]
+      : LLMObsTagger.tagMap.get(mlObsTags?.[PARENT_AGENT])?.[AGENT_VERSION]
+    if (agentVersion) tags[AGENT_VERSION_TAG_KEY] = agentVersion


Reject versions that cannot survive tag serialization

When an agent version contains a comma (for example, release,canary), this copies it verbatim into the tag map, and #objectTagsToStringArrayTags later emits agent_version:release,canary. That serializer explicitly documents commas as intake-side delimiters, so the value is split into agent_version:release plus an orphan tag instead of being reported as the supplied version. Validate or safely encode agent versions before adding this tag.

Useful? React with 👍 / 👎.

Adds `version` to the `agent` annotation option. It is reported as an `agent_version` tag on agent spans, resolved when the span finishes, and wins over a user tag of the same name. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Adds `version` to the `trace`, `wrap` and `decorate` options, matching `LLMObs.agent(version=...)` and `@agent(version=...)` in dd-trace-py. It applies to agent spans only, is kept out of the APM span options, and wins over a version declared by an enclosing annotation context. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

A numeric version lost its formatting when converted (1.10 became 1.1, 2.0 became 2), merging distinct versions. dd-trace-py and dd-trace-java accept strings only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

The agent panel attributes tokens and cost to a version by reading agent_version on an agent's llm and embedding spans, matched to the agent by pagent_name. Each span now reports the version of the agent it is attributed to, resolved with the attribution at registration, matching dd-trace-java. A nested agent without a version does not inherit its ancestor's, since its spans are attributed to it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…a non-string one Spans cached their agent's version at registration, so a version annotated after an intermediate span started was missed or stale on the spans under it. Each span now keeps a reference to its agent and reads the agent's version when it finishes. A set but non-string version is now dropped with a warning, as the Agent type documents. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…race-java Each span copies its nearest agent's version when it starts. An agent without its own version reports the one it inherited, so an unversioned nested agent and its children carry the outer agent's version. A version changed after a child starts is not guaranteed to reach it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

ncybul · 2026-10-09T18:24:12Z

        })
      }

+      it('emits the declared version over a user tag of the same name, and leaves the tag on other spans', () => {


This kinda seems like a weird behavior? I would expect the declared version to win even for the descendant spans. Wdyt?

Co-authored-by: ncybul <124532568+ncybul@users.noreply.github.com>

mz1119 mentioned this pull request Sep 28, 2026

feat(llmobs): accept an agent manifest manually #10480

Merged

mz1119 added the semver-minor label Sep 28, 2026

mz1119 force-pushed the max.zhang/llmobs-agent-manifest branch from 7c81b89 to 813fe04 Compare September 29, 2026 18:51

mz1119 force-pushed the max.zhang/llmobs-agent-version branch from 567ec65 to 3017732 Compare September 29, 2026 18:51

mz1119 mentioned this pull request Sep 30, 2026

feat(llmobs): propagate agent_version to the agent's subtree DataDog/dd-trace-py#20693

Open

mz1119 marked this pull request as ready for review October 5, 2026 20:19

mz1119 requested review from a team as code owners October 5, 2026 20:20

mz1119 requested review from BridgeAR and bojbrook and removed request for a team October 5, 2026 20:20

chatgpt-codex-connector Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread packages/dd-trace/src/llmobs/util.js

Comment thread packages/dd-trace/src/llmobs/agent-manifest.js Outdated

chatgpt-codex-connector Bot reviewed Oct 6, 2026

View reviewed changes

Base automatically changed from max.zhang/llmobs-agent-manifest to master October 8, 2026 17:24

mz1119 and others added 6 commits October 8, 2026 13:28

fix(llmobs): accept only string agent versions

823d394

A numeric version lost its formatting when converted (1.10 became 1.1, 2.0 became 2), merging distinct versions. dd-trace-py and dd-trace-java accept strings only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

test(llmobs): cover the agent version on integration llm and tool spans

5533205

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

mz1119 force-pushed the max.zhang/llmobs-agent-version branch from e7c745b to 0ae80a1 Compare October 8, 2026 17:30

ncybul reviewed Oct 9, 2026

View reviewed changes

Comment thread packages/dd-trace/src/llmobs/span_processor.js Outdated

Comment thread packages/dd-trace/src/llmobs/tagger.js Outdated

ncybul reviewed Oct 9, 2026

View reviewed changes

mz1119 and others added 2 commits October 9, 2026 14:25

Update packages/dd-trace/src/llmobs/span_processor.js

bc291ca

Co-authored-by: ncybul <124532568+ncybul@users.noreply.github.com>

Update packages/dd-trace/src/llmobs/tagger.js

76c20c4

Co-authored-by: ncybul <124532568+ncybul@users.noreply.github.com>

Conversation

mz1119 commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

Uh oh!

dd-octo-sts Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Uh oh!

datadog-official Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

⚠️ Warnings

ℹ️ Info

Uh oh!

codecov Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pr-commenter Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

Unstable benchmarks

scenario:debugger-line-probe-without-snapshot-26

scenario:encoders-0.4-immediate-flush-20

scenario:openfeature-scale-full-24

scenario:openfeature-typical-full-20

scenario:plugin-claude-agent-sdk-compact-stream-scan-24

Uh oh!

chatgpt-codex-connector Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ncybul Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mz1119 commented Sep 28, 2026 •

edited

Loading

dd-octo-sts Bot commented Sep 28, 2026 •

edited

Loading

datadog-official Bot commented Sep 28, 2026 •

edited

Loading

codecov Bot commented Sep 28, 2026 •

edited

Loading

pr-commenter Bot commented Sep 28, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 5, 2026 •

edited

Loading