Repository navigation
feat(llmobs): accept an agent manifest manually - #10480
Conversation
Overall package sizeSelf size: 9.46 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.5.1 | 127.66 kB | 531.94 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10480 +/- ##
==========================================
+ Coverage 98.84% 98.87% +0.03%
==========================================
Files 1043 1059 +16
Lines 167818 172270 +4452
Branches 74 74
==========================================
+ Hits 165873 170332 +4459
+ Misses 1945 1938 -7
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
✅ All CI checks and tests passed. Datadog automation helped this PR pass. 🎉 All green!🧪 All tests passed 🔄 Datadog retried 2 tests - 2 passed on retry 🎯 Code Coverage (details) 🔗 Commit SHA: 1378ecf | Docs | View more details | Give us feedback! |
BenchmarksBenchmark execution time: 2026-10-08 15:43:23 Comparing candidate commit 1378ecf in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2304 metrics, 9 unstable metrics.
|
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0e13f295b6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b82e8a490
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: caaf6c0cfe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 744541f996
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ccaba8c139
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Adds an `agent` option to `llmobs.annotate()` and `llmobs.annotationContext()` that reports `version` as an `agent_version` tag and `name`, `instructions`, `model`, `modelSettings` and `tools` as the agent's manifest on agent spans. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Keep context declarations outside the caller-controlled annotationContext options, so a caller key named `agents` can no longer crash span registration or bypass validation. - Snapshot each declaration once when the context is entered, never throwing on unreadable caller objects. - Give each context manifest to the first agent span of an LLMObs trace, so a sibling handoff target keeps its own agent. - Resolve `agent_version` at finish so spans promoted to agents after registration get it, and let it win over a user tag, matching dd-trace-py. - Treat '' settings as unset, accept only plain objects, drop the `model_settings` alias, and log dropped fields by name. - Copy context tags on first use so annotations cannot write into them. - Tighten the public Agent types. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Copying context tags at span registration read caller getters and Proxy traps, so an unreadable tags object threw out of span creation (and disabled integrations) where it was previously contained at finish. Drop and warn instead. Widen the public `Agent.version` type to accept the numbers the runtime already reports as strings. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Match dd-trace-py (DataDog/dd-trace-py#19771, 915ac6b46): each validated declaration shallow-updates the manifest, so a later `model_settings` replaces an earlier one whole, and an annotation context declares every agent span in its block instead of the first per trace. Context declarations are now recorded on every span in the block and emitted if the span is an agent at finish, so spans promoted to agents after registration get the manifest too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A schema with `properties` but neither `type` nor `required` is read as a parameter map at runtime (matching dd-trace-py), so its parameters were silently dropped while the public type accepted it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The public type accepts numeric versions and reports them as strings, but a truthiness check dropped `0`. Only `undefined`, `null`, `''` and non-finite numbers now declare no version. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Keeps this change to the agent manifest. The `agent_version` tag lands separately. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Tool parameters and `logit_bias` are built with `Object.fromEntries`, so a key named `__proto__` from a JSON-parsed schema becomes an own property instead of setting the prototype and being dropped. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
7c81b89 to
813fe04
Compare
|
Hi @mz1119 is this PR ready for human review? if not please consider converting the PR back to draft :) |
ncybul
left a comment
ncybul
left a comment
There was a problem hiding this comment.
I still need to take a look at the tests, but leaving some initial comments here!
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 participants
What does this PR do?
Adds an
agentoption tollmobs.annotate()andllmobs.annotationContext().name,instructions,model,modelSettingsandtoolsare reported as the agent's manifest undermeta.metadata._dd.agent_manifest.Behavior:
frameworkis set tomanual, andnamedefaults to the span name.annotationContextdeclaration is read once when the context is entered and reaches every agent span in the block, like every other annotation.annotateon a nested agent span overrides it for that span.modelSettingsandtoolsincluded) replaces an earlier one whole, and unset or invalid values (undefined,null,'',[]) leave it in place.modelSettingsis limited to an allowlist, so keys likeextra_headersare never reported. Only plain objects are read, so schema-library objects must be converted to JSON Schema first. Toolparametersaccept{ param: { type, required } }or a JSON Schema object.Also fixes a pre-existing bug where
annotate({ tags })insideannotationContext({ tags })wrote into the context's tags object.Testing
Example traces:
trace with nested manifest
simple trace
Motivation
Parity with dd-trace-py (DataDog/dd-trace-py#19771) and dd-trace-java (DataDog/dd-trace-java#12318), so manually instrumented agents render a manifest.
Additional Notes
agent.version(theagent_versiontag) follows in feat(llmobs): accept an agent version manually #10536.meta.metadata._dd.agent_manifest, moved there in chore(llmobs): move agent manifest to _dd field dd-trace-py#16621). The Java PR writesmeta.agent_manifest.modelSettingsaccepts camelCase keys (maxTokens) and reports them in snake_case.🤖 Generated with Claude Code