Add annotateAgentManifest manual API to LLMObs SDK by yahya-mouman · Pull Request #12318 · DataDog/dd-trace-java

yahya-mouman · 2026-08-27T14:01:13Z

Summary

Adds annotateAgentManifest(LLMObs.AgentManifest) to the Java LLMObs SDK, allowing users to manually declare an agent span's configuration (name, instructions, model, model_settings, tools). Mirrors the Python implementation (DataDog/dd-trace-py#19771) and follows the annotatePrompt pattern from #12161.

Changes

LLMObsTags.java — adds AGENT_MANIFEST = "agent_manifest" constant
LLMObs.java — adds two public immutable builder classes:
- LLMObs.AgentTool — represents a single tool (name, optional description, optional parameters)
- LLMObs.AgentManifest — builder with name, instructions, model, model_settings, tools
LLMObsSpan.java — adds default void annotateAgentManifest(LLMObs.AgentManifest) (no-op default for backwards compat)
NoOpLLMObsSpan.java — explicit @Override no-op
DDLLMObsSpan.java — real implementation: validates span kind (agent only), builds manifest map, stores as _ml_obs_tag.agent_manifest
LLMObsSpanMapper.java — adds agent_manifest to TAGS_FOR_REMAPPING; serializes to meta.agent_manifest as a msgpack map

Behaviour

LLMObs.AgentManifest manifest = LLMObs.AgentManifest.builder()
    .name("travel_desk")
    .instructions("Book travel for the user.")
    .model("gpt-4o")
    .modelSettings(Map.of("temperature", 0.7))
    .tools(List.of(LLMObs.AgentTool.from("get_weather", "Look up weather", null)))
    .build();

agentSpan.annotateAgentManifest(manifest);

Only applies to agent span kind; other span kinds emit a log warning and no-op
Calling twice on the same span overwrites the previous manifest (no merge)
framework is set to "AgentObs SDK" automatically by the SDK
name defaults to the span name if not provided
model_settings keys are forwarded as-is (no allowlist in this initial PR)
Null manifest is silently ignored

Test plan

./gradlew :dd-trace-api:test --tests "datadog.trace.api.llmobs.LLMObsTest" — 27 tests pass (4 new)
./gradlew :dd-java-agent:agent-llmobs:test --tests "datadog.trace.llmobs.domain.DDLLMObsSpanTest" — 39 tests pass (7 new)
./gradlew :dd-trace-core:test --tests "datadog.trace.llmobs.writer.ddintake.LLMObsSpanMapperTest" — 18 tests pass (2 new)

🤖 Generated with Claude Code

- Add AgentManifest immutable value class with Builder pattern in LLMObs.java - Add AgentTool immutable value class in LLMObs.java - Add AGENT_MANIFEST constant to LLMObsTags - Add annotateAgentManifest() default method to LLMObsSpan interface - Add comprehensive builder tests for AgentManifest and AgentTool Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Stores agent manifest fields (name, instructions, model, model_settings, tools) as an internal tag `_ml_obs_tag.agent_manifest`. Only applies to agent spans; warns and no-ops on other span kinds. Null manifest is silently ignored. Tools with null/empty names are skipped with a warning. A second call overwrites the previous manifest. Framework field "AgentObs SDK" is always added when any manifest fields are present. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add AGENT_MANIFEST_KEY byte constant, include the tag in TAGS_FOR_REMAPPING, and handle it in the meta serialization loop as a msgpack map. Add two tests: one verifying all manifest fields appear in meta, one verifying the tag does not leak into the tags list. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

ncybul · 2026-08-27T16:51:14Z

@codex

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b5f56f11d6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

ncybul

A couple minor suggestions but overall looks good to me! Would be nice to see a manual verification of setting the manifest via the SDK.

dd-octo-sts · 2026-08-28T11:08:53Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

datadog-prod-us1-6

More details

The default API method keeps compatibility. The implementation accepts manifests only for agent spans and writes them to meta.agent_manifest.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit b5f56f1 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

…test hardening - Change annotateAgentManifest to merge successive calls instead of overwriting: name/instructions/model use new-value-wins logic with fallbacks; model_settings shallow-merges; tools replace only when caller provides a non-empty list. - Change MANUAL_FRAMEWORK constant from "AgentObs SDK" to "manual". - Update all test assertions that expected "AgentObs SDK" to "manual". - Replace overwrite test with merge-semantics test; add model_settings merge test. - Tests for post-finish no-op and empty-tools-list were already present. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

sabrenner

one main comment a couple nits otherwise lgtm!

…domain/DDLLMObsSpan.java Co-authored-by: Sam Brenner <106700075+sabrenner@users.noreply.github.com>

…nce; update docstring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

dd-octo-sts · 2026-08-31T10:59:47Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-08-31T10:59:51Z

View all feedbacks in Devflow UI.

2026-08-31 10:59:50 UTC ℹ️ Start processing command /merge

2026-08-31 10:59:55 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-08-31 11:47:18 UTC ❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for 5fa0942:

test_inst: [zulu8, 8/8]

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

Investigate the failures and when ready, re-add your pull request to the queue!
If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
Any question, go check the FAQ.

Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

yahya-mouman · 2026-08-31T13:21:43Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-08-31T13:21:47Z

View all feedbacks in Devflow UI.

2026-08-31 13:21:47 UTC ℹ️ Start processing command /merge

2026-08-31 13:21:51 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-08-31 14:25:17 UTC ℹ️ MergeQueue: This merge request was merged

dd-octo-sts · 2026-08-31T14:01:15Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-08-31T14:01:19Z

View all feedbacks in Devflow UI.

2026-08-31 14:01:18 UTC ℹ️ Start processing command /merge

2026-08-31 14:01:21 UTC ❌ MergeQueue

PR already in the queue with status in_progress

## Description `LLMObs.annotate(agent=...)` and `annotation_context(agent=...)` accepted a mapping but only read `version`, so a manually declared agent rendered an empty manifest panel. This adds validated support for `name`, `instructions`, `model`, `model_settings` and `tools`. All of these already exist in `AgentManifest`, so there is no wire schema change. ```python LLMObs.annotate(agent={ "version": "2.1.0", "name": "travel_desk", "instructions": "Book travel.", "model": "gpt-4o", "model_settings": {"temperature": 0.1, "max_tokens": 1024}, "tools": [{"name": "get_weather", "description": "Look up the weather.", "parameters": {"city": {"type": "string", "required": True}}}], }) ``` Behavior: - The manifest is emitted on agent spans only, under `_dd.agent_manifest`. `version` stays an `agent_version` tag. A version-only agent emits no manifest. - `framework` is set to `"manual"` by the SDK. - `name` defaults to the agent span's name; a declared `name` overrides it. - An `annotation_context` declaration reaches every agent span in its block, as `agent_version` already does. `annotate(span=...)` declares whatever span it names. - Each declaration is validated, then shallow-updates the manifest key by key, so repeated `annotate()` calls and nested `annotation_context` blocks compose. A value that is unset (`None`, `""`, `[]`, `{}`) or unreportable declares nothing rather than erasing an earlier declaration. - Declared fields merge into an integration-built manifest instead of replacing it. The integration's `framework` is kept. - Input is validated per field and bad values are dropped. `model_settings` is limited to `ALLOWED_MODEL_SETTINGS_KEYS`, so provider-specific keys such as `extra_headers` (which can carry secrets) are not reported. Tool `parameters` accept the documented `{param: {type, required}}` mapping or a JSON Schema object, flattened to the same shape the integrations emit, with `required` omitted when false. - The builder never raises, so a malformed field cannot drop the span event. Java equivalent: DataDog/dd-trace-java#12318. ## Testing Unit tests in `tests/llmobs/test_integrations_utils.py` (builder validation, JSON Schema parameters) and `tests/llmobs/test_llmobs_service.py` (agent-only emission, outermost-span scoping, merge semantics including `model_settings`, unset values, name fallback, malformed input). `pydantic_ai`, `crewai` and `openai_agents` suites cover the integrations that share the manifest path. Traces: [baggage agent (nested manifest)](https://app.datadoghq.com/llm/traces?query=%40ml_app%3Amax-test%20%40event_type%3Aspan%20%40is_root_span%3Atrue&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&is_llm_session=false&refresh_mode=sliding&selectedTab=overview&sidepanelContextScopeKind=agent&sp=%5B%7B%22p%22%3A%7B%22eventId%22%3A%22AwAAAaDUVLWsIKRuAQAAABhBYURVVkxXc0FBQzJfOG5oNW1wNUFBQUEAAAAkZjFhMGQ0NTgtMjU0OC00YmViLWEyNDAtMzg1ZTliZGIxNTVhAADofg%22%7D%2C%22i%22%3A%22llm-obs-panel%22%7D%5D&spanId=18175047169571987787&start=1790265508007&end=1790269108007&paused=false) [simple travel desk agent](https://app.datadoghq.com/llm/traces?query=%40ml_app%3Amax-test%20%40event_type%3Aspan%20%40is_root_span%3Atrue&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&is_llm_session=false&refresh_mode=sliding&selectedTab=overview&sidepanelContextScope=17920727384912341806&sidepanelContextScopeKind=agent&sp=%5B%7B%22p%22%3A%7B%22eventId%22%3A%22AwAAAaDUVLWsWXlvDQAAABhBYURVVkxXc0FBQnV2Y2kyN08xY0FBQUEAAAAkZjFhMGQ0NTgtMjU0OC00YmViLWEyNDAtMzg1ZTliZGIxNTVhAADofw%22%7D%2C%22i%22%3A%22llm-obs-panel%22%7D%5D&spanId=17920727384912341806&start=1790265508007&end=1790269108007&paused=false) ## Risks Low. Additive to an existing argument; version-only usage is unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: mz1119 <max.zhang@datadoghq.com>

yahya-mouman and others added 4 commits August 27, 2026 13:36

feat(llmobs): add annotateAgentManifest default method to LLMObsSpan

ff0f77c

yahya-mouman added the type: feature Enhancements and improvements label Aug 27, 2026

yahya-mouman requested review from ncybul and sabrenner August 27, 2026 14:15

This comment has been minimized.

Sign in to view

chatgpt-codex-connector Bot reviewed Aug 27, 2026

View reviewed changes

Comment thread dd-java-agent/agent-llmobs/src/main/java/datadog/trace/llmobs/domain/DDLLMObsSpan.java Outdated

ncybul approved these changes Aug 27, 2026

View reviewed changes

yahya-mouman marked this pull request as ready for review August 28, 2026 11:08

yahya-mouman requested a review from a team as a code owner August 28, 2026 11:08

dd-octo-sts Bot added the tag: ai generated Largely based on code generated by an AI or LLM label Aug 28, 2026

datadog-prod-us1-6 Bot reviewed Aug 28, 2026

View reviewed changes

yahya-mouman mentioned this pull request Aug 28, 2026

feat(llmobs): accept an agent manifest manually DataDog/dd-trace-py#19771

Merged

sabrenner reviewed Aug 28, 2026

View reviewed changes

Comment thread dd-java-agent/agent-llmobs/src/main/java/datadog/trace/llmobs/domain/DDLLMObsSpan.java Outdated

yahya-mouman and others added 2 commits August 28, 2026 19:43

Update dd-java-agent/agent-llmobs/src/main/java/datadog/trace/llmobs/…

a0e864c

…domain/DDLLMObsSpan.java Co-authored-by: Sam Brenner <106700075+sabrenner@users.noreply.github.com>

Extract manifest getter calls to local variables; fix spanName refere…

df4a559

…nce; update docstring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

sabrenner approved these changes Aug 28, 2026

View reviewed changes

yahya-mouman added the comp: mlobs ML Observability (LLMObs) label Aug 31, 2026

yahya-mouman added this pull request to the merge queue Aug 31, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 31, 2026

yahya-mouman added this pull request to the merge queue Aug 31, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 31, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit e149937 into master Aug 31, 2026
604 of 605 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the yahya/llmobs-agent-manifest-manual branch August 31, 2026 14:25

github-actions Bot added this to the 1.66.0 milestone Aug 31, 2026

daniel-mohedano changed the title ~~feat(llmobs): add annotateAgentManifest manual API to LLMObs SDK~~ Add annotateAgentManifest manual API to LLMObs SDK Sep 8, 2026

mz1119 mentioned this pull request Sep 24, 2026

feat(llmobs): accept an agent manifest manually DataDog/dd-trace-js#10480

Merged

mz1119 mentioned this pull request Sep 28, 2026

feat(llmobs): accept an agent version manually DataDog/dd-trace-js#10536

Open

Conversation

yahya-mouman commented Aug 27, 2026

Summary

Changes

Behaviour

Test plan

Uh oh!

This comment has been minimized.

ncybul commented Aug 27, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

ncybul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dd-octo-sts Bot commented Aug 28, 2026

Uh oh!

datadog-prod-us1-6 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

sabrenner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dd-octo-sts Bot commented Aug 31, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What to do next?

Uh oh!

Uh oh!

yahya-mouman commented Aug 31, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Aug 31, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gh-worker-devflow-routing-ef8351 Bot commented Aug 31, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Aug 31, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Aug 31, 2026 •

edited

Loading