Keep span events structured until serialization by PerfectSlayer · Pull Request #12768 · DataDog/dd-trace-java

PerfectSlayer · 2026-10-07T09:24:33Z

Note

WIP - EARLY DRAFT - NOT READY FOR REVIEW

What Does This Do

This PR addresses my feedback from #11855:

a single event collection per span instead of one in OtelSpan and one in DDSpan;
no v0 encoding logic in the OTel classes;
no core implementation details in internal-api.

About implementation details:

Stores span events as DDSpanEvent objects on DDSpan, created in core through two new AgentSpan methods: addEvent(name, attributes) and addEvent(name, attributes, timestamp, unit). Previously the OTel shim JSON-encoded them into the events tag when the span finished. internal-api no longer has an event type.
Encodes the events tag in core (DDSpanContext) at serialization time, next to the span links tag. This covers the tag-based protocols and writers: v0.4/v0.5, OTLP, CI Visibility, LLM Obs and file-based.
Writes structured events directly in v1 payloads (span field 12) so TraceMapperV1 no longer parses JSON.
Events without an explicit time are now timestamped with the tracer clock, the same one used for span start and end. The OTel shim previously used a static SystemTimeSource, which has only millisecond precision.
All JSON encoding code is removed from the OTel shim.

Motivation

Events were JSON-encoded on the application thread for every protocol. v1 then parsed that JSON back to write native events: encode → decode → re-encode.
This also moves work off the application thread: attributes are copied into a map instead of being JSON-encoded when an event is added, and no tag is built when the span finishes.

Additional Notes

Behaviour changes:

The events tag is only added when the span is serialized, so trace interceptors no longer see it. This already applies to _dd.span_links.

Fixes:

Event names are now escaped in the events tag. A name containing ", \ or a control character used to produce invalid JSON, or to be read as a different name. Names without those characters are encoded exactly as before.
Span events recorded without a timestamp could get a time up to 1 ms before the span start, because they didn't use the span's clock. They now use the tracer clock.

Notes:
The v0.x tag keeps the previous StringBuilder encoding and escaping rules. It now uses one builder per tag instead of one per event and per attribute set. I didn't use datadog.json.JsonWriter: on JDK 17 and earlier its OutputStreamWriter allocates an 8 KB buffer per call, and it encodes one character at a time, which is costly for exception stack traces.

Known limitations, to address separately:

Like links, events are stored in a CopyOnWriteArrayList, so each add copies the list. That's fine for a few events per span but costly for spans with many. There is still no limit on the number of events, as before.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]

🤖 Generated with Claude Code

This would avoid encoding span tags as JSON, to then decode them as Java objects in order then to encode them as msgstruct

…ingBuilder than JsonWriter

dd-octo-sts · 2026-10-07T09:46:56Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.04 s	14.00 s	[-0.7%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent	13.00 s	13.03 s	[-1.0%; +0.6%] (no difference)
startup:petclinic:appsec:Agent	17.27 s	17.19 s	[-0.4%; +1.5%] (no difference)
startup:petclinic:iast:Agent	16.98 s	17.04 s	[-1.1%; +0.4%] (no difference)
startup:petclinic:profiling:Agent	16.68 s	16.75 s	[-1.5%; +0.7%] (no difference)
startup:petclinic:sca:Agent	17.15 s	16.93 s	[+0.3%; +2.3%] (maybe worse)
startup:petclinic:tracing:Agent	16.19 s	16.28 s	[-1.4%; +0.3%] (no difference)

Commit: 447e49a2 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

No longer use hardcoded system time source but the tracer one.

PerfectSlayer · 2026-10-07T15:17:42Z

@codex review

chatgpt-codex-connector · 2026-10-07T15:18:07Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-07T15:26:39.766175Z	`8f00694`	Manual request
🔒 Security Review	✅ Completed 2026-10-07T15:24:58.347076Z	`8f00694`	Manual request

Security findings

Advisory findings (1)

🟡 Run span events through pre-write redaction · Medium

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

🛡️ Codex Security Review · Automatically triggered

Here are some automated security review suggestions for this pull request.

Reviewed commit: 8f00694578

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f00694578

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-07T15:26:35Z

+      if (events != NO_EVENT) {
+        events.add(event);
+      } else {
+        this.events = new CopyOnWriteArrayList<>(singletonList(event));


Replace copy-on-write storage for span events

For spans that record many events—such as the concurrent GraphQL/DataLoader scenario covered by the new 16,000-event test—every add to this CopyOnWriteArrayList copies all preceding events, producing quadratic work and roughly 128 million copied references for that test alone. Span events are write-heavy until completion, so use synchronized append storage with a stable snapshot at finish rather than the link-oriented copy-on-write strategy.

AGENTS.md reference: AGENTS.md:L81-L81

Useful? React with 👍 / 👎.

PerfectSlayer added 2 commits October 7, 2026 11:09

feat(core): Move span event encoding as tag into V0.4 encoder

c2a2cb6

This would avoid encoding span tags as JSON, to then decode them as Java objects in order then to encode them as msgstruct

fix(core): Restore the original span events as tag encoding using Str…

9b8e62c

…ingBuilder than JsonWriter

This comment has been minimized.

Sign in to view

PerfectSlayer added 2 commits October 7, 2026 14:37

fix(core): Fix span event time source

3f81bc3

No longer use hardcoded system time source but the tracer one.

fix(core): Add missing test coverage

8f00694

chatgpt-codex-connector Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/core/DDSpanContext.java

chatgpt-codex-connector Bot reviewed Oct 7, 2026

View reviewed changes

fix(core): Prevent wrapper allocation in common case

447e49a

Navigation Menu

Repository navigation

Keep span events structured until serialization - #12768

Keep span events structured until serialization#12768
PerfectSlayer wants to merge 5 commits into
masterfrom
bbujon/span-links

PerfectSlayer commented Oct 7, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

PerfectSlayer commented Oct 7, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot Oct 7, 2026

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

PerfectSlayer commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

PerfectSlayer commented Oct 7, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Security findings

Advisory findings (1)

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

🛡️ Codex Security Review · Automatically triggered

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

PerfectSlayer commented Oct 7, 2026 •

edited

Loading

dd-octo-sts Bot commented Oct 7, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 7, 2026 •

edited

Loading