Repository navigation
feat(tracer): add DD_TRACE_OTEL_SEMANTICS_ENABLED for OTLP export - #4889
Conversation
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 1033fca | Docs | Datadog PR Page | Give us feedback! |
Codecov Report❌ Patch coverage is
Additional details and impacted files
🚀 New features to boost your workflow:
|
BenchmarksBenchmark execution time: 2026-07-10 21:47:23 Comparing candidate commit 1033fca in PR branch Found 1 performance improvements and 0 performance regressions! Performance is the same for 320 metrics, 2 unstable metrics, 1 flaky benchmarks without significant changes.
|
b23cf4e to
6dda781
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6dda781634
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
6dda781 to
62ed88e
Compare
darccio
left a comment
There was a problem hiding this comment.
LGTM. I left some comments but they are not blocking.
62ed88e to
74fa425
Compare
Opt-in mode that aligns OTLP-exported spans with the pure OpenTelemetry SDK by suppressing Datadog-specific attributes and stopping DD-convention remaps. Mirrors dd-trace-py #18495 and libdatadog #2091. When DD_TRACE_OTEL_SEMANTICS_ENABLED=true: - omit operation.name, resource.name, span.type (DD-only span attributes) - omit error.message/type/stack/handling_stack from the span (they live on the exception span event for Error Tracking) - omit span.kind (already carried by the first-class OTLP SpanKind field) - keep http.response.status_code under its OTel key instead of remapping to the legacy http.status_code string tag - the OTel bridge no longer interprets Datadog reserved tags (operation.name, analytics.event, http.response.status_code) in SetAttributes - SetName updates the Datadog resource (the OTLP span name) instead of the DD-only operation name; gated here because changing it by default shifts RED metrics Registers the key in supported_configurations so env.Get resolves it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
74fa425 to
c57edb1
Compare
Config Audit |
genesor
left a comment
There was a problem hiding this comment.
LGTM, a small NIT about the style.
…tter Rename OtelSemanticsEnabled/SetOtelSemanticsEnabled to the OTel initialism per Go style, and update the call sites in the OTLP writer and OTel bridge so the package compiles. Co-authored-by: Benjamin De Bernardi <debernardi.benjamin@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
0b6b15a to
1033fca
Compare
|
View all feedbacks in Devflow UI.
This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
The expected merge time in
|
The main merge brought in #4889's 3-arg convertSpan(*Span, string, bool); update the TestOTLPExportModeSpanEvents call site (otelSemantics=false) so the tracer test package compiles. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Document that the semantics branch applies to every export format: an APM/MessagePack-exported error span keeps error=1 and error.message but drops error.handling_stack, matching dd-trace-dotnet under the same flag. - Correct error.stack to error.handling_stack in the tests and comments. error.stack is only set for errors implementing xerrors.Formatter, fmt.Formatter, or *errortrace.TracerError, so it was never emitted on this path and the old assertion could not fail. Assert on the ext.* constants. - Explain in ddOnlyMetaKeys why #4889 suppressed error.type and how it differs from RecordError's exception event. - Add TestSpanEndErrorTypeOtelSemanticsOTLPRoundTrip, which drives a bridge span through the real OTLP writer and asserts on the decoded protobuf, so End() and the OTLP conversion are verified as a composed whole.
### What does this PR do? Under `DD_TRACE_OTEL_SEMANTICS_ENABLED`, preserves the OpenTelemetry `error.type` attribute end-to-end instead of clobbering it and dropping it. - **Bridge (`ddtrace/opentelemetry/span.go`)**: on an error-status span, the OTel semantics path now marks the span errored via `SetTag(ext.Error, true)` instead of `tracer.WithError(errors.New(description))`. `WithError` runs after instrumentation attributes are flushed and overwrote `error.type` with the reflect type of the status wrapper (`"*errors.errorString"`), and also injected `error.handling_stack`. The non-semantics path is unchanged. - **OTLP converter (`ddtrace/tracer/span_to_otlp.go`)**: removes `error.type` from `ddOnlyMetaKeys`, so it is no longer suppressed from OTLP output under semantics. Net effect under semantics: an instrumentation-set `error.type` (e.g. `"*net.OpError"` from otelhttp client spans) survives to OTLP; when none is set, `error.type` is simply absent (the error state is carried by the OTLP span status) rather than a meaningless `"*errors.errorString"`. Both halves are required. Removing only the `ddOnlyMetaKeys` entry would export `"*errors.errorString"`, because the bridge overwrote the instrumentation value before export. ### Motivation `error.type` is a Stable attribute in the OpenTelemetry semantic conventions (https://opentelemetry.io/docs/specs/semconv/registry/attributes/error/). Under OTel semantics we should emit the instrumentation-provided value, not a Datadog-internal reflect-type string, and we must not drop it. Follow-up to #4889. ### Cross-SDK validation Verified against a real application through the FTF OTLP scorecard (`dd-otel-sdk`, Go app instrumented with `otelhttp`), comparing the dd-trace-go bridge with the pure OpenTelemetry SDK, both exporting OTLP, on a request that fails its outbound call: | | `error.type` on the failing client span | |---|---| | dd-trace-go `main` | absent entirely | | this branch | `*net.OpError` | | pure OpenTelemetry SDK | `*net.OpError` | The scorecard reports `error.type` as preserved, `{"score": 1.0, "exact": 1, "mismatch": 0}`. Note the official `semconv.ErrorType` helper derives the value by reflection as well, so passing our value through converges with the reference SDK rather than diverging from it. ### Behavior under APM export (intended) The bridge change applies regardless of export format. Under semantics, an error span exported to the Datadog Agent (MessagePack, not OTLP) carries `error=1` and `error.message`, but no longer `error.handling_stack`, and carries `error.type` only if instrumentation set one. Previously it always received `error.type="*errors.errorString"` plus a handling stack. This is intended, and it is the shape the other tracers already ship: dd-trace-dotnet does the same under the same flag (`OtlpHelpers.cs`, `AgentConvertSpan`: `span.Error = true` plus `SetTag(Tags.ErrorMsg, ...)` when OTel semantics is enabled, otherwise the legacy `Status2Error` flattening). dd-trace-js/py/java/rb never injected a reflect-type `error.type` from OTel status at all. `error.message` is still set unconditionally because it feeds the OTLP status message. Note: `error.stack` is not affected. It is only set for errors implementing `xerrors.Formatter`, `fmt.Formatter`, or `*errortrace.TracerError`, and the bridge passed `*errors.errorString`, which implements none of them. ### Testing - `ddtrace/tracer/span_to_otlp_test.go`: `TestConvertSpanAttributesOtelSemantics` asserts `error.type` passes through (with a realistic `"*net.OpError"` value) while `error.message`/`error.stack`/`error.handling_stack`/`span.kind` remain suppressed. - `ddtrace/opentelemetry/span_test.go`: `TestSpanEndErrorTypeOtelSemantics` covers the bridge `End()` branch under semantics, both with an instrumentation-set `error.type` (it survives, no `error.handling_stack` injected) and without one (no reflect-type value injected). `TestSpanEnd` gains one assertion pinning the non-semantics `WithError` path. - `ddtrace/opentelemetry/span_test.go`: `TestSpanEndErrorTypeOtelSemanticsOTLPRoundTrip` drives a bridge span through the real OTLP writer to a local listener and asserts on the decoded protobuf (error status, `error.type` preserved, DD-only tags absent), so the bridge and the converter are verified as a composed whole. Confirmed to fail if either half of the fix is reverted. - All of the above pass under `go test -race`. ### Reviewer's Checklist - [x] Changed code has unit tests for its functionality. - [ ] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added. - [ ] If this interacts with the agent in a new way, a system test has been added. Co-authored-by: maximo.bautista <maximo.bautista@datadoghq.com>
…g resource naming
One of this branch's own dependency-bump commits pulled
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp up to
v0.69.0 across 7 separate go.mod files (as an MVS side effect, not a
deliberate choice). v0.69.0 changed otelhttp's default span-name
formatter to conform to OpenTelemetry's HTTP semantic conventions
(upstream #8871): the default span name became '{method} {route}' or
just '{method}' instead of the caller-supplied operation name.
Since ddtrace/opentelemetry.tracer.go maps the OTel span name directly
onto the Datadog resource (tracer.ResourceName(spanName)), this
silently changed resource naming for every OTel-bridge HTTP server
span, breaking TestHttpDistributedTrace ('testOperation' expected,
'GET' actual).
main handles this exact transition via a dedicated
DD_TRACE_OTEL_SEMANTICS_ENABLED toggle (#4889) that this branch never
picked up. Backporting that whole feature is a real product decision;
in the meantime, downgrading otelhttp to v0.68.0 (its own default
formatter unconditionally preserves the operation name, matching what
this repo shipped before) is the minimal fix that restores the
pre-existing, still-shipping behavior. v0.69.0's changes are all
semconv-compliance and cleanup per upstream's changelog — no security
fix is being reverted.
Full ddtrace/... suite green, go vet clean, govulncheck shows no new
findings.
What does this PR do?
Adds an opt-in
DD_TRACE_OTEL_SEMANTICS_ENABLEDthat aligns OTLP-exported spans (and the OTel bridge) with the pure OpenTelemetry SDK by suppressing Datadog-specific attributes and stopping DD-convention remaps. Mirrors dd-trace-py #18495 and libdatadog #2091.Behavior when
DD_TRACE_OTEL_SEMANTICS_ENABLED=trueoperation.name,resource.name,span.type(DD-only span attributes)error.message/type/stack/handling_stackfrom the span (they live on the exception span event for Error Tracking)span.kind(already carried by the first-class OTLPSpanKindfield)http.response.status_codeunder its OTel key instead of remapping to the legacyhttp.status_codestring tagoperation.name,analytics.event,http.response.status_code) inSetAttributes— they pass through as providedSetNameupdates the Datadog resource (which the OTLP exporter emits as the span name) instead of the DD-only operation nameDefault (flag off) behavior is unchanged.
Why
SetNameis gated here (moved from #4887)Changing
SetNameto update the resource is the correct OTel-aligned behavior, but doing it unconditionally is a breaking change: it shifts both the operation name and resource name, which changes users' RED metrics. Per team discussion (matching the dd-trace-js decision), this belongs behind the semantics flag rather than as a default change in a minor release. This supersedes #4887.Performance
The flag is read once per export batch in
otlpTraceWriter.add()and resolved once at provider creation for the bridge (cached on theoteltracer), then threaded as a parameter / read as a field — no per-span/per-attribute global config lookup.Testing
TestConvertSpanAttributesOtelSemantics,TestRemapStatusCodeOtelSemantics,TestToReservedAttributesOtelSemantics, andTestSpanSetNameUpdatesResourceOtelSemantics(flag on) plus the existingTestSpanSetName(flag off, legacy operation-name behavior). Fullddtrace/tracer,ddtrace/opentelemetry, andinternal/configsuites pass.🤖 Generated with Claude Code