Repository navigation
Conversation
link04
added
the
changelog/no-changelog
Codeowners resolved asResolved from the full PR diff against |
Circular import analysis
|
Dependency direction analysis
|
❌ ErrorsYour PR has failed checks. Please review the issues below and take necessary action before merging. 🚦 7 Pipeline jobs failed
ℹ️ InfoNo other issues found (see more)❄️ No new flaky tests detected 🔄 Datadog retried 7 tests - 1 passed on retry 🚧 2 tests that failed were ignored due to quarantine Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: 2353c5d | Docs | View more details | Give us feedback! |
BenchmarksBenchmark execution time: 2026-10-08 16:23:16 Comparing candidate commit 2353c5d in PR branch Found 0 performance improvements and 3 performance regressions! Performance is the same for 357 metrics, 9 unstable metrics, 3 known flaky benchmarks, 5 flaky benchmarks without significant changes.
|
link04
added
OpenTelemetry
AI Generated
…nsions (#2323) ### What does this PR do? Preserves OpenTelemetry HTTP semantics in libdatadog's in-process consumers and OTLP output. #### Trace stats and OTLP trace metrics - reads canonical Datadog HTTP method/status keys first, then falls back to `http.request.method` and `http.response.status_code` - accepts valid numeric or string status values without allowing empty, malformed, non-finite, or non-integral values to shadow a valid fallback - emits OTel trace-metric dimensions for method, status, route, span name, and kind - leaves successful `status.code` unset and prevents additional tags from overriding the computed status ### HTTP tag handling libdatadog does not rename tracer span tags in this PR. It accepts the Datadog and OpenTelemetry names below as equivalent inputs to trace-stat aggregation and emits the canonical OpenTelemetry name in OTLP trace metrics. | Concept | Datadog span input | OpenTelemetry span input | OTLP trace-metric output | | --- | --- | --- | --- | | HTTP method | `http.method` | `http.request.method` | `http.request.method` | | HTTP response status | `http.status_code` | `http.response.status_code` | `http.response.status_code` as an integer | | Span error status | Span error flag | Span error flag | `status.code=STATUS_CODE_ERROR` for errors; omitted for successful spans | When both naming variants are present, the Datadog name wins. Empty or malformed values fall through to the alternate name instead of hiding a valid value. #### OTLP sampled flags Sampling priority is a trace-level decision, but v0.4/v0.5 inputs may attach `_sampling_priority_v1` to only one span. The mapper now uses the same first-priority policy as `drop_chunks`, treats a missing priority as sampled, and applies one decision to every span in the chunk. This prevents sampled traces from exporting child spans with `flags=0` and keeps OTLP flags consistent with the existing chunk-retention policy. No public signature, wire-format, setting, or environment-variable change is introduced. ### Motivation Under `DD_TRACE_OTEL_SEMANTICS_ENABLED`, spans use OTel HTTP names before client-side stats are computed. Stats and OTLP metrics must preserve those dimensions, and all spans in an exported trace must carry the same sampled decision. Companion changes: - Historical full Python PR: [DataDog/dd-trace-py#19501](DataDog/dd-trace-py#19501) - Native Python stack: [foundation #20008](DataDog/dd-trace-py#20008), [client #20009](DataDog/dd-trace-py#20009), [server #20010](DataDog/dd-trace-py#20010) - [DataDog/dd-trace-js#9688](DataDog/dd-trace-js#9688) - [DataDog/system-tests#7455](DataDog/system-tests#7455) ### Testing - `cargo fmt --check -p libdd-trace-utils` - mapper unit tests: **31 passed** - OTLP trace-exporter integration tests: **3 passed** - trace-stats tests: **43 passed** End-to-end re-run from libdatadog head `0ed7fd9bd` with the final Python server layer and companion system tests: | Scenario | Result | | --- | ---: | | `OTEL_SEMANTICS_OTLP_TRACE_METRICS` | **4 passed** | | `OTEL_SEMANTICS_OTLP_SAMPLING_RULES` | **2 passed** | Co-authored-by: maximo.bautista <maximo.bautista@datadoghq.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3802bd6045
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
3802bd6 to
6f6a6ab
Compare
…tizing with http.URL, and resolve OTLP trace export in a single live helper
…ule so tracing code does not import the contrib package
…ith the same name
…t_for_num_traces in the OTel snapshot harness
…stead of start times
…uested and order identical traces by resource and scope
3f165df to
9df533c
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9df533c273
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if not config._otel_trace_semantics_enabled: | ||
| span._set_attribute(http.METHOD, method) | ||
| return | ||
| normalized_method, original_method = normalize_http_method(method) |
There was a problem hiding this comment.
When OTel semantics is enabled and a user configures an extension method such as OTEL_INSTRUMENTATION_HTTP_KNOWN_METHODS=PROPFIND, this call still uses the hard-coded default method set; a repository-wide search finds no reader for that variable or another known-method override. The client span is consequently emitted with http.request.method=_OTHER and resource HTTP rather than PROPFIND, preventing sampling rules and dashboards from matching the configured method. The OTel HTTP convention requires an override whenever valid methods can be converted to _OTHER; make normalization honor the configured full method set.
Useful? React with 👍 / 👎.
… a client-side harness
Apply OTel HTTP attributes, error handling, and sampling-visible resource names to outbound spans while delegating span mutations to the tracing layer. Co-authored-by: Cursor <cursoragent@cursor.com>
Name Datadog semantics explicitly in client tests and keep the resource ownership comment concise. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…tion so sampling sees it
…logging raw malformed URLs
… client instrumentation
9df533c to
2353c5d
Compare
## Description Adds the shared foundation for emitting OpenTelemetry HTTP semantic conventions when `DD_TRACE_OTEL_SEMANTICS_ENABLED=true`. This PR does **not** change HTTP integrations yet. The client and server integrations start using this foundation in #20009 and #20010. This is the first part of the implementation for #19501. ### Configuration `DD_TRACE_OTEL_SEMANTICS_ENABLED` remains disabled by default. When enabled, it enforces the settings required by the OTel HTTP semantics path: - OTLP trace export - Span attribute schema `v0` - Peer-service defaults disabled - Native span events enabled for OTLP export Conflicting settings are overridden and reported through logs and telemetry. ### Shared HTTP semantics The implementation lives under `ddtrace/_trace/otel/http/`: - `tags.py` handles HTTP method normalization, URL attributes, server address and port extraction, query-string handling, status codes, and client/server error classification. - `resource.py` generates low-cardinality HTTP resource names and prevents later instrumentation updates from overwriting resources set by user code. `OTelHTTPSpanAttributes` is shared by the upcoming client and server layers. Those layers will instantiate it from `set_http_meta()` and provide metadata as it becomes available during the request lifecycle. For example: ```python attributes = OTelHTTPSpanAttributes(span, integration_config) attributes.set_method("get") attributes.set_url("https://example.com/users/42") attributes.set_status_code(200) attributes.set_resource("/users/{id}") ``` This produces: ```text http.request.method = "GET" http.request.method_original = "get" url.scheme = "https" url.path = "/users/42" http.response.status_code = 200 resource = "GET /users/{id}" ``` Recreating the helper during later metadata phases is supported: method, resource, and status-error ownership are stored on the span. ### OTel requirements covered The behavior follows the [OpenTelemetry HTTP span conventions](https://opentelemetry.io/docs/specs/semconv/http/http-spans/). | Scenario | OTel behavior | Covered behavior | | --- | --- | --- | | Method `get` | Normalize the method and preserve its original spelling | Emits `http.request.method="GET"` and `http.request.method_original="get"` | | Unknown method `PROPFIND` | Emit `_OTHER` and retain the original method | Emits `_OTHER`, preserves `PROPFIND`, and uses `HTTP` for resource naming | | Server route `/users/{id}` | Use the low-cardinality route template instead of the raw request path | Refines the resource to `GET /users/{id}` | | Client URL containing credentials | Redact user information from `url.full` | Emits `https://REDACTED:REDACTED@example.com/search` | | Client 4xx response | Mark the span as an error | Sets `span.error=1` and a string status-derived `error.type` | | Server 4xx response | Do not mark the span as an error by default | Leaves `span.error=0` | | Server 5xx response | Mark the span as an error | Sets `span.error=1` | Existing Datadog query capture, query obfuscation, and custom server error-range settings remain supported. Exception information takes precedence over status-derived error metadata. ### Resource ownership Resource naming distinguishes instrumentation-generated values from user replacements: 1. Instrumentation may progressively refine a resource as more metadata becomes available. 2. If user code replaces that resource, subsequent instrumentation updates preserve the user value. 3. Raw request paths are never used as OTel server resources; route templates are used when available. WebSocket resources are left unchanged to avoid collapsing them to the unknown-method resource name. ## Testing Unit coverage includes: - Method normalization and stale `method_original` cleanup - Client and server URL attributes - Credential redaction and query-string obfuscation - Malformed URLs - Client/server status-code error classification - Repeated metadata updates - Exception preservation - User-owned resource preservation - Schema and peer-service overrides - OTLP writer selection and native span events End-to-end OTLP snapshots are added later in the stack because they depend on the harness introduced in #20688: - #20689: HTTP client snapshots - #20690: HTTP server and AppSec snapshots ## Risks The feature remains opt-in. This PR configures the enabled mode and adds the shared implementation, but it does not change emitted HTTP integration spans until the client and server follow-up PRs are merged. ## PR stack 1. #20008 — shared configuration and HTTP semantics foundation 2. #20688 — OTLP snapshot harness 3. #20009 — HTTP client integration 4. #20689 — client configuration and snapshots 5. #20010 — HTTP server/framework and AppSec integration 6. #20690 — AppSec block-path attributes and server snapshots Co-authored-by: Hanane-Laassilia <hanane.laassilia@datadoghq.com>
3 participants
Description
Adds OpenTelemetry HTTP semantic conventions for outbound HTTP client spans when
DD_TRACE_OTEL_SEMANTICS_ENABLED=true.This PR wires the shared HTTP semantics introduced in #20008 into the HTTP client lifecycle and
set_http_meta. The snapshot coverage and configurable client error-status ranges are added separately in #20689.This is the client layer of the implementation replacing #19501.
Emitted attributes
When OTel semantics are enabled, client spans use the following attributes:
http.request.methodhttp.request.method_originalurl.fullserver.addressserver.porthttp.response.status_codeuser_agent.originalerror.typeNumeric values such as
server.portandhttp.response.status_codeare stored as numeric attributes so they retain the correct type when exported over OTLP.For example, a request using:
emits:
The equivalent Datadog HTTP attributes are not emitted on the OTel path.
Method normalization
Known methods are normalized to their canonical uppercase spelling:
When normalization changes the original value, the input is preserved in
http.request.method_original.Unknown methods use
_OTHER:Client semantics are applied only when the span explicitly has
span.kind=client. An HTTPspan_typealone is not enough because some server integrations also use it.URL handling
Client URLs are emitted as
url.full.The implementation:
@;server.addressandserver.portfrom the URL;Error handling
Client responses with a status code of
400or greater are marked as errors.The status code is stored as the string value of
error.typewhen no more specific error type already exists. Exception metadata always takes precedence and is never replaced by a status-derived error type.Configurable client error-status ranges are intentionally deferred to #20689.
Resource naming
OTel client spans use a low-cardinality, method-only resource:
Client routes and raw URL paths are not included in the resource.
For event-based HTTP clients, the resource is assigned during the span-start lifecycle, before propagation and sampling. This ensures that resource-based sampling rules see the final OTel resource rather than the integration’s previous
METHOD pathvalue.The implementation also preserves user ownership:
set_http_metado not overwrite a user-owned resource.httplibperforms the same early resource assignment before distributed-tracing injection, where its sampling decision occurs.Compatibility
When
DD_TRACE_OTEL_SEMANTICS_ENABLEDis disabled, the existing Datadog behavior is unchanged:The feature flag is evaluated at each
set_http_metacall.Validation includes:
scripts/lint checksTest_OtelSemantics_Spans_Http_Clientsystem tests against libdatadog#2323Risks
The change affects the shared
set_http_metapath used by HTTP integrations. The behavior is guarded by an opt-in feature flag, requires an explicit client span kind, and has coverage for both the OTel and existing Datadog paths.PR stack
This PR is based on #20688. The original #19501 remains available as the complete historical implementation and system-test reference.