Repository navigation
fix(opentelemetry): prevent OTLP logs self-export loop - #19266
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits intoJul 27, 2026
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
Conversation
…r's own logs When DD_LOGS_OTEL_ENABLED=true, the OpenTelemetry SDK attaches a LoggingHandler to the Python root logger that captures every propagated log record. Because ddtrace's internal loggers (ddtrace.*) and the OpenTelemetry exporter's loggers (opentelemetry.*) propagate to root, the exporter's own log records -- the per-batch export debug line and the exporter's export-failure warnings/errors -- were captured, exported, and captured again, producing a self-amplifying export loop (triggered by DD_TRACE_DEBUG or by export failures at the default log level). Attach a logging.Filter to the OTLP LoggingHandler that rejects records whose logger name is in the ddtrace or opentelemetry namespaces, so the tracer never feeds its own telemetry-pipeline logs back into its own log export. Application logs are still captured and exported. The filter is attached only to the handler ddtrace itself installs (identified by diffing the root logger's handlers), so application-configured handlers are left untouched. Because it binds to the handler instance, it also persists across logging.basicConfig reconfiguration, which the OpenTelemetry SDK patches to re-add that same handler instance (dictConfig/fileConfig are not patched by the SDK). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bm1549
added
the
AI Generated
This was referenced Jul 24, 2026
Contributor
🎉 All green!🧪 All tests passed 🔄 Datadog auto-retried 1 job - 1 passed on retry 🔗 Commit SHA: b35341a | Docs | Datadog PR Page | Give us feedback! |
BenchmarksBenchmark execution time: 2026-07-24 02:13:52 Comparing candidate commit 31b6da5 in PR branch Found 0 performance improvements and 4 performance regressions! Performance is the same for 616 metrics, 10 unstable metrics. scenario:iastaspects-lstrip_aspect
scenario:iastaspects-translate_aspect
scenario:iastaspectsospath-ospathbasename_aspect
scenario:telemetryaddmetric-1-count-metric-1-times
|
…logs-self-telemetry-loop
Circular import analysis
|
mabdinur
approved these changes
Jul 27, 2026
gh-worker-dd-mergequeue-cf854d
Bot
deleted the
brian.marks/fix-otlp-logs-self-telemetry-loop
branch
July 27, 2026 15:46
Codeowners resolved as |
brettlangdon
pushed a commit
that referenced
this pull request
Aug 3, 2026
## Description When `DD_LOGS_OTEL_ENABLED=true`, the OTLP logs pipeline attaches an OpenTelemetry `LoggingHandler` to the Python root logger. ddtrace's own loggers (`ddtrace.*`) and the OpenTelemetry exporter's loggers (`opentelemetry.*`) propagate to root, so the handler captured the tracer's own log records: the per-export debug line and the exporter's export-failure warnings/errors. Exporting those records emits more log records, which were captured and exported again. The result is a self-amplifying export loop. This happens whenever those records reach the effective log level: with `DD_TRACE_DEBUG=true` (the per-export debug line), or on export failures (warnings/errors at the default level). The fix attaches a self-telemetry exclusion filter to the OTLP `LoggingHandler` that drops records whose logger name is in the `ddtrace` or `opentelemetry` namespaces. Matching is exact-or-dotted-prefix (`ddtrace` / `ddtrace.*`, `opentelemetry` / `opentelemetry.*`), so application loggers such as `ddtrace_myapp` are not affected. The filter is attached only to the handler ddtrace itself installs; application-configured handlers are not modified. Application logs are still captured and exported. This was discovered during cross-tracer startup-log work. It is an independent, pre-existing bug, so it ships on its own branch. ## Testing Adds `tests/opentelemetry/test_logs.py::test_otel_logs_exporter_excludes_self_telemetry`. With `DD_LOGS_OTEL_ENABLED=true`, it emits a `ddtrace.*` record, an `opentelemetry.*` record, and an application record, flushes the exporter to a mock gRPC log service, and asserts the application record is exported while the two self-telemetry records are not. The test fails without the fix and passes with it. It runs across the OpenTelemetry versions already covered by the `opentelemetry` test venv. ## Risks Low. The change only adds a logging filter to the handler ddtrace creates. Records from the `ddtrace` and `opentelemetry` logger namespaces are no longer exported through this pipeline; application logs are unchanged. The filter binds to the handler instance, so it persists across `logging.basicConfig` reconfiguration, which is the reconfiguration API the OpenTelemetry SDK patches to re-add the same handler instance (`dictConfig`/`fileConfig` are not patched by the SDK). ## Additional Notes Includes a reno release note. No configuration or public API changes. --- ### Context Independent, pre-existing bug surfaced while adding OTLP export-status fields to the tracer startup log across dd-trace-* (the Python system-test for the logs field hung on this loop). Related: - Startup-log feature (dd-trace-py): #19265 - system-tests coverage that surfaced it: DataDog/system-tests#7376 Co-authored-by: brian.marks <brian.marks@datadoghq.com>
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
to DataDog/dd-trace-go
that referenced
this pull request
Aug 4, 2026
### What does this PR do? Adds three boolean fields to the `DATADOG TRACER CONFIGURATION` startup log: - `otlp_traces_export_enabled` reports whether the tracer selected the OTLP trace writer. - `otlp_metrics_export_enabled` reports whether the tracer will start OTel runtime metrics. - `otlp_logs_export_enabled` reports whether OTel logs are enabled in the tracer configuration. Updates the startup-log expectations, clears the OTLP-related environment variables in `TestStartupLog`, and tests the metrics startup gate with and without the OTel metrics hook installed. ### Motivation Part of the cross-tracer work to use the same OTLP export-status fields in each tracer's startup diagnostics. The fields show which signals a service exports over OTLP. Related PRs: - dd-trace-js: DataDog/dd-trace-js#9517 - dd-trace-py: DataDog/dd-trace-py#19265 - dd-trace-java: DataDog/dd-trace-java#12062 - dd-trace-dotnet: DataDog/dd-trace-dotnet#8936 - dd-trace-rb: DataDog/dd-trace-rb#6096 - dd-trace-php: DataDog/dd-trace-php#4056 - system-tests: DataDog/system-tests#7376 - dd-trace-py OTLP logs fix: DataDog/dd-trace-py#19266 ### Reviewer's Checklist - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [ ] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag. (N/A: startup diagnostic-log fields only.) - [ ] There is a benchmark for any new code, or changes to existing code. (N/A: no hot-path or performance-sensitive code.) - [ ] If this interacts with the agent in a new way, a system test has been added. (N/A: no new agent interaction.) - [ ] New code is free of linting errors. You can check this by running `make lint` locally. (Full lint not run; `gofmt` and `git diff --check` are clean.) - [x] New code doesn't break existing tests. You can check this by running `make test` locally. (`go test ./ddtrace/tracer -count=1` passes.) - [ ] Add an appropriate team label so this PR gets put in the right place for the release notes. - [ ] All generated files are up to date. You can check this by running `make generate` locally. (N/A: no generated files changed.) - [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally. (N/A: no module changes.) Co-authored-by: dario.castane <dario.castane@datadoghq.com>
juanjux
added a commit
that referenced
this pull request
Sep 23, 2026
#20506) ## Description Backport of #19266 to the 4.13 release line. When OpenTelemetry logs are enabled (`DD_LOGS_OTEL_ENABLED=true`, or `OTEL_SDK_DISABLED=false`), the OTLP logs handler is attached to the root logger. ddtrace's own loggers and the OpenTelemetry exporter's loggers propagate there, so the per-export debug line is captured and exported again. With `DD_TRACE_DEBUG=true` that becomes a self-amplifying loop: the flush never returns, and the process writes logs as fast as it can. Parametric system tests always set `DD_TRACE_DEBUG=true`. `test_stable_false` sets `OTEL_SDK_DISABLED=false`, which enables logs on 4.13. The test client's logs flush has no timeout, so the pytest worker blocks. On the CI docker-in-docker runner the log flood also stalls the other workers, and `system-tests parametric` sits at about 97% until the one-hour job timeout. 4.14 and 4.15 already have this filter, which is why they finish the same suite. The filter is attached only to the handler ddtrace installs. Application logs are still exported. Loggers named `ddtrace_myapp` are not excluded. ## Testing - `scripts/run-tests --venv 1ca3564 -- tests/opentelemetry/test_logs.py::test_otel_logs_exporter_excludes_self_telemetry` passed on Python 3.11 (opentelemetry-exporter-otlp). - Locally, the same parametric test against the 4.13.3 wheel hung in `otel_logs_flush()` and the container log grew to ~142k lines in 25 seconds. That wheel does not include this change. ## Risks Records from the `ddtrace` and `opentelemetry` logger namespaces are no longer exported through the OTLP logs pipeline ddtrace installs. Application logs are unchanged. The filter stays on the handler instance across `logging.basicConfig`. ## Additional Notes Original change: #19266 (`3e949c45d3`).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When
DD_LOGS_OTEL_ENABLED=true, the OTLP logs pipeline attaches an OpenTelemetryLoggingHandlerto the Python root logger. ddtrace's own loggers (ddtrace.*) and the OpenTelemetry exporter's loggers (opentelemetry.*) propagate to root, so the handler captured the tracer's own log records: the per-export debug line and the exporter's export-failure warnings/errors. Exporting those records emits more log records, which were captured and exported again. The result is a self-amplifying export loop. This happens whenever those records reach the effective log level: withDD_TRACE_DEBUG=true(the per-export debug line), or on export failures (warnings/errors at the default level).The fix attaches a self-telemetry exclusion filter to the OTLP
LoggingHandlerthat drops records whose logger name is in theddtraceoropentelemetrynamespaces. Matching is exact-or-dotted-prefix (ddtrace/ddtrace.*,opentelemetry/opentelemetry.*), so application loggers such asddtrace_myappare not affected. The filter is attached only to the handler ddtrace itself installs; application-configured handlers are not modified. Application logs are still captured and exported.This was discovered during cross-tracer startup-log work. It is an independent, pre-existing bug, so it ships on its own branch.
Testing
Adds
tests/opentelemetry/test_logs.py::test_otel_logs_exporter_excludes_self_telemetry. WithDD_LOGS_OTEL_ENABLED=true, it emits addtrace.*record, anopentelemetry.*record, and an application record, flushes the exporter to a mock gRPC log service, and asserts the application record is exported while the two self-telemetry records are not. The test fails without the fix and passes with it. It runs across the OpenTelemetry versions already covered by theopentelemetrytest venv.Risks
Low. The change only adds a logging filter to the handler ddtrace creates. Records from the
ddtraceandopentelemetrylogger namespaces are no longer exported through this pipeline; application logs are unchanged. The filter binds to the handler instance, so it persists acrosslogging.basicConfigreconfiguration, which is the reconfiguration API the OpenTelemetry SDK patches to re-add the same handler instance (dictConfig/fileConfigare not patched by the SDK).Additional Notes
Includes a reno release note. No configuration or public API changes.
Context
Independent, pre-existing bug surfaced while adding OTLP export-status fields to the tracer startup log across dd-trace-* (the Python system-test for the logs field hung on this loop). Related: