Repository navigation
fix(id): reset DogStatsD and exporter state on refresh - #10323
litianningdatadog wants to merge 1 commit into
Conversation
Overall package sizeSelf size: 8.96 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.5.1 | 127.66 kB | 513.61 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10323 +/- ##
=======================================
Coverage 98.71% 98.71%
=======================================
Files 1025 1025
Lines 160626 160673 +47
Branches 74 74
=======================================
+ Hits 158563 158610 +47
Misses 2063 2063
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🟡 Changes recommended
Three moderate issues remain around telemetry accumulators, AgentExporter reset wiring, and OTLP trace-root preservation.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
This PR resets stale DogStatsD, span-statistics, and exporter state when a MicroVM identity refresh occurs.
Changes:
- Clears buffered telemetry and exporter batches on refresh.
- Resets span-statistics buckets.
- Adds regression tests for refresh behavior.
File summaries
| File | Summary |
|---|---|
packages/dd-trace/test/span_stats.spec.js |
Tests span-statistics reset behavior. |
packages/dd-trace/test/exporters/common/writer.spec.js |
Tests pending writer-batch resets. |
packages/dd-trace/test/exporters/agentless/exporter.spec.js |
Tests agentless refresh subscriptions. |
packages/dd-trace/test/dogstatsd.spec.js |
Tests DogStatsD refresh handling. |
packages/dd-trace/src/span_stats.js |
Resets span-stat buckets; OTLP root-span configuration must be preserved. |
packages/dd-trace/src/exporters/common/writer.js |
Adds pending-batch reset support; normal AgentExporter wiring remains incomplete. |
packages/dd-trace/src/exporters/agentless/index.js |
Resets agentless batches on identity refresh. |
packages/dd-trace/src/dogstatsd.js |
Resets aggregation maps, but telemetry accumulators also require clearing. |
Review details
Suppressed comments (2)
packages/dd-trace/src/dogstatsd.js:610
- On a tag-changing refresh, this call resets only the aggregation maps. A telemetry-enabled client still retains
metricsByType,aggregatedContextsByType, and sent/dropped counters inDogStatsDClient.telemetry; if those values were collected before the snapshot, the next telemetry flush emits them with the refreshed tags. Clear the client telemetry accumulators as part of the same identity-boundary reset.
if (this._client.updateTags(tags)) {
this.reset()
}
packages/dd-trace/src/exporters/common/writer.js:109
- Although the new reset API is on the shared Writer, the identity-refresh listener is only installed by AgentlessExporter. The default AgentExporter (used by
exporter.jsfor normal tracing) also buffers spans in a common Writer whenflushInterval > 0, so a clone can still flush pre-snapshot traces after refresh. Wire the reset into AgentExporter as well, or centralize it in the shared exporter lifecycle.
resetPendingBatch () {
this._encoder.reset()
- Files reviewed: 8/8 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // A clone resume shouldn't export buckets accumulated before the snapshot under its own identity. | ||
| unsubscribeBucketReset?.() | ||
| const onIdentityRefresh = () => { | ||
| this.buckets = new TimeBuckets() |
|
✅ All CI checks and tests passed. 🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 6533cee | Docs | View more details | Give us feedback! |
![datadog-prod-us1-4[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
With the default DogStatsD setting, an identity refresh does not remove inherited aggregate data. The span statistics reset removes OTLP trace-root grouping after the first refresh.
🤖 Datadog Autotest · Commit 6533cee · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
| */ | ||
| updateTags (tags) { | ||
| this._client.updateTags(tags) | ||
| if (this._client.updateTags(tags)) { |
There was a problem hiding this comment.
MicroVM clones can send duplicate aggregate metrics from the snapshot.
Assertion details
- Input: A MicroVM identity refresh with the default runtimeMetricsRuntimeId setting and pending aggregate metrics.
- Expected:
Each identity refresh must remove aggregate data that the clone inherits from the snapshot. - Actual:
The new condition resets aggregate data only when the serialized tag prefix changes. The default setting omits the runtime ID, so a clone refresh can leave the prefix unchanged and preserve inherited counters, gauges, and histograms.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session
| // A clone resume shouldn't export buckets accumulated before the snapshot under its own identity. | ||
| unsubscribeBucketReset?.() | ||
| const onIdentityRefresh = () => { | ||
| this.buckets = new TimeBuckets() |
There was a problem hiding this comment.
OTLP span metrics become incorrect after the first identity refresh.
Assertion details
- Input: An identity refresh while OTLP span metrics are active, followed by root and child spans with the same dimensions.
- Expected:
The new bucket set must keep trace-root grouping when the OTLP exporter is active. - Actual:
The refresh handler replaces OTLP TimeBuckets with the default mode. This mode merges trace roots with other spans and removes the datadog.is_trace_root attribute.
| this.buckets = new TimeBuckets() | |
| this.buckets = new TimeBuckets(Boolean(this.otlpExporter)) |
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6533ceed93
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // A clone resume shouldn't export buckets accumulated before the snapshot under its own identity. | ||
| unsubscribeBucketReset?.() | ||
| const onIdentityRefresh = () => { | ||
| this.buckets = new TimeBuckets() |
There was a problem hiding this comment.
When OTLP span statistics are enabled and the MicroVM identity-refresh channel fires, this replaces the original TimeBuckets(true) with the default non-OTLP configuration. All subsequently recorded root and child spans are therefore merged under the same aggregation key and exported without the datadog.is_trace_root attribute; recreate the buckets with Boolean(this.otlpExporter) so the constructor's OTLP behavior survives the reset.
AGENTS.md reference: AGENTS.md:L79-L80
Useful? React with 👍 / 👎.
| if (this._client.updateTags(tags)) { | ||
| this.reset() | ||
| } |
There was a problem hiding this comment.
When runtimeMetricsRuntimeId has its default value of false, generateClientConfig() excludes the refreshed runtime ID, so the DogStatsD tag prefix remains unchanged on every MicroVM identity refresh. This condition then preserves counters, gauges, and histograms captured before the snapshot, allowing every clone to flush the same inherited aggregate; the identity-refresh path must reset pending aggregation independently of whether the serialized tags changed.
Useful? React with 👍 / 👎.
BenchmarksBenchmark execution time: 2026-09-14 20:34:03 Comparing candidate commit 6533cee in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2304 metrics, 9 unstable metrics.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6533ceed93
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| // A clone resume shouldn't flush spans buffered before the snapshot under its own identity. | ||
| unsubscribeBatchReset?.() | ||
| const onIdentityRefresh = () => this._writer.resetPendingBatch() |
There was a problem hiding this comment.
When a MicroVM uses the normal Agent exporter—for example through a local Agent or Lambda extension—packages/dd-trace/src/exporters/agent/index.js:23-31 constructs the same common writer but never subscribes to the identity-refresh channel. Traces buffered before the image snapshot therefore remain in its AgentEncoder, and every clone can flush the inherited traces when its timer fires; wire resetPendingBatch() into AgentExporter as well.
AGENTS.md reference: AGENTS.md:L79-L79
Useful? React with 👍 / 👎.
| payload.offset = 0 | ||
| } | ||
|
|
||
| return true |
There was a problem hiding this comment.
When the refreshed tags do change, this method drops the encoded telemetry payload but leaves telemetry.metricsByType, aggregatedContextsByType, bytesSent/bytesDropped, and packetsSent/packetsDropped untouched. If the image is snapshotted before the telemetry interval flushes, every clone later reports those inherited pre-snapshot counts under the refreshed prefix, duplicating DogStatsD client telemetry even though the application metric buffers were reset; clear these interval accumulators at the same identity boundary.
AGENTS.md reference: AGENTS.md:L79-L79
Useful? React with 👍 / 👎.
Stacked PRs
tianning.li/1-dogstatsd-exporter-refresh-resets— current PRtianning.li/2-otel-runtime-refresh-resetstianning.li/3-exporter-otel-identity-gapstianning.li/4-runtime-datastream-identity-gapstianning.li/5-exporter-buffer-statetianning.li/6-otel-datastream-counter-statetianning.li/7-spanstats-otel-retry-statetianning.li/8-agent-otel-counter-baselinestianning.li/9-datastream-dogstatsd-exporter-requestsOrigin
This change set was split from the monolithic PR #9814 to keep the identity-refresh fixes smaller and easier to review.
What does this PR do?
Reset inherited DogStatsD buffers and exporter batches when a MicroVM identity refresh occurs.
Motivation
A clone can inherit pre-snapshot DogStatsD and exporter state. Discarding that state prevents duplicate or stale telemetry from being reported under the refreshed runtime identity.
Logical change
This is the 1-dogstatsd-exporter-refresh-resets layer of the stacked MicroVM identity-refresh fix. It is based on
masterand keeps the subsystem-specific reset behavior isolated for review.Jira
SVLS-9768
Tests
Additional Notes
This PR is intentionally opened as a draft and is part of a linear stack.