Skip to content

fix(id): reset remaining exporter buffers on refresh - #10327

Draft
litianningdatadog wants to merge 1 commit into
tianning.li/4-runtime-datastream-identity-gapsfrom
tianning.li/5-exporter-buffer-state
Draft

litianningdatadog wants to merge 1 commit into
tianning.li/4-runtime-datastream-identity-gapsfrom
tianning.li/5-exporter-buffer-state

Conversation

@litianningdatadog

@litianningdatadog litianningdatadog commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Stacked PRs

  • #10323 tianning.li/1-dogstatsd-exporter-refresh-resets
  • #10324 tianning.li/2-otel-runtime-refresh-resets
  • #10325 tianning.li/3-exporter-otel-identity-gaps
  • #10326 tianning.li/4-runtime-datastream-identity-gaps
  • #10327 tianning.li/5-exporter-buffer-state — current PR
  • #10328 tianning.li/6-otel-datastream-counter-state
  • #10329 tianning.li/7-spanstats-otel-retry-state
  • #10330 tianning.li/8-agent-otel-counter-baselines
  • #10331 tianning.li/9-datastream-dogstatsd-exporter-requests

Origin

This change set was split from the monolithic PR #9814 to keep the identity-refresh fixes smaller and easier to review.

What does this PR do?

Reset remaining pending exporter buffers on identity refresh.

Motivation

Pending encoded batches and request-buffer state may contain pre-snapshot payloads. Discarding those buffers prevents stale data from crossing the identity boundary.

Logical change

This is the 5-exporter-buffer-state layer of the stacked MicroVM identity-refresh fix. It is based on tianning.li/4-runtime-datastream-identity-gaps and keeps the subsystem-specific reset behavior isolated for review.

Jira

SVLS-9768

Tests

./node_modules/.bin/mocha packages/dd-trace/test/exporters/agent/writer.spec.js packages/dd-trace/test/exporters/common/request.spec.js packages/dd-trace/test/exporters/common/writer.spec.js

Additional Notes

This PR is intentionally opened as a draft and is part of a linear stack.

@dd-octo-sts

dd-octo-sts Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 8.97 MB
Deduped: 9.7 MB
No deduping: 9.7 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.5.1 | 127.66 kB | 513.61 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@litianningdatadog
litianningdatadog requested a lite review from Copilot September 14, 2026 20:17
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

Pipelines  Tests

✨ Unblock PR with BitsAI

⚠️ Warnings

❌ Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 664 Pipeline jobs failed

System Tests | main / End-to-end #1 / google_genai-js@1.34.0 1 — ❌ 17 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌ tests.integration_frameworks.llm.google_genai.test_google_genai_apm.TestGoogleGenAiEmbedContent.test_embed_content[google_genai-js@1.34.0] from system_tests_suite
failed on setup with "Container nodejs-test-library-google_genai-js-1.34.0-85d76451fc341be9 status is exited. Please check logs."

Container nodejs-test-library-google_genai-js-1.34.0-85d76451fc341be9 status is exited. Please check logs.
❌ tests.integration_frameworks.llm.google_genai.test_google_genai_apm.TestGoogleGenAiGenerateContent.test_generate_content[False, google_genai-js@1.34.0] from system_tests_suite
failed on setup with "Container nodejs-test-library-google_genai-js-1.34.0-94fbaa1e31074cb5 status is exited. Please check logs."

Container nodejs-test-library-google_genai-js-1.34.0-94fbaa1e31074cb5 status is exited. Please check logs.
❌ tests.integration_frameworks.llm.google_genai.test_google_genai_apm.TestGoogleGenAiGenerateContent.test_generate_content[True, google_genai-js@1.34.0] from system_tests_suite
failed on setup with "Container nodejs-test-library-google_genai-js-1.34.0-e9be9324ed07ffbb status is exited. Please check logs."

Container nodejs-test-library-google_genai-js-1.34.0-e9be9324ed07ffbb status is exited. Please check logs.
↳ and 14 more — View all
System Tests | main / End-to-end #1 / nextjs 1 — ❌ 12 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌ tests.appsec.test_remote_config_rule_changes.Test_BlockingActionChangesWithRemoteConfig.test_block_405[nextjs] from system_tests_suite
AssertionError: assert <RemoteConfigApplyState.UNKNOWN: 0> == <RemoteConfigApplyState.ACKNOWLEDGED: 2>
 +  where <RemoteConfigApplyState.UNKNOWN: 0> = <utils._remote_config.RemoteConfigStateResults object at 0x7f609bed3290>.state
 +    where <utils._remote_config.RemoteConfigStateResults object at 0x7f609bed3290> = <tests.appsec.test_remote_config_rule_changes.Test_BlockingActionChangesWithRemoteConfig object at 0x7f609881b920>.config_state_1
 +  and   <RemoteConfigApplyState.ACKNOWLEDGED: 2> = <enum 'RemoteConfigApplyState'>.ACKNOWLEDGED
 +    where <enum 'RemoteConfigApplyState'> = rc.ApplyState

self = <tests.appsec.test_remote_config_rule_changes.Test_BlockingActionChangesWithRemoteConfig object at 0x7f609881b920>

    def test_block_405(self):
        # normal block
...
❌ tests.appsec.test_remote_config_rule_changes.Test_Empty_Config.test_empty_config[nextjs] from system_tests_suite
AssertionError: assert <RemoteConfigApplyState.UNKNOWN: 0> == <RemoteConfigApplyState.ACKNOWLEDGED: 2>
 +  where <RemoteConfigApplyState.UNKNOWN: 0> = <utils._remote_config.RemoteConfigStateResults object at 0x7f6070b7bd40>.state
 +    where <utils._remote_config.RemoteConfigStateResults object at 0x7f6070b7bd40> = <tests.appsec.test_remote_config_rule_changes.Test_Empty_Config object at 0x7f6098818200>.config_state_1
 +  and   <RemoteConfigApplyState.ACKNOWLEDGED: 2> = <enum 'RemoteConfigApplyState'>.ACKNOWLEDGED
 +    where <enum 'RemoteConfigApplyState'> = rc.ApplyState

self = <tests.appsec.test_remote_config_rule_changes.Test_Empty_Config object at 0x7f6098818200>

    def test_empty_config(self):
>       assert self.config_state_1.state == rc.ApplyState.ACKNOWLEDGED
...
❌ tests.appsec.test_remote_config_rule_changes.Test_Invalid_Config.test_invalid_config[nextjs] from system_tests_suite
AssertionError: assert <RemoteConfigApplyState.UNKNOWN: 0> == <RemoteConfigApplyState.ACKNOWLEDGED: 2>
 +  where <RemoteConfigApplyState.UNKNOWN: 0> = <utils._remote_config.RemoteConfigStateResults object at 0x7f60705a7ef0>.state
 +    where <utils._remote_config.RemoteConfigStateResults object at 0x7f60705a7ef0> = <tests.appsec.test_remote_config_rule_changes.Test_Invalid_Config object at 0x7f60989733e0>.config_state_1
 +  and   <RemoteConfigApplyState.ACKNOWLEDGED: 2> = <enum 'RemoteConfigApplyState'>.ACKNOWLEDGED
 +    where <enum 'RemoteConfigApplyState'> = rc.ApplyState

self = <tests.appsec.test_remote_config_rule_changes.Test_Invalid_Config object at 0x7f60989733e0>

    def test_invalid_config(self):
>       assert self.config_state_1.state == rc.ApplyState.ACKNOWLEDGED
...
↳ and 9 more — View all
System Tests | main / End-to-end #10 / nextjs 10 — ❌ 6 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌ tests.appsec.rasp.test_api10.Test_API10_without_downstream_body_analysis_using_max.test_api10_res_body[nextjs] from system_tests_suite
ValueError: No span validates this test

self = <tests.appsec.rasp.test_api10.Test_API10_without_downstream_body_analysis_using_max object at 0x7f4c96970110>

    def test_api10_res_body(self):
        assert self.r.status_code == 200
        body = json.loads(self.r.text)
        assert "error" not in body
        assert int(body["status"]) == 200
>       interfaces.library.validate_one_span(self.r, validator=self.validate_absence)
...
❌ tests.appsec.smoke_tests.test_apm_standalone.Test_AppSecAPMStandalone_ApiSecurity.test_api_security_smoke[nextjs] from system_tests_suite
assert False
 +  where False = any(<generator object BaseApiSecuritySmokeTests.test_api_security_smoke.<locals>.<genexpr> at 0x7f9304260040>)

self = <tests.appsec.smoke_tests.test_apm_standalone.Test_AppSecAPMStandalone_ApiSecurity object at 0x7f9320e920f0>

    def test_api_security_smoke(self) -> None:
>       assert any(
            any(key.startswith("_dd.appsec.s.") for key in span.meta) for _, span in interfaces.agent.get_spans(self.r)
        )
E       assert False
...
❌ tests.appsec.smoke_tests.test_apm_standalone.Test_AppSecAPMStandalone_RemoteConfig.test_ip_blocking_smoke[nextjs] from system_tests_suite
assert 200 == 403
 +  where 200 = HttpResponse(status_code:200, headers:{'vary': 'RSC, Next-Router-State-Tree, Next-Router-Prefetch, Next-Url', 'content...:29 GMT', 'Connection': 'keep-alive', 'Keep-Alive': 'timeout=5', 'Transfer-Encoding': 'chunked'}, text:{"mesage":"OK"}).status_code
 +    where HttpResponse(status_code:200, headers:{'vary': 'RSC, Next-Router-State-Tree, Next-Router-Prefetch, Next-Url', 'content...:29 GMT', 'Connection': 'keep-alive', 'Keep-Alive': 'timeout=5', 'Transfer-Encoding': 'chunked'}, text:{"mesage":"OK"}) = <tests.appsec.smoke_tests.test_apm_standalone.Test_AppSecAPMStandalone_RemoteConfig object at 0x7f9320e91ee0>.r

self = <tests.appsec.smoke_tests.test_apm_standalone.Test_AppSecAPMStandalone_RemoteConfig object at 0x7f9320e91ee0>

    def test_ip_blocking_smoke(self) -> None:
>       assert self.r.status_code == 403
E       assert 200 == 403
E        +  where 200 = HttpResponse(status_code:200, headers:{'vary': 'RSC, Next-Router-State-Tree, Next-Router-Prefetch, Next-Url', 'content...:29 GMT', 'Connection': 'keep-alive', 'Keep-Alive': 'timeout=5', 'Transfer-Encoding': 'chunked'}, text:{"mesage":"OK"}).status_code
...
↳ and 3 more — View all

View all 664 failed jobs.

🧪 46 Tests failed in 2 jobs

Profiling | macos — ❌ 2 tests failed

View more details · View in GitHub Actions

❌ can be turned off by env var from Mocha Tests   View in Datadog
Cannot access 'connectionOptions' before initialization
ReferenceError: Cannot access 'connectionOptions' before initialization
    at request (packages/dd-trace/src/exporters/common/request.js:140:3)
    at sendData (packages/dd-trace/src/telemetry/send-data.js:181:3)
    at Object.updateConfig (packages/dd-trace/src/telemetry/telemetry.js:383:5)
    at Object.updateConfig (packages/dd-trace/src/telemetry/index.js:17:15)
    at #applyCalculated (packages/dd-trace/src/config/index.js:754:15)
    at new Config (packages/dd-trace/src/config/index.js:251:26)
    at getConfig (packages/dd-trace/src/config/index.js:850:22)
    at getConfigFresh (packages/dd-trace/test/helpers/config.js:17:18)
...
❌ should be on by default from Mocha Tests   View in Datadog
Cannot access 'connectionOptions' before initialization
ReferenceError: Cannot access 'connectionOptions' before initialization
    at request (packages/dd-trace/src/exporters/common/request.js:140:3)
    at sendData (packages/dd-trace/src/telemetry/send-data.js:181:3)
    at Object.updateConfig (packages/dd-trace/src/telemetry/telemetry.js:383:5)
    at Object.updateConfig (packages/dd-trace/src/telemetry/index.js:17:15)
    at #applyCalculated (packages/dd-trace/src/config/index.js:754:15)
    at new Config (packages/dd-trace/src/config/index.js:251:26)
    at getConfig (packages/dd-trace/src/config/index.js:850:22)
    at getConfigFresh (packages/dd-trace/test/helpers/config.js:17:18)
...
Profiling | ubuntu — ❌ 44 tests failed

View more details · View in GitHub Actions

❌ can not be turned on by env var from Mocha Tests   View in Datadog
Cannot access 'connectionOptions' before initialization
ReferenceError: Cannot access 'connectionOptions' before initialization
    at request (packages/dd-trace/src/exporters/common/request.js:140:3)
    at sendData (packages/dd-trace/src/telemetry/send-data.js:181:3)
    at Object.updateConfig (packages/dd-trace/src/telemetry/telemetry.js:383:5)
    at Object.updateConfig (packages/dd-trace/src/telemetry/index.js:17:15)
    at #applyCalculated (packages/dd-trace/src/config/index.js:754:15)
    at new Config (packages/dd-trace/src/config/index.js:251:26)
    at getConfig (packages/dd-trace/src/config/index.js:850:22)
    at getConfigFresh (packages/dd-trace/test/helpers/config.js:17:18)
...
❌ does not warn when left at its default from Mocha Tests   View in Datadog
Cannot access 'connectionOptions' before initialization
ReferenceError: Cannot access 'connectionOptions' before initialization
    at request (packages/dd-trace/src/exporters/common/request.js:140:3)
    at sendData (packages/dd-trace/src/telemetry/send-data.js:181:3)
    at Object.updateConfig (packages/dd-trace/src/telemetry/telemetry.js:383:5)
    at Object.updateConfig (packages/dd-trace/src/telemetry/index.js:17:15)
    at #applyCalculated (packages/dd-trace/src/config/index.js:754:15)
    at new Config (packages/dd-trace/src/config/index.js:251:26)
    at getConfig (packages/dd-trace/src/config/index.js:850:22)
    at getConfigFresh (packages/dd-trace/test/helpers/config.js:17:18)
...
❌ should accept known methods from Mocha Tests   View in Datadog
Cannot access 'connectionOptions' before initialization
ReferenceError: Cannot access 'connectionOptions' before initialization
    at request (packages/dd-trace/src/exporters/common/request.js:140:3)
    at sendData (packages/dd-trace/src/telemetry/send-data.js:181:3)
    at Object.updateConfig (packages/dd-trace/src/telemetry/telemetry.js:383:5)
    at Object.updateConfig (packages/dd-trace/src/telemetry/index.js:17:15)
    at #applyCalculated (packages/dd-trace/src/config/index.js:754:15)
    at new Config (packages/dd-trace/src/config/index.js:251:26)
    at getConfig (packages/dd-trace/src/config/index.js:850:22)
    at getConfigFresh (packages/dd-trace/test/helpers/config.js:17:18)
...
↳ and 41 more — View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c21a740 | Docs | View more details | Give us feedback!

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-14T20:32:07.116233Z c21a740 Draft marked ready
🔒 Security Review ✅ Completed 2026-09-14T20:36:26.921516Z c21a740 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@litianningdatadog litianningdatadog added the aws-microvm AWS MicroVM related features label Sep 14, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical runtime and test defects remain, and an agent retry path is not reset-aware.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Resets pending exporter buffers and retry state during identity refresh to prevent stale payloads from being retried.

Changes:

  • Adds per-writer reset controllers and retry-generation checks.
  • Propagates reset controllers through common and agent exporters.
  • Adds request, writer, and propagation tests.
File summaries
File Summary
packages/dd-trace/test/exporters/common/writer.spec.js Adds writer reset behavior tests.
packages/dd-trace/test/exporters/common/request.spec.js Adds retry reset tests; the log stub lacks NoTransmitError.
packages/dd-trace/test/exporters/agent/writer.spec.js Verifies controller propagation.
packages/dd-trace/src/exporters/common/writer.js Integrates reset behavior for encoder and request state.
packages/dd-trace/src/exporters/common/request.js Adds retry reset control; contains a connectionOptions initialization error.
packages/dd-trace/src/exporters/agent/writer.js Propagates reset state, but test-optimization retry paths remain outside the controller.
Review details

Suppressed comments (1)

packages/dd-trace/src/exporters/agent/writer.js:155

  • When isTestOptimization is true, this request is ci-visibility/exporters/request, not commonRequest. That implementation owns its own retryTimer and backpressure retry queue and never registers them in this controller, so a refresh while one is pending still retries the captured pre-refresh payload. Integrate the controller into that retry lifecycle (including backpressure) or use a controller-aware transport.
    resetController,
  • Files reviewed: 6/6 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +140 to 142
delete connectionOptions.resetController

const connectionOptions = { ...options, agent }
},
http: httpStub,
'./docker': docker,
'../../log': log,

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c21a740364

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

return
}
}
delete connectionOptions.resetController

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P0 Badge Construct connection options before deleting the controller

Every normal Buffer/string/array request reaches this statement before the const connectionOptions declaration on line 142, so evaluating the binding in its temporal dead zone throws a synchronous ReferenceError. As a result, common exporter requests never reach client.request, callbacks are not invoked, and default agent trace delivery is broken; create connectionOptions first and then delete its internal resetController property.

AGENTS.md reference: AGENTS.md:L107-L109

Useful? React with 👍 / 👎.

@litianningdatadog
litianningdatadog marked this pull request as draft September 14, 2026 20:25
@litianningdatadog
litianningdatadog marked this pull request as ready for review September 14, 2026 20:29
@litianningdatadog
litianningdatadog marked this pull request as draft September 14, 2026 21:41

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aws-microvm AWS MicroVM related features semver-patch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants