feat(debugger): coordinate snapshot sampling per trace

[watson]

What does this PR do?

Makes the sampling decision for snapshot probes once per trace instead of once per probe hit, so a trace emits the snapshots of all of its probes or none of them.

on snapshot-probe hit (after its condition matched)
  trace = trace of the active span
  if there is no trace, or all of its spans have finished
    sample independently: per-probe + global rate limits       # unchanged
  else if the trace has no decision yet
    decide with the per-probe + global rate limits → EMIT or DROP for the whole trace
  else if the trace was dropped, or this probe already emitted in it
    skip (counted as rateLimitProbe)
  else
    emit: counts toward both rate limits, but isn't limited by them

• Who decides: the first snapshot-producing probe hit in a trace (capture snapshot or capture expressions), using its own snapshotsPerSecond and the global 25/s snapshot limit.
• Once per probe per trace: in a sampled trace, each probe emits once, so a probe in a loop no longer emits a snapshot per sampling period for the rest of the request.
• Rate limits: snapshots emitted after the decision bypass both rate limits, so the set stays complete, but are still counted toward them. Once the global window is used up, later traces are dropped, so the overall volume stays close to the limit.
• Unchanged: probes hit without an active span, probes that don't produce snapshots, and condition error reporting.

The decision is made by the main-thread sampler inside the breakpoint condition, before the thread pauses, so a dropped trace costs no pause. It's keyed in a WeakMap on span.context()._trace, the object shared by all spans of a trace in this process, so it's released together with the trace.

Motivation

DEBUG-5831, from the RFC: Improving Correlation for Live Debugger Snapshots. With every probe sampling on its own, related snapshots from one request arrive incomplete, with no signal that anything is missing, which misleads both users and AI agents reading a session.

The decision semantics match Java (DataDog/dd-trace-java#12452), with two Node-specific differences:

• Emitted snapshots count toward the rate limits. In Java they don't. In Node every snapshot pauses the main thread, so the global limit should reflect the actual load.
• A trace whose spans have all finished counts as no trace. In Node, callbacks bound to a request that has ended (connection pools, event emitters, timers) keep running in its async context. Coordinating those hits would let the old request's decision silence the probe, or limit it to one snapshot, for as long as the context lives.

Additional Notes

• Behavior users can notice:
  • A snapshot probe in a loop emits once per request.
  • A probe hit after another probe in a sampled trace emits even if its own snapshotsPerSecond would have skipped it.
  • Probes in a dropped trace are skipped even if their own rate would have allowed them.
• Sessions: probes are coordinated regardless of Live Debugger session, like in Java. Node has no session-scoped sampling today.
• Async context inside breakpoint conditions: before relying on it, a throwaway script set a breakpoint from a worker connected to the main thread, the way DI does, and read the active span from inside the breakpoint condition. It returned the expected span and _trace on Node 18, 20, 22 (with and without --experimental-async-context-frame), 24 and 26. Scenarios covered: no span, synchronous, child span, activate(null), after await, setTimeout, a stale callback after the trace finished, and 20 concurrent HTTP requests interleaving across awaits.
• System tests: Node needs line-probe variants of the method-probe-based Test_Debugger_Coordinated_Sampling and new weblog endpoints. That will be a follow-up system-tests PR once test(debugger): add snapshot-correlation gate and Go weblog endpoints system-tests#7425 has landed.

Performance (Node 24.14.1, macOS arm64):

The sampler call itself (makeSampleDecision for a snapshot probe; 2M hits per run, median of 15 runs):

Scenario	master	this PR	Δ
No active span, rate limited	30.6 ns	31.6 ns	+1.0 ns
Active span, dropped trace (master: rate limited)	31.3 ns	41.6 ns	+10.3 ns
Active span, probe already emitted in the trace (master: rate limited)	31.1 ns	42.1 ns	+11.0 ns

A whole rejected breakpoint hit, from V8's debug break through condition evaluation, costs about 100–300 µs, so the extra ~10 ns can't be measured end to end. Inside an active span, master vs this PR measured −0.4% (20k hits per run, median of 11 runs). The same setup with master against itself measured −6.8%, which gives the noise floor.

Existing sirun debugger variants, before vs after (these run without an active span):

Variant	master ops/s	this PR ops/s	Δ
line-probe-with-snapshot-default	506.5	514.5	+1.6%
line-probe-with-snapshot-minimal	2646.7	2667.1	+0.8%
line-probe-without-snapshot	3755.0	3675.6	−2.1%

For the minimal and without-snapshot variants, the local runs relaxed STARTUP_GUARD_MAX_SHARE: run outside the CI harness, sirun counts startup time toward that guard.

Validation:

• ./node_modules/.bin/mocha packages/dd-trace/test/debugger/probe_sampler.spec.js: 57 passing, 15 of them new.
• npm run test:debugger: 836 passing.
• ./node_modules/.bin/mocha --timeout 60000 "integration-tests/debugger/*.spec.js": 130 passing. The new coordinated-sampling.spec.js also passed 5 runs in a row.
• Mutation checks:
  • No coordination: 7 unit tests and 2 integration tests fail.
  • No finished-trace fallback: 1 unit test and 1 integration test fail.
  • No per-trace cap: 3 unit tests and 1 integration test fail.
  • Each of these is caught by at least one unit test: emitted snapshots not counted toward the global or per-probe limit, emitted snapshots rate limited, a full shared buffer recording a drop, and keying the cap on the probe id instead of the sampling index.
• ESLint on all changed files. tsc -p tsconfig.debugger.json reports no new errors.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-07T18:50:34.936333Z	9ce8658	PR opened
🔒 Security Review	✅ Completed 2026-10-07T18:54:53.480239Z	9ce8658	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[dd-octo-sts]

Overall package size

Self size: 9.45 MB
Deduped: 10.2 MB
No deduping: 10.2 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.5.1 | 127.66 kB | 531.94 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.87%. Comparing base (a505931) to head (9ce8658).
⚠️ Report is 7 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #10677    +/-   ##
========================================
  Coverage   98.87%   98.87%            
========================================
  Files        1058     1058            
  Lines      171887   171989   +102     
  Branches       74       74            
========================================
+ Hits       169949   170051   +102     
  Misses       1938     1938            

Flag	Coverage Δ
ai-guard	63.76% <ø> (-0.02%)	⬇️
apm-capabilities	54.70% <13.88%> (-0.03%)	⬇️
apm-integrations	80.98% <ø> (-0.02%)	⬇️
appsec	57.88% <ø> (-0.01%)	⬇️
debugger	69.94% <100.00%> (+0.04%)	⬆️
instrumentation	55.36% <ø> (-0.01%)	⬇️
llmobs	80.70% <ø> (-0.02%)	⬇️
master-coverage	98.87% <100.00%> (?)
openfeature	67.34% <ø> (-0.02%)	⬇️
platform	68.57% <ø> (+<0.01%)	⬆️
profiling	65.98% <ø> (-0.02%)	⬇️
serverless	65.59% <ø> (-0.02%)	⬇️
test-optimization	81.86% <ø> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[datadog-prod-us1-5]

✅ All CI checks and tests passed. Datadog automation helped this PR pass.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog retried 1 test - 1 passed on retry

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 98.29% (+0.02%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 9ce8658 | Docs | View more details | Give us feedback!}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-10-07 19:01:39

Comparing candidate commit 9ce8658 in PR branch watson/DEBUG-5831/coordinated-snapshot-sampling with baseline commit a505931 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2386 metrics, 17 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-with-attacks-20

• unstable max_rss_usage [-14.637MB; +4.329MB] or [-8.473%; +2.506%]

scenario:debugger-line-probe-with-snapshot-minimal-24

• unstable cpu_user_time [-349.601ms; +551.013ms] or [-3.919%; +6.177%]

scenario:debugger-line-probe-without-snapshot-24

• unstable cpu_user_time [-855.856ms; +1495.163ms] or [-9.997%; +17.465%]
• unstable execution_time [-870.768ms; +1515.275ms] or [-9.000%; +15.661%]
• unstable throughput [-164.425op/s; +94.414op/s] or [-13.969%; +8.021%]

scenario:debugger-line-probe-without-snapshot-26

• unstable cpu_user_time [-903.413ms; +2091.795ms] or [-9.920%; +22.970%]
• unstable execution_time [-899.966ms; +2096.808ms] or [-8.797%; +20.496%]
• unstable throughput [-209.740op/s; +90.184op/s] or [-18.769%; +8.070%]

scenario:llmobs-encode-unicode-ascii-20

• unstable cpu_usage_percentage [-13.405%; +8.616%]
• unstable execution_time [-202.445ms; +308.851ms] or [-12.041%; +18.369%]
• unstable throughput [-2085.926op/s; +1394.381op/s] or [-14.416%; +9.637%]

scenario:openfeature-scale-full-24

• unstable max_rss_usage [-29.361MB; +40.320MB] or [-10.553%; +14.493%]

scenario:openfeature-scale-full-26

• unstable max_rss_usage [-28.238MB; +19.542MB] or [-11.036%; +7.637%]

scenario:openfeature-typical-full-20

• unstable max_rss_usage [-10.421MB; +6.480MB] or [-6.989%; +4.346%]

scenario:plugin-claude-agent-sdk-delayed-noisy-stream-scan-26

• unstable execution_time [-175.367ms; +202.563ms] or [-5.280%; +6.099%]

scenario:plugin-dns-lookup-24

• unstable execution_time [-102.363ms; +150.853ms] or [-4.399%; +6.483%]

scenario:plugin-graphql-long-with-depth-on-max-26

• unstable max_rss_usage [-38.086MB; +23.372MB] or [-14.764%; +9.060%]