Skip to content

[nodejs] test(debugger): enable the snapshot correlation tests - #7985

Draft
watson wants to merge 4 commits into
watson/correlation-tests-rebased-onto-mainfrom
watson/DEBUG-5831/nodejs-snapshot-correlation
Draft

watson wants to merge 4 commits into
watson/correlation-tests-rebased-onto-mainfrom
watson/DEBUG-5831/nodejs-snapshot-correlation

Conversation

@watson

@watson watson commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Coordinated snapshot sampling for Node.js (DEBUG-5831, DataDog/dd-trace-js#10677) needs the snapshot correlation tests from #7425 to run for Node.js. Node.js only supports line probes, and the Node.js weblogs don't have the correlation endpoints yet.

Important

This PR builds on #7425, which adds the tests, probe files and line-map keys extended here. #7425's branch is far behind main, so its CI fails on unrelated, outdated infrastructure: Debian security mirrors, the Go toolchain required by dd-trace-go dev, and capabilities.yml. So this PR's base is watson/correlation-tests-rebased-onto-main: #7425's two commits rebased onto main, with conflicts resolved by keeping main's side and adding #7425's new entries, plus one fix on top: #7425's using System.Runtime.CompilerServices; moved every line of the .NET DebuggerController.cs down by one, which broke all .NET line probes, so the base fully qualifies the MethodImpl attributes instead. It also includes #7572's approved commit (accepting the flat dd.trace_id key that .NET sends), which the .NET enablement in #7425 needs to pass. Once #7425 is updated or merged, this PR gets retargeted.

Changes

  • Add /debugger/correlation and /debugger/correlation/loop/:count to the Node.js Express, TypeScript Express and Fastify weblogs, mirroring the Go endpoints:
    • correlationLeaf, correlationMiddle and correlation return about 0, 400 and 800ms into the request, using real async sleeps, so trace context also crosses awaits.
    • The loop sleeps 1s per iteration, and its body and sibling lines are probed.
    • All three files keep identical line numbers.
    • The additions avoid everything the IAST rewriter instruments (+, +=, template literals and some string methods) and respond with constant strings. The debugger scenarios run with IAST enabled, and when the rewriter rewrites a file, Dynamic Instrumentation places line probes in it two lines too early. An earlier revision used template literals, which broke the line probes of the existing debugger tests too.
  • In test_debugger_snapshot_correlation.py, both test classes read their probes through a small shared base class. For Node.js, it points each method probe at the line its method returns on, the same approach as the Node.js-only method-to-line conversions in the condition-error tests.
  • Add the Node.js line-map entries: Correlation (178), CorrelationMiddle (184), CorrelationLeaf (188), CorrelationLoopBody (194) and CorrelationLoopSibling (197). The new routes move CaptureTimeout from 157 to 159 and Budgets from 163 to 165.
  • In manifests/nodejs.yml, enable Test_Debugger_Coordinated_Sampling and test_per_span_budget from >=6.21.0 || ^5.132.0, the next release, which should ship feat(debugger): coordinate snapshot sampling per trace dd-trace-js#10677. The two Test_Debugger_Snapshot_Correlation_Fields tests stay missing_feature. The runtime_id one can be enabled once feat(debugger): include runtime_id in snapshot payloads dd-trace-js#10678 ships.
  • Refresh base-images.lock.json and the Node.js entries of the image mirror files, because the fixture changes alter the express4, express4-typescript, express5 and fastify base-image hashes. The images are published.

Testing

Docker wasn't available locally. Instead, the real express/debugger/index.js routes ran under dd-trace-js with a fake agent, replaying both scenarios with the same probes (line probes on 178/184/188 at 1 snapshot/s, 30 sequential requests; loop probes with default sampling and /loop/3), with IAST enabled as in CI:

dd-trace-js /debugger/correlation ×30 /debugger/correlation/loop/3
DataDog/dd-trace-js#10677 15 traces with snapshots: 15 full chains, 0 partial body: 1 snapshot, sibling: 1 snapshot
master 15 traces with snapshots: 15 full chains, 0 partial body: 3 snapshots, sibling: 1 snapshot

Warning

For Node.js, Test_Debugger_Coordinated_Sampling passes without coordinated sampling too. Each request takes nearly the same time (about 840ms), and every probe has the same 1 snapshot/s limit, so each probe sees the same time since its own last snapshot. Independent per-probe sampling then stays in lockstep and emits whole chains anyway. Making it discriminating would need the probes' sampling to drift apart, for example different snapshotsPerSecond per probe, or one probe also hit outside the chain. DataDog/dd-trace-js#10677's own integration test does the latter. I left that for the RFC owner to decide rather than changing the shared test here. test_per_span_budget does discriminate.

With IAST enabled and disabled, a snapshot probe on each mapped Node.js line (20, 159, 165, 178, 184, 188, 194 and 197) pauses on exactly that line.

Checks:

  • ./format.sh --check: mypy, import policy, ruff, yamlfmt, yamllint, the manifest parser and shellcheck pass. The Node.js linter step needs Docker, so I ran ESLint from the Express and Fastify weblog setups directly on their debugger/index.js: both pass.
  • express4-typescript/debugger/index.ts type-checks with tsc --strict against a minimal Express type stub, since the weblog's dependencies aren't installed locally.

Expected CI failures:

Before merging

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on you PR until the CI passes
  3. Mark it as ready for review
    • Test logic is modified? -> Get a review from RFC owner.
    • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

manifests/nodejs.yml                                                    @DataDog/system-tests-reviewers
mirror_images.lock.yaml                                                 @DataDog/system-tests-core
mirror_images.yaml                                                      @DataDog/system-tests-core
tests/debugger/test_debugger_snapshot_correlation.py                    @DataDog/debugger
tests/debugger/utils.py                                                 @DataDog/debugger
utils/build/docker/base-images.lock.json                                @DataDog/system-tests-reviewers
utils/build/docker/nodejs/express/debugger/index.js                     @DataDog/system-tests-reviewers
utils/build/docker/nodejs/express4-typescript/debugger/index.ts         @DataDog/system-tests-reviewers
utils/build/docker/nodejs/fastify/debugger/index.js                     @DataDog/system-tests-reviewers

@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Tests

🎉 All green!

❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3d3ef41 | Docs | View more details | Give us feedback!

@watson
watson force-pushed the watson/DEBUG-5831/nodejs-snapshot-correlation branch from 9380af5 to 71810fd Compare October 7, 2026 19:43
@watson watson changed the title [nodejs] test(debugger): run the snapshot correlation tests for Node.js [nodejs@watson/DEBUG-5831/coordinated-snapshot-sampling] [nodejs] test(debugger): enable the snapshot correlation tests Oct 7, 2026
@watson
watson changed the base branch from grantseltzer/correlation-tests-go to watson/correlation-tests-rebased-onto-main October 7, 2026 19:45
@watson
watson force-pushed the watson/DEBUG-5831/nodejs-snapshot-correlation branch from 497ee58 to 42f1063 Compare October 8, 2026 02:17
watson and others added 4 commits October 8, 2026 00:14
Add the `/debugger/correlation` and `/debugger/correlation/loop/:count`
endpoints to the Node.js Express, TypeScript Express and Fastify weblogs,
mirroring the Go endpoints: the probed functions return 400ms apart, and
the loop sleeps for a second per iteration.

Node.js doesn't support method probes, so the correlation tests point each
probe at the line its method returns on for Node.js instead, using new
Node.js entries in the line map. The new routes move the capture timeout
line from 157 to 159, and the budgets line from 163 to 165.

Enable `Test_Debugger_Coordinated_Sampling` and `test_per_span_budget`
from the dd-trace-js release that ships coordinated snapshot sampling
(DataDog/dd-trace-js#10677).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Point the image mirror at the Node.js base images published for the new
debugger fixtures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The debugger scenarios run the Node.js weblogs with IAST enabled. The IAST
rewriter rewrites any file that contains string concatenation, template
literals or certain string methods, and Dynamic Instrumentation then places
line probes in that file two lines too early. The template literals and
`+=` in the new correlation endpoints made the rewriter rewrite all of
`debugger/index.js`, which broke the line probes of the existing debugger
tests too, e.g. the capture timeout probe paused inside the callback above
its line.

Write the endpoints without anything the rewriter instruments: respond with
constant strings, and probe a plain assignment in the loop body, like the
budgets fixture does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Point the image mirror at the Node.js base images published for the
debugger fixture fix.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@watson
watson force-pushed the watson/DEBUG-5831/nodejs-snapshot-correlation branch from 42f1063 to 3d3ef41 Compare October 8, 2026 04:15

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant