Add sampling_rules_agent_rate regression test scenario by cecile75 · Pull Request #7813 · DataDog/system-tests

cecile75 · 2026-09-24T11:45:44Z

Motivation

When DD_TRACE_SAMPLING_RULES is configured without a catch-all rule, spans that don't match any rule fall back to the agent-driven rate-by-service sampler. That fallback should keep receiving the rates published by the agent — but a bug (fixed in dd-trace-java#12490) left it stuck at the default rate of 1.0 for the life of the process. This is a regression test to guard against that behavior across all tracers.

Changes

Add tests/test_sampling_rules_agent_rate.py (Test_SamplingRulesAgentRate): configures a non-matching sampling rule, sends bursts of traffic, and asserts rule-miss spans eventually carry the agent-published _dd.agent_psr rate instead of the default 1.0.
Register the new sampling_rules_agent_rate scenario in utils/_context/_scenarios/__init__.py, and wire it into .github/workflows/run-end-to-end.yml alongside the existing sampling_rate_capping step.
Dedup the shared mocked-tracer-response helper: extract send_mocked_tracer_responses() in utils/proxy/mocked_response.py (used by both test_sampling_rate_capping.py and the new test), typed with Sequence[MockedTracerResponse] rather than list[...] for correct covariance.
Gate the known-affected tracers in the manifests until fixed:
- manifests/java.yml: bug (APMAPI-2506), scoped to component_version: "<=1.67.0-SNAPSHOT" (the PR's fix milestone).
- manifests/python.yml: bug (APMAPI-2508), PR to fix it created but not merged yet.
- manifests/rust.yml: missing_feature, since dd-trace-rs does not report agent-published sampling rates via _dd.agent_psr.

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

github-actions · 2026-09-24T11:46:21Z

CODEOWNERS have been resolved as:

tests/test_sampling_rules_agent_rate.py                                 @DataDog/system-tests-reviewers
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
manifests/java.yml                                                      @DataDog/system-tests-reviewers
manifests/python.yml                                                    @DataDog/system-tests-reviewers
manifests/rust.yml                                                      @DataDog/system-tests-reviewers
tests/test_sampling_rate_capping.py                                     @DataDog/system-tests-reviewers
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/proxy/mocked_response.py                                          @DataDog/system-tests-core

datadog-datadog-us1-prod · 2026-09-24T11:48:19Z

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 0b1f11b | Docs | View more details | Give us feedback!}

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

cecile75 · 2026-09-24T13:13:32Z

@DataDog review

datadog-datadog-us1-prod

The most critical blocker is that the new SAMPLING_RULES_AGENT_RATE scenario fails the repository scenario merge check, preventing the PR from merging. Additionally, the test setup uses fixed sleeps totalling ~60 seconds per affected run, which wastes CI time and risks timeouts on known-bug jobs for Python and older Java weblogs.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit b9e20cd · @DataDog review to ask questions}

dd-trace-rs does not report agent-published sampling rates via _dd.agent_psr, causing test_agent_rate_applies_to_rule_miss to fail. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Cuts worst-case setup time from ~60s (15 bursts + 30s wait) to ~26s (8 bursts + 10s wait). Since bug/missing_feature manifest entries compile to xfail (not skip), setup still runs in full on jobs already known to fail, wasting CI time. Verified locally against nodejs (express4): the rate converges within 2 bursts, well under the new budget. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

dd-trace-java's dev builds report as 1.67.0-SNAPSHOT (versioning is derived from git tags, and v1.66.0 was cut on 2026-09-08). The existing "<1.67.0" boundary is strictly less than that snapshot per SemVer precedence, which the dev-mode sanity check (assert_versions_not_ahead_of_current) flags as "declared version ahead of current version" and aborts the entire pytest session before any test runs -- failing every java dev end-to-end job regardless of weblog. "<=1.67.0-SNAPSHOT" satisfies the dev-mode check (build metadata is ignored, so it's not considered ahead of the current snapshot) while still correctly classifying the current snapshot as buggy under the npm-style SemverRange used for actual xfail matching, and correctly excludes both the eventual 1.67.0 release and the next dev cycle (1.67.1-SNAPSHOT). Verified against Manifest.assert_versions_not_ahead_of_current and Manifest.get_declarations directly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

cecile75 · 2026-09-24T17:16:46Z

@DataDog review

datadog-datadog-us1-prod

The setup method in tests/test_sampling_rules_agent_rate.py uses up to eight fixed 2-second sleeps plus a 10-second interface wait instead of framework-level event waiting, adding as much as 26 seconds of serialized delay per tracer run and making the scenario sensitive to flush timing.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit dfe30ec · @DataDog review to ask questions}

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0b1f11b629

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

nccatoni

LGTM (for @DataDog/system-tests-core) but you should probably also get a review from someone familiar with the feature

adding test scenario for spans that don't match any sampling rules

32dc19e

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

cecile75 and others added 2 commits September 24, 2026 13:57

fix(lint): move type-only Sequence import into TYPE_CHECKING block

5080bdb

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Merge branch 'main' into cecile/samplingrules_agentrate

b9e20cd

datadog-datadog-us1-prod Bot reviewed Sep 24, 2026

View reviewed changes

Comment thread utils/_context/_scenarios/__init__.py

Comment thread tests/test_sampling_rules_agent_rate.py

cecile75 and others added 3 commits September 24, 2026 16:32

manifests: gate rust for SAMPLING_RULES_AGENT_RATE test

3208117

dd-trace-rs does not report agent-published sampling rates via _dd.agent_psr, causing test_agent_rate_applies_to_rule_miss to fail. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

datadog-datadog-us1-prod Bot reviewed Sep 24, 2026

View reviewed changes

Comment thread tests/test_sampling_rules_agent_rate.py

cecile75 and others added 3 commits September 25, 2026 10:29

reducing loops number from 8 to 5

fb66714

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Merge branch 'main' into cecile/samplingrules_agentrate

7c3e8d6

Merge branch 'main' into cecile/samplingrules_agentrate

0b1f11b

cecile75 marked this pull request as ready for review September 25, 2026 12:29

cecile75 requested review from a team as code owners September 25, 2026 12:29

chatgpt-codex-connector Bot reviewed Sep 25, 2026

View reviewed changes

Comment thread tests/test_sampling_rules_agent_rate.py

Comment thread tests/test_sampling_rules_agent_rate.py

nccatoni approved these changes Sep 28, 2026

View reviewed changes

mcculls approved these changes Sep 29, 2026

View reviewed changes

cecile75 requested a review from vpellan September 29, 2026 09:33

vpellan approved these changes Sep 29, 2026

View reviewed changes

cecile75 merged commit 10863ed into main Sep 29, 2026
2834 of 2836 checks passed

cecile75 deleted the cecile/samplingrules_agentrate branch September 29, 2026 12:36

Navigation Menu

Repository navigation

Add sampling_rules_agent_rate regression test scenario - #7813

Add sampling_rules_agent_rate regression test scenario#7813
cecile75 merged 9 commits into
mainfrom
cecile/samplingrules_agentrate

cecile75 commented Sep 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 24, 2026 •

edited

Loading

Uh oh!

datadog-datadog-us1-prod Bot commented Sep 24, 2026 •

edited by datadog-prod-us1-3 Bot

Loading

Uh oh!

cecile75 commented Sep 24, 2026

Uh oh!

datadog-datadog-us1-prod Bot left a comment

Uh oh!

Uh oh!

Uh oh!

cecile75 commented Sep 24, 2026

Uh oh!

datadog-datadog-us1-prod Bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

nccatoni left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

cecile75 commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

github-actions Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-us1-prod Bot commented Sep 24, 2026 • edited by datadog-prod-us1-3 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cecile75 commented Sep 24, 2026

Uh oh!

datadog-datadog-us1-prod Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cecile75 commented Sep 24, 2026

Uh oh!

datadog-datadog-us1-prod Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

nccatoni left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cecile75 commented Sep 24, 2026 •

edited

Loading

github-actions Bot commented Sep 24, 2026 •

edited

Loading

datadog-datadog-us1-prod Bot commented Sep 24, 2026 •

edited by datadog-prod-us1-3 Bot

Loading