Skip to content

feat(stats)!: per-field cardinality limits - #2211

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 42 commits into
mainfrom
oscarld/stats-per-key-cardinality-limit
Jul 29, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 42 commits into
mainfrom
oscarld/stats-per-key-cardinality-limit

Conversation

@Eldolfin

@Eldolfin Eldolfin commented Jul 8, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Implement per-key cardinality limits to stats computation.

Also:

  • fix overflow key peer_tags: [] -> [tracer_blocked_value]
  • fix encode_grouped_stats meta with empty value: key: -> key (breaking change ?)

Motivation

CSS spec compliance.

Additional Notes

Todo:

How to test the change?

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Jul 8, 2026 •

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 97.39%
• Overall Coverage: 74.93% (+0.16%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e8ffd6e | Docs | Datadog PR Page | Give us feedback!

@Eldolfin

Eldolfin commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@Eldolfin
Eldolfin force-pushed the oscarld/stats-per-key-cardinality-limit branch from a5a4786 to 2845fa8 Compare July 8, 2026 12:50
@Eldolfin
Eldolfin changed the base branch from main to oscarld/fix-css-obfuscation-logic July 8, 2026 12:51
@Eldolfin
Eldolfin force-pushed the oscarld/stats-per-key-cardinality-limit branch from 2845fa8 to fca0f65 Compare July 8, 2026 12:52

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a5a4786925

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread libdd-trace-stats/src/span_concentrator/aggregation.rs Outdated
Comment thread libdd-data-pipeline/src/trace_exporter/builder.rs
Comment thread libdd-trace-stats/src/span_concentrator/aggregation.rs Outdated
@dd-octo-sts

dd-octo-sts Bot commented Jul 8, 2026 •

Copy link
Copy Markdown
Contributor

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 87.20 MB 87.33 MB +.15% (+135.77 KB) 🔍
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 8.01 MB 8.01 MB +0% (+48 B) 👌
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 98.45 MB 98.59 MB +.14% (+142.12 KB) 🔍
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 10.78 MB 10.78 MB +.04% (+4.53 KB) 🔍
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 26.06 MB 26.08 MB +.10% (+28.50 KB) 🔍
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 89.18 KB 89.60 KB +.47% (+430 B) 🔍
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 188.08 MB 188.23 MB +.07% (+152.00 KB) 🔍
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 979.11 MB 980.23 MB +.11% (+1.12 MB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 8.47 MB 8.48 MB +.15% (+13.50 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 89.18 KB 89.60 KB +.47% (+430 B) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 25.05 MB 25.08 MB +.15% (+40.00 KB) 🔍
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 49.83 MB 49.90 MB +.14% (+71.80 KB) 🔍
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 22.62 MB 22.65 MB +.12% (+28.50 KB) 🔍
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 90.58 KB 91.00 KB +.46% (+434 B) 🔍
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 192.55 MB 192.71 MB +.08% (+160.00 KB) 🔍
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 970.38 MB 971.14 MB +.07% (+778.24 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 6.54 MB 6.55 MB +.20% (+13.50 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 90.58 KB 91.00 KB +.46% (+434 B) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 26.91 MB 26.94 MB +.14% (+40.00 KB) 🔍
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 47.43 MB 47.50 MB +.14% (+71.32 KB) 🔍
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 77.70 MB 77.82 MB +.14% (+116.47 KB) 🔍
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 8.92 MB 8.93 MB +.13% (+12.06 KB) 🔍
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 93.26 MB 93.38 MB +.12% (+122.60 KB) 🔍
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 10.85 MB 10.87 MB +.13% (+15.23 KB) 🔍

@Eldolfin
Eldolfin marked this pull request as ready for review July 8, 2026 15:15
@Eldolfin
Eldolfin requested review from a team as code owners July 8, 2026 15:15

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d82ebfd172

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread libdd-data-pipeline-ffi/src/trace_exporter.rs
Comment thread libdd-trace-stats/src/span_concentrator/aggregation.rs
Comment thread libdd-trace-stats/src/span_concentrator/mod.rs Outdated
Comment thread libdd-trace-stats/src/span_concentrator/aggregation.rs Outdated
Comment thread libdd-trace-stats/src/span_concentrator/aggregation.rs Outdated
Comment thread libdd-trace-stats/src/span_concentrator/mod.rs
@Eldolfin
Eldolfin force-pushed the oscarld/stats-per-key-cardinality-limit branch from abb8d3a to efe280f Compare July 10, 2026 11:53

@VianneyRuhlmann VianneyRuhlmann left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, although I think we should make sure that this is shipped in the same release as the telemetry PR

Comment thread libdd-data-pipeline-ffi/src/trace_exporter.rs
@gh-worker-ownership-write-b05516
gh-worker-ownership-write-b05516 Bot removed the request for review from a team July 21, 2026 09:55
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jul 21, 2026
…2170)

# What does this PR do?

- Add additional metric tags to the stats aggregation key. For each configured key, the matching span meta tag is included in the key and populates `ClientGroupedStats.additional_metric_tags`.
- Limit the number of keys for `additional_metric_tags` to 4, not configurable.
- Limit the maximum length of individual additional_metric_tags values to 200 characters, not configurable. Values exceeding this are replaced with `tracer_blocked_value`.

# Motivation

- Allow Serverless Compatibility Layer to send additional metric tags on trace stats. See [RFC](https://datadoghq.atlassian.net/wiki/spaces/APM/pages/6482919540/PENDING+Span-Derived+Primary+Tags+-+V1).
- DataDog/serverless-components#140 to follow.
- https://datadoghq.atlassian.net/browse/SVLS-8787

# Additional Notes

Additional metric tags still need to be implemented in libdd-data-pipeline. Omitted from this PR since the goal of this PR is to support additional metric tags tags for Serverless without affecting other teams.

https://github.com/DataDog/libdatadog/blob/42e3296aa3fa76749caf5d96b5bbf1ff2b1913b6/libdd-data-pipeline/src/trace_exporter/stats.rs#L146


Per-field cardinality limit for additional metric tags will be implemented in a following PR, #2211.
- Limit the number of distinct aggregation entries per bucket that carry additional metric tags to 100 by default, configurable via `additional_metric_tags_max_entries`. Once exceeded, new entries have their tag values masked to `tracer_blocked_value`.

---

Implementing additional metric tags instead of span derived primary tags. See previous PR for span derived primary tags - #1815.

# How to test the change?

- Unit tests
- Deploy to Azure Functions with the following environment variables and validate that additional tags appear on trace metrics
  * `DD_TAGS: custom.primary:val,workload:azure_functions`
  * `DD_TRACE_EXPERIMENTAL_FEATURES_ENABLED: true`
  * `DD_TRACE_STATS_ADDITIONAL_TAGS: custom.primary,workload`

<img width="685" height="411" alt="Screenshot 2026-07-06 at 3 39 47 PM" src="https://github.com/user-attachments/assets/b91495b6-11aa-4051-922f-63655666c4e5" />


Co-authored-by: duncan.harvey <duncan.harvey@datadoghq.com>
@Eldolfin

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2906b45c52

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread libdd-trace-stats/src/span_concentrator/mod.rs
@Eldolfin Eldolfin changed the title feat(stats)!: per-key cardinality limits feat(stats)!: per-field cardinality limits Jul 22, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 72fa868 into main Jul 29, 2026
112 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the oscarld/stats-per-key-cardinality-limit branch July 29, 2026 17:05
hoolioh added a commit that referenced this pull request Aug 18, 2026
)

# Release proposal for libdd-data-pipeline, libdd-telemetry and their
dependencies

This PR contains version bumps based on public API changes and commits
since last release.

## libdd-ddsketch
**Next version:** `1.1.1`
**Semver bump:** `patch`
**Tag:** `libdd-ddsketch-v1.1.1`

### Commits

- refactor: migrate to workspace dependencies, phase 4 (#2296)
- chore: moving to workspace-level dependencies, phase 2 (#2270)

## libdd-shared-runtime
**Next version:** `3.0.0`
**Semver bump:** `major`
**Tag:** `libdd-shared-runtime-v3.0.0`

### ⚠️ major bump forced due to:

- `libdd-capabilities`: ^2.1.0 → ^3.0.0
- `libdd-capabilities-impl`: ^3.0.0 → ^4.0.0

### Commits

- feat(shared-runtime): add block_on_with_timeout to BlockingRuntime
(#2333)
- chore(runtime): restart all workers on invalid state (#2262)
- refactor: migrate to workspace dependencies, phase 3 (#2283)
- refactor: consolidate core dependencies at workspace level (phase 1)
(#2253)

## libdd-dogstatsd-client
**Next version:** `5.0.0`
**Semver bump:** `major`
**Tag:** `libdd-dogstatsd-client-v5.0.0`

### Commits

- refactor: consolidate core dependencies at workspace level (phase 1)
(#2253)
- feat(dogstatsd)!: add shared_runtime buffered sink (#2224)
- refactor(statsd)!: make client clonable (#2222)

## libdd-telemetry
**Next version:** `7.0.0`
**Semver bump:** `major`
**Tag:** `libdd-telemetry-v7.0.0`

### Commits

- refactor(rc)!: make conversion from RemoteConfigProduct back and forth
generally available (#2325)
- feat(telemetry)!: Add Installation signature and AppProduct changes
payloads (#2213)
- fix(telemetry)!: stop sending Connection: close to the Agent (#2286)
- refactor: migrate to workspace dependencies, phase 3 (#2283)
- chore: moving to workspace-level dependencies, phase 2 (#2270)
- refactor: consolidate core dependencies at workspace level (phase 1)
(#2253)
- fix: drain the mailbox before stopping (#2258)
- feat(telemetry)!: make telemetry worker wasm-compatible for the
TraceExporter (#2172)
- fix(sidecar): don't double-encode file:// telemetry endpoints (#2230)

## libdd-trace-obfuscation
**Next version:** `6.0.0`
**Semver bump:** `major`
**Tag:** `libdd-trace-obfuscation-v6.0.0`

### ⚠️ major bump forced due to:

- `libdd-trace-utils`: ^9.0.0 → ^10.1.0

### Commits

- refactor: migrate to workspace dependencies, phase 4 (#2296)
- chore(lint): update nightly (#2281)
- chore: moving to workspace-level dependencies, phase 2 (#2270)
- refactor: consolidate core dependencies at workspace level (phase 1)
(#2253)
- fix: new clippy lints (#2219)

## libdd-trace-stats
**Next version:** `7.0.0`
**Semver bump:** `major`
**Tag:** `libdd-trace-stats-v7.0.0`

### ⚠️ major bump forced due to:

- `libdd-capabilities`: ^2.1.0 → ^3.0.0
- `libdd-capabilities-impl`: ^3.0.0 → ^4.0.0
- `libdd-dogstatsd-client`: ^4.0.0 → ^5.0.0
- `libdd-telemetry`: ^6.0.0 → ^7.0.0
- `libdd-trace-utils`: ^9.0.0 → ^10.1.0

### Commits

- feat: add v1 isolated bricks for trace exporter (#2280)
- refactor: migrate to workspace dependencies, phase 4 (#2296)
- feat(data-pipeline)!: add compression option for agentless export
(#2205)
- feat(stats)!: per-field cardinality limits (#2211)
- refactor: migrate to workspace dependencies, phase 3 (#2283)
- feat(stats)!: string field length limit (#2267)
- feat(trace_exporter)!: add `restart_after_fork` configuration option
[APMSP-3843] (#2263)
- refactor: consolidate core dependencies at workspace level (phase 1)
(#2253)
- feat(stats)!: implement additional metric tags in libdd-trace-stats
(#2170)
- refactor(statsd)!: make client clonable (#2222)
- fix(stats-exporter): disable retries for stats payload sends (#2249)
- feat(telemetry)!: make telemetry worker wasm-compatible for the
TraceExporter (#2172)
- fix!: css obfuscation logic (#2203)

## libdd-data-pipeline
**Next version:** `8.0.0`
**Semver bump:** `major`
**Tag:** `libdd-data-pipeline-v8.0.0`

### ⚠️ major bump forced due to:

- `libdd-capabilities`: ^2.1.0 → ^3.0.0
- `libdd-capabilities-impl`: ^3.0.0 → ^4.0.0
- `libdd-dogstatsd-client`: ^4.0.0 → ^5.0.0
- `libdd-telemetry`: ^6.0.0 → ^7.0.0
- `libdd-trace-stats`: ^6.0.0 → ^7.0.0
- `libdd-trace-utils`: ^9.0.0 → ^10.1.0

### Commits

- fix(trace-exporter): mark OTLP client-computed stats (#2245)
- feat(data-pipeline): adds new attributes to OTLP trace metrics export
(#2316)
- feat: add v1 isolated bricks for trace exporter (#2280)
- feat(data-pipeline)!: add flush_and_close to the trace buffer (#2313)
- feat(telemetry)!: Add Installation signature and AppProduct changes
payloads (#2213)
- refactor: migrate to workspace dependencies, phase 4 (#2296)
- feat(data-pipeline)!: add compression option for agentless export
(#2205)
- fix(telemetry)!: stop sending Connection: close to the Agent (#2286)
- feat(stats)!: per-field cardinality limits (#2211)
- refactor: migrate to workspace dependencies, phase 3 (#2283)
- feat(stats)!: string field length limit (#2267)
- refactor(trace-utils)!: split generic TracerHeaderTags (#2279)
- chore: moving to workspace-level dependencies, phase 2 (#2270)
- feat(trace_exporter)!: add `restart_after_fork` configuration option
[APMSP-3843] (#2263)
- feat(data-pipeline): emit additional_metric_tags on OTLP span duration
metric (#2264)
- refactor: consolidate core dependencies at workspace level (phase 1)
(#2253)
- feat(dogstatsd)!: add shared_runtime buffered sink (#2224)
- feat(stats)!: implement additional metric tags in libdd-trace-stats
(#2170)
- refactor(statsd)!: make client clonable (#2222)
- feat(telemetry)!: make telemetry worker wasm-compatible for the
TraceExporter (#2172)
- feat(trace_exporter)!: remove deprecated methods (#2250)
- feat(trace-utils)!: add v1 decoder (#2174)
- feat(trace-utils)!: add encoder v1 to v04 + refactor (#2145)
- feat(otlp): add instrumentation scope metadata (#2235)
- fix(trace-utils)!: serialize v0.5 span links and events into meta
(#980)
- fix!: css obfuscation logic (#2203)
- test(data-pipeline): gate telemetry-only tests behind the telemetry
feature (#2204)


[APMSP-3843]:
https://datadoghq.atlassian.net/browse/APMSP-3843?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: hoolioh <107922352+hoolioh@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants