feat(openfeature): protect flagevaluation PII

[vjfridge]

Description

Implements protected-by-default OpenFeature flagevaluation telemetry.

• Hash valid non-empty targeting keys as sha256_<lowercase hex> over exact UTF-8 bytes. Preserve explicit empty strings; omit missing or invalid keys without dropping evaluation counts.
• Omit evaluation context by default. Only exact root observeFullEvaluationData: true permits raw valid targeting keys and a bounded immutable context snapshot.
• Emit only approved OpenFeature error codes in both privacy modes. Full-data consent never permits raw evaluator/exception text.
• Bind consent to the configuration used for the evaluation; carry it through private metadata, queueing, aggregation, and serialization. Full-tier identity includes consent; degraded identity excludes it and always omits key/context. Both tiers retain defensive AND-folds.
• Preserve asynchronous bounded delivery, existing OTel metrics/exposures, DoLog independence, the kill switch, and the span helper's bare digest format.
• Count invalid-key omissions separately from dropped rows and caught hook failures. Telemetry-sink failures cannot interrupt delivery.

Supersedes the stale implementation in #19554 and builds on #19881.

Context: FFL-2964, FFL-2529, approved PII RFC, shared implementation guidance.

Testing

Validation refreshed on 2026-09-18 for 1bfd2eab9d36c935999c261bbade7c37a5f4f9cc.

• OpenFeature: 848 passed, 2 skipped in each CPython 3.11 dependency selection (pinned 0.8 and latest).
• Six isolated fault-injection checks detect bypassed context guards, queue aliasing, missing full-tier folding, unsafe serializer errors, and missing hook error-code forwarding.
• Full mypy: 1,139 source files passed. Targeted style, spelling, security, error-log, suitespec, and SLO ownership checks passed. The broad scripts/lint checks entrypoint hit generated cache files in its Cython scan; source-scoped checks were run separately.
• Exact ARM64 wheels compared with main 63de8eb2d675e2ebe8428f058b01249674ab5079: dogfooding 18/18 HTTP 200 and 18 evaluations on both sides, no corrupt rows. Installed candidate writer checksum matches source.
• Installed-wheel error-only canaries: candidate passes both consent modes and both tiers, preserving default counts; main leaks the canary in all four cases.
• System tests: candidate 11 focused passed; 46 full passed, 2 skipped, 4 existing XPASS. Main full has the two expected privacy failures. Its initial focused degradation failure passed in the full run and isolated rerun; candidate passed first try.
• All ten benchmark means meet SLOs. Protected enqueue/aggregation measure 18.27/12.20 µs, with 24/16 µs thresholds. Legacy full-data scenarios retain their original work and limits. Full aggregate-typical is 28.75 vs 27.65 µs (+3.96%), close to its unchanged 29 µs limit; do not interpret protected improvements as same-mode optimization.
• Five review lenses plus an adversarial pass and a read-only Codex sweep completed. The SRE finding about telemetry failure losing a flush was fixed and rechecked. No remaining actionable local findings.

See the latest two Note to self comments for harness commits, wheel hashes, commands, and first-run/retry counts. New remote CI must finish independently of this local evidence.

Risks / Additional Notes

This intentionally changes the default wire shape. Deterministic unsalted hashes preserve cross-SDK subject joins but remain susceptible to dictionary attacks for low-entropy keys.

Protected mode does not read/copy context attributes; hashing remains in the background worker. flagevaluation.targeting_key.omitted{reason:invalid} counts rejected inputs that reach aggregation, not transmitted rows or dropped evaluations. flagevaluation.hook.errors counts caught hook failures separately from writer-owned drops.

All 16 review threads have replies. Three contract/design discussions remain open for acknowledgement: code-only errors in full mode, retaining the degraded defensive fold, and the approved __dd_ metadata prefix. Empty-string preservation already matches the approved amendment and remains unchanged.

[datadog-prod-us1-6]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a65eeb1 | Docs | View more details | Give us feedback!}

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 1 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector

[cit-pr-commenter-54b7da]

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

No remaining files require a CODEOWNERS review.

[cit-pr-commenter-54b7da]

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 207 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 207 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=133)
ddtrace.internal.opentelemetry.context -×-> ddtrace.trace  (product:opentelemetry -> product:tracing, score=131)
ddtrace.internal.openfeature._span_enrichment -×-> ddtrace.trace  (product:openfeature -> product:tracing, score=131)
ddtrace.llmobs._integrations.base -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=131)
ddtrace.llmobs._integrations.openai -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=131)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-09-22 18:12:06

Comparing candidate commit a65eeb1 in PR branch vickie/FFL-2964/pii-protection-refresh with baseline commit 40dcf9f in branch main.

📊 Benchmarking dashboard

Found 2 performance improvements and 1 performance regressions! Performance is the same for 588 metrics, 10 unstable metrics, 7 known flaky benchmarks, 17 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:msgpackencoderscenario-simple_one_span

• 🟥 execution_time [+615.928ns; +661.802ns] or [+18.488%; +19.865%]

scenario:openfeatureflagevaluation-aggregate-protected-typical

• 🟩 execution_time [-13.642µs; -13.505µs] or [-52.949%; -52.416%]

scenario:openfeatureflagevaluation-hook-enqueue-protected-typical

• 🟩 execution_time [-12.804µs; -12.583µs] or [-41.540%; -40.822%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

• unstable execution_time [-808.668ns; +675.630ns] or [-7.238%; +6.047%]

scenario:coreapiscenario-core_dispatch_1_listener

• unstable execution_time [-35.552ns; +30.068ns] or [-5.805%; +4.910%]

scenario:coreapiscenario-core_dispatch_50_listeners

• unstable execution_time [-2023.264ns; +1244.039ns] or [-11.804%; +7.258%]

scenario:coreapiscenario-core_dispatch_exception_listeners

• unstable execution_time [-1050.497ns; +1385.647ns] or [-8.082%; +10.660%]

scenario:coreapiscenario-core_dispatch_listeners

• unstable execution_time [-322.680ns; +318.531ns] or [-8.766%; +8.654%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

• unstable execution_time [-311.606ns; +194.349ns] or [-10.546%; +6.578%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

• unstable execution_time [-70.806ns; +77.266ns] or [-6.116%; +6.674%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

• unstable execution_time [-3789.479ns; +4236.085ns] or [-9.252%; +10.342%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

• unstable execution_time [-805.426ns; +745.890ns] or [-9.613%; +8.903%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

• unstable execution_time [-58.590µs; +53.841µs] or [-9.918%; +9.114%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:httppropagationinject-ids_only

• 🟥 execution_time [+2.589µs; +2.779µs] or [+15.407%; +16.538%]

scenario:iastaspects-rstrip_aspect

• 🟥 execution_time [+91.651µs; +95.602µs] or [+23.845%; +24.873%]

scenario:iastaspects-swapcase_aspect

• 🟥 execution_time [+34.666µs; +39.616µs] or [+12.296%; +14.052%]

scenario:iastaspectsospath-ospathbasename_aspect

• 🟥 execution_time [+120.117µs; +125.066µs] or [+28.380%; +29.550%]

scenario:span-start

• 🟥 execution_time [+1.755ms; +1.908ms] or [+12.466%; +13.554%]

scenario:telemetryaddmetric-1-count-metric-1-times

• 🟥 execution_time [+587.855ns; +642.697ns] or [+21.287%; +23.273%]

scenario:tracer-small

• 🟥 execution_time [+34.824µs; +37.767µs] or [+10.773%; +11.683%]

Known flaky benchmarks without significant changes:

• scenario:errortrackingflasksqli-baseline
• scenario:flasksimple-iast-get
• scenario:iastaspects-casefold_aspect
• scenario:iastaspects-casefold_noaspect
• scenario:iastaspects-index_aspect
• scenario:iastaspects-ljust_noaspect
• scenario:iastaspects-lower_aspect
• scenario:iastaspects-replace_aspect
• scenario:iastaspects-title_noaspect
• scenario:iastaspects-translate_aspect
• scenario:iastaspects-translate_noaspect
• scenario:iastaspects-upper_noaspect
• scenario:iastaspectssplit-rsplit_aspect
• scenario:packagespackageforrootmodulemapping-cache_off
• scenario:packagespackageforrootmodulemapping-cache_on
• scenario:sethttpmeta-all-enabled
• scenario:telemetryaddmetric-record-100-metrics

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-16T19:49:01.498636Z	3da0dd9	Draft marked ready
🔒 Security Review	✅ Completed 2026-09-16T19:48:51.199310Z	3da0dd9	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3da0dd9626

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[aarsilv]

Thanks for taking this on!

I pointed my "aaron-in-the-loop" agentic review skill at this. No blockers, mostly cleanups and testing gaps.

It did also find that when protecting against PII, an empty targeting key does different stuff here than in Go, Java, and Ruby. We'll probably want to align on a behavior and conform. (my vote would be keep it as the empty string, but the hash of the empty string could be defensible too)

[vjfridge]

Note to self: system-tests — latest main merge

PASS for the candidate — both selections passed on their first run. No unexpected candidate regression or SDK iteration indicated by these results.

2026-09-21, workspace-vickie-boettcher-sept-2026, native Linux ARM64. Harness runner: Python 3.12; Flask weblog: CPython 3.11; Agent 7.83.1. Fresh local SDK wheels, rather than published packages or remote PR artifacts:

• Candidate: 4d9fe49cb218ae264a597f829855ac9cc0513306 (vickie/FFL-2964/pii-protection-refresh).
• Main: 40dcf9f324f08a4cc11dbeffee8f9bd9bf60f92c.
• Both wheel filenames: ddtrace-4.16.0rc1-cp311-cp311-linux_aarch64.whl.
• Candidate wheel SHA-256: bdbcc244219f5e836f3aac28a2684afed660e319622b80b8e0e6f47530f8c395.
• Main wheel SHA-256: fd9d7b499f0ecd6dc9c9b18900b9047cda33bddd76b97e0673960ef2388a0af7.

System-tests: 546eb1117c6f96c3ebfdf1095e42bfe1df758636, updated once and held fixed across both builds. Preserved the existing local-source marker, staged exactly one wheel at a time, and used the native ARM64 Flask base and freshly built native proxy:

docker buildx build --load --platform linux/arm64/v8 \
  --build-context system_tests_base_python_flask_poc=docker-image://ffl2964/flask-poc-base:arm64 \
  -f utils/build/docker/python/flask-poc.Dockerfile \
  --label system-tests-library=python \
  --label system-tests-weblog-variant=flask-poc -t system_tests/weblog .

./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION --library python \
  -F tests/ffe/test_flag_eval_evp.py -k Test_FFE_EVP_Flagevaluation
./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION --library python \
  -F tests/ffe/test_flag_eval_evp.py

The second command runs the full FFE scenario: dynamic evaluation, exposures, flag-evaluation events, and OTel evaluation metrics.

Selection	Main, first run	Candidate, first run
Focused	8 passed, 3 failed	11 passed
Full scenario	44 passed, 2 failed, 2 skipped, 4 XPASS	46 passed, 2 skipped, 4 XPASS

Main's two repeatable failures are the expected absent/false-consent hashing assertions: it still emits the raw targeting key. Its additional focused degradation failure was “Expected degraded EVP flagevaluation buckets”; degradation passed in the full run and an isolated -k Test_FFE_EVP_Flagevaluation_Degradation retry (1 passed in 41.33s). This is consistent with the intermittent baseline behavior in the previous report; the first failure is preserved in the evidence. The four XPASS cases on both builds are the existing EX-3413 exposure serial-ID gates, and the two skips are unchanged.

Candidate coverage confirms protected absent/false consent, full-data opt-in, bounded context, runtime defaults, counts, load/burst aggregation, cardinality/degradation, exposures, and OTel metrics. Its installed writer SHA-256 is 98b52ce3c95f6372c0c6451b59a6deab498ab6f14113ef99e8d83f752a9925f7, matching the local branch source. No candidate retries were needed.

Evidence: remote /tmp/ffl2964-sept21-L9elcR/system-{feature,main}-*, including preserved per-run captured traffic and JUnit reports, plus system.sh, wheel build logs, wheels, and source clones. The original local-source marker and proxy tag were restored; staged wheels and the run-created weblog tag were removed from the harness. No run containers remain; SDK and harness tracked files are clean.

[vjfridge]

Note to self: ffe-dogfooding — latest main merge

PASS — no regression found in the shared request matrix; no SDK iteration indicated by this run.

2026-09-21, workspace-vickie-boettcher-sept-2026, native Linux ARM64 / CPython 3.11. Built fresh local wheels from:

• Candidate: 4d9fe49cb218ae264a597f829855ac9cc0513306 (vickie/FFL-2964/pii-protection-refresh).
• Main: 40dcf9f324f08a4cc11dbeffee8f9bd9bf60f92c.
• Both wheel filenames: ddtrace-4.16.0rc1-cp311-cp311-linux_aarch64.whl; identity is established by source commit and SHA-256, not the version string.
• Candidate wheel SHA-256: bdbcc244219f5e836f3aac28a2684afed660e319622b80b8e0e6f47530f8c395.
• Main wheel SHA-256: fd9d7b499f0ecd6dc9c9b18900b9047cda33bddd76b97e0673960ef2388a0af7.

Harness: a9f046c7f0fa8e47b02d8ba3ac580048f0e7116b, pulled from its new home, ddoghq/ffe-dogfooding, and held fixed for both runs. Staging DD_SITE=datad0g.com, DD_ENV=dev, Agent 7.83.1. Used the exact local SDK wheels through the checked-in override:

DD_TRACE_PY_PATH=<exact-SDK-clone> docker compose \
  -f docker-compose.yml -f local/docker-compose.python.yml \
  up -d --no-build app-python
python3 /tmp/ffl2964_dogfood_runner.py

The updated harness images were built first. Reset mock intake before each identical 18-request matrix: five flag types with three subjects each, plus oversized attribute-count, long-field, and deeply nested-context cases.

Result	Main	Candidate
Provider	READY	READY
HTTP 200	18/18	18/18
Intake rows / evaluation count	18 / 18	18 / 18
Full-data rows / corrupt rows	18 / 0	18 / 0
Maximum retained context fields	256	256

All 18 logged evaluation results match, including values, reasons, variants, and error codes/messages. The intentionally invalid nested context produces the same INVALID_CONTEXT result on both builds. No candidate writer failure/drop diagnostics were found. No retries were needed.

Live Remote Configuration enables full data, so this run proves full-data parity; the companion system-tests run covers absent/false consent and hashing.

Supplemental installed-wheel provider → hook → queue → aggregation → serialized-byte checks passed for the candidate in all four consent/tier combinations. The error-only PII canary is absent, the wire error is GENERAL, and runtime-default plus evaluation_count=1 are preserved. Main emits the raw error canary in all four combinations. The candidate installed writer SHA-256 is 98b52ce3c95f6372c0c6451b59a6deab498ab6f14113ef99e8d83f752a9925f7, matching the source checkout.

Evidence: remote /tmp/ffl2964-sept21-L9elcR/dogfood-{feature,main}*, wheels/source clones, and dogfood.sh. Run-created containers were stopped and removed; original image tags were restored. SDK and harness tracked files are clean.

[vjfridge]

/code blockers

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-09-22 18:25:39 UTC ℹ️ Start processing command /code blockers

---

2026-09-22 18:25:40 UTC ℹ️ Devflow:

Checking merge blockers for #20354...

---

2026-09-22 18:25:58 UTC ℹ️ Devflow: /code blockers

Detected 1 merge blocker(s) to address:

🟠 Pending

• Merge gate rule reviewers-approval is running: 1 approval missing
  All required reviewers must approve this pull request before it can be merged. Learn more in our FAQ.

  • Needs approval from Team: apm-core-python

  Hint: Questions about this check? Reach out in #dx-source-code-management on Slack.

[brettlangdon]

might be worth setting the codeowner on benchmarks/openfeature_flagevaluation to FFE

should this be a fix instead of a feat? "don't send PII anymore" sounds more like a bug fix than a feature?

not a blocker, just how it reads to me.

[vjfridge]

might be worth setting the codeowner on benchmarks/openfeature_flagevaluation to FFE

Thanks @brettlangdon! I will fold this into my next PR!

should this be a fix instead of a feat? "don't send PII anymore" sounds more like a bug fix than a feature?

Fair, it's kindof borderline since new functionality also. I think I'll leave it as-is for now