Repository navigation
feat(openfeature): emit server-side EVP flagevaluation - #18613
gh-worker-dd-mergequeue-cf854d[bot] merged 43 commits into
Conversation
…gator - Two-tier aggregation (full → degraded → drop-counted); no ultra-degraded tier - Canonical context key: sorted, type-tagged, length-delimited (NOT a hash) - Caps GLOBAL_CAP=131072 / PER_FLAG_CAP=10000 / DEGRADED_CAP=32768 - Context pruning: ≤256 fields / ≤256 chars per reviewer concern #1 - Non-blocking queue.Queue(4096) enqueue with observable drop counter - EVP POST to /evp_proxy/v2/api/v2/flagevaluations with X-Datadog-EVP-Subdomain - Unit tests: canonical key, two-tier overflow, prune, enqueue, payload shape
…ATION_COUNTS_ENABLED killswitch - FlagEvaluationHook(Hook).finally_after: cheap capture + non-blocking enqueue only - Eval-time from metadata["dd.eval.timestamp_ms"]; fallback to hook-fire time - runtime_default_used from absent/None variant (reviewer concern #5) - Provider stamps dd.eval.timestamp_ms at eval boundary (_resolve_details) - Killswitch DD_FLAGGING_EVALUATION_COUNTS_ENABLED (default on) gates EVP path only - OTel FlagEvalHook (_flageval_metrics.py) byte-for-byte unchanged (PRES-01) - Tests: hook lifecycle, killswitch gating, OTel non-regression, metadata extraction
…OVIDER_ERROR Root cause: initialize() raised ProviderNotReadyError after the 10s RC wait, causing the OpenFeature SDK to dispatch PROVIDER_ERROR and mark the provider as ERROR. In ffe-dogfooding, with no FFE flags configured in the org, RC never delivers FFE_FLAGS config, so every start ended in PROVIDER_ERROR and the container exited. Fix: on timeout, log a warning, set _status = READY, and return normally. The SDK dispatches PROVIDER_READY; evaluations degrade to (default_value, ERROR, PROVIDER_NOT_READY) via _resolve_details() until config eventually arrives. On_configuration_received() remains the authoritative path for making config available, unchanged. Update affected tests to reflect the new READY-on-timeout contract.
…ad of PROVIDER_ERROR" This reverts commit 114216b.
…icorn pre-fork workers Root cause: initialize() was blocking on _config_received.wait(timeout=10s) and raising ProviderNotReadyError on expiry. Under gunicorn (and uWSGI) pre-fork workers, the OpenFeature SDK runs initialize() in a background thread; that thread is killed by fork() in each child process, so every worker's initialize() either misses the RC notification (READY in master fires before worker registers via _register_provider) or times out waiting — causing PROVIDER_ERROR and container exit. Evidence (dogfooding logs): 22:44:31 PROVIDER_READY ← worker 1 took fast path (config in master) 22:44:41 PROVIDER_ERROR ← worker 2 timed out (10 s exactly) Fix: remove the blocking wait entirely. initialize() now returns immediately if no config is available yet, matching the original pre-#16650 contract. The async path via on_configuration_received() remains the authoritative READY transition for both the master process and each forked worker (worker RC subscriber replays FFE_FLAGS from the inherited connector queue, which calls _notify_providers_config_received() once the worker's provider is registered). The fast path (config already available at initialize() time) is preserved — this covers the case where the master received config before forking. Update tests: replace TestProviderInitializationBlocking with TestProviderInitializationAsync verifying the new non-blocking contract. Remove the fast_initialization_timeout conftest fixture (no longer needed since initialize() does not block).
…arks + e2e tests - Read DD_FLAGGING_EVALUATION_COUNTS_ENABLED through OpenFeatureConfig (ddtrace config var system) instead of raw os.environ; register it in supported-configurations.json. - Stamp eval timestamp at the provider resolution boundary so the finally_after hook receives accurate eval-time. - Normalize typing annotations (dict[...] generics) and mypy overrides for the flagevaluation writer/hook. - Add openfeature_flagevaluation microbenchmark suite (hook-enqueue / aggregate / hook-plus-drain, incl. a 2500-flag high-cardinality variant) wired into benchmarks/suitespec.yml. - Add end-to-end provider tests covering hook firing on the real eval path, all exit paths (success/error/runtime-default/disabled), and OTel non-regression.
Strip internal validation-gate markers and review-tracking references from test docstrings and comments; keep the substantive test descriptions intact.
This comment has been minimized.
This comment has been minimized.
Codeowners resolved as |
BenchmarksBenchmark execution time: 2026-07-14 19:29:10 Comparing candidate commit fddd338 in PR branch Found 0 performance improvements and 5 performance regressions! Performance is the same for 614 metrics, 10 unstable metrics. scenario:iastaspects-index_aspect
scenario:iastaspects-ljust_aspect
scenario:iastaspects-swapcase_aspect
scenario:iastaspectsospath-ospathbasename_aspect
scenario:span-start
|
This comment has been minimized.
This comment has been minimized.
Circular import analysis
|
brettlangdon
left a comment
There was a problem hiding this comment.
benchmarks and release notes lgtm
note: the benchmarks aren't being run in CI, if you want that you'll need to add SLOs for them and then added to the suitespec for benchmarks.
…46-evp-flagevaluation-python # Conflicts: # ddtrace/internal/openfeature/_provider.py
|
PR Shepherd is now watching this PR: fixing basic CI failures, rebasing when it falls behind, and re-queueing after transient merge-queue failures. To disable it, add the If you have any questions, reach the team in #ai-devx-flow. |
## Motivation
Server-side feature-flag evaluations need to be observable in the same backend track as the rest of the cross-SDK rollout so APM can judge rollout smoothness and time-to-land from real evaluation counts, not only app-local behavior or metric-side approximations. This contribution gives Python services backend-visible `flagevaluation` telemetry with bounded cardinality and batching, so customers and internal dogfooding can verify which flags evaluated, for which targeting keys, and how often, while preserving the existing `feature_flag.evaluations` metric path.
## Changes
- Adds a Python EVP `flagevaluation` writer behind `DD_FLAGGING_EVALUATION_COUNTS_ENABLED`; existing OTel `feature_flag.evaluations` metrics remain unchanged.
- Registers an OpenFeature `finally_after` logging hook that captures a cheap evaluation snapshot and hands it to a background writer.
- Renames the capture hook to `FlagEvalEVPHook` / `_flag_eval_evp_hook.py` so it is distinct from `FlagEvalMetricsHook`.
- Posts batched payloads to the Agent EVP proxy route `/evp_proxy/v2/api/v2/flagevaluation` with `X-Datadog-EVP-Subdomain: event-platform-intake`.
- Aggregates evaluations by schema-visible dimensions: flag key, variant key, allocation key, runtime-default state, error message, targeting key, and pruned evaluation context.
- Flattens and prunes evaluation context before enqueue with deterministic 256-field / 256-character bounds.
- Splits encoded `flagEvaluations` request bodies under the shared EVP payload limit before POSTing.
- Degrades a single oversized full-tier row by dropping `targeting_key` and `context`; if the degraded row still cannot fit, the writer drops and logs it.
- Uses evaluation-time metadata to populate `first_evaluation` and `last_evaluation` rather than flush time.
- Keeps accepting the legacy `DataDogProvider(initialization_timeout=...)` keyword for compatibility, while async initialization no longer uses it.
### Evaluation Capture And Flush
```mermaid
flowchart TD
Eval["OpenFeature evaluation"] --> Metrics["FlagEvalMetricsHook\nexisting OTel metric"]
Eval --> Logging["FlagEvalEVPHook\nfinally_after"]
Logging --> Snapshot["Cheap scalar snapshot\nflag, variant, allocation,\ntargeting_key, context, eval time"]
Snapshot --> Queue["Bounded non-blocking queue\nQueue size 4096"]
Queue --> Drain["Background drain worker\n0.1s queue drain"]
Drain --> Aggregate["FlagEvaluationWriter aggregation\nfull tier then degraded tier"]
Aggregate --> Flush["Periodic flush\n10s interval"]
Flush --> EVP["Agent EVP proxy\n/evp_proxy/v2/api/v2/flagevaluation"]
```
### Payload Limit Handling
```mermaid
flowchart TD
Events["Aggregated flagEvaluations"] --> Encode["Encode each event with compact JSON"]
Encode --> FitsEvent{"Single event fits\ninside shared EVP limit?"}
FitsEvent -- "yes" --> Batch["Add to current payload"]
FitsEvent -- "no" --> Degrade["Drop targeting_key and context\nfor that event only"]
Degrade --> FitsDegraded{"Degraded event fits?"}
FitsDegraded -- "yes" --> Batch
FitsDegraded -- "no" --> Drop["Drop and count\npayload_limit"]
Batch --> FitsPayload{"Payload still <=\nDEFAULT_EVP_PAYLOAD_SIZE_LIMIT?"}
FitsPayload -- "yes" --> Continue["Continue batching"]
FitsPayload -- "no" --> Split["Close current payload\nstart next payload"]
Split --> Continue
Continue --> Post["POST each payload sequentially"]
```
## Decisions
- Keep this as a dual-path implementation: the EVP logging writer is additive and separately gated, while the existing OTel metric hook remains registered when the provider is enabled.
- Use the existing Agent EVP proxy transport and the canonical `flagevaluation` backend route instead of adding a Python-specific transport or schema.
- Keep the evaluation logging hook non-blocking on application evaluation threads: queueing is best-effort, bounded, and drop-counted under backpressure.
- Do aggregation, JSON serialization, payload splitting, and POSTs only in the background writer path.
- Split `flagEvaluations` payloads by encoded byte size using `DEFAULT_EVP_PAYLOAD_SIZE_LIMIT`; send split payloads sequentially from the writer thread.
- Use a two-tier aggregation model: full buckets retain `targeting_key` and pruned context; degraded buckets retain only lower-cardinality schema dimensions.
- Store `targeting_key` in the dedicated EVP field and remove `targetingKey` / `targeting_key` from context before building context keys or payloads.
- Do not emit OpenFeature `reason` because it is not accepted by the canonical worker schema, and do not populate `targeting_rule.key` until Python exposes real targeting-rule metadata.
- Keep `openfeature-sdk` as an application/test/lint dependency rather than adding it as a runtime dependency of `ddtrace`; the dogfooding app dependency floor is being tightened separately in https://github.com/DataDog/ffe-dogfooding/pull/88.
## Validation Evidence
### Dogfooding App
- `ffe-dogfooding` `app-python` was run with local `dd-trace-py` artifacts and reached `PROVIDER_READY`.
- The running dogfooding app reported `openfeature-sdk=0.9.0`.
- Evaluated `ffe-dogfooding-string-flag` through the Python dogfooding app with 5 evaluations per targeting key over one persistent connection:
- `python-batch-keepalive-20260623T004034Z-alpha`
- `python-batch-keepalive-20260623T004034Z-bravo`
- `python-batch-keepalive-20260623T004034Z-charlie`
- App-side result: all evaluations returned `variant_2` with allocation `allocation-override-392dd7c149f8`.
### System Tests
- Companion draft PR: DataDog/system-tests#7183
### Staging End-To-End
- Dogfooding ran without the local mock-intake EVP tee/proxy, so the Agent sent EVP traffic through the normal backend path.
- Retriever staging query returned 3 backend `flagevaluation` rows for the exact targeting keys above.
- Each targeting key returned one row with `evaluation_count=5`:
- `python-batch-keepalive-20260623T004034Z-alpha`
- `python-batch-keepalive-20260623T004034Z-bravo`
- `python-batch-keepalive-20260623T004034Z-charlie`
- All rows had `flag.key=ffe-dogfooding-string-flag`, `variant.key=variant_2`, and `allocation.key=allocation-override-392dd7c149f8`.
Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>
## Motivation
Server-side feature-flag evaluations need to be observable in the same backend track as the rest of the cross-SDK rollout so APM can judge rollout smoothness and time-to-land from real evaluation counts, not only app-local behavior or metric-side approximations. This contribution gives Python services backend-visible `flagevaluation` telemetry with bounded cardinality and batching, so customers and internal dogfooding can verify which flags evaluated, for which targeting keys, and how often, while preserving the existing `feature_flag.evaluations` metric path.
## Changes
- Adds a Python EVP `flagevaluation` writer behind `DD_FLAGGING_EVALUATION_COUNTS_ENABLED`; existing OTel `feature_flag.evaluations` metrics remain unchanged.
- Registers an OpenFeature `finally_after` logging hook that captures a cheap evaluation snapshot and hands it to a background writer.
- Renames the capture hook to `FlagEvalEVPHook` / `_flag_eval_evp_hook.py` so it is distinct from `FlagEvalMetricsHook`.
- Posts batched payloads to the Agent EVP proxy route `/evp_proxy/v2/api/v2/flagevaluation` with `X-Datadog-EVP-Subdomain: event-platform-intake`.
- Aggregates evaluations by schema-visible dimensions: flag key, variant key, allocation key, runtime-default state, error message, targeting key, and pruned evaluation context.
- Flattens and prunes evaluation context before enqueue with deterministic 256-field / 256-character bounds.
- Splits encoded `flagEvaluations` request bodies under the shared EVP payload limit before POSTing.
- Degrades a single oversized full-tier row by dropping `targeting_key` and `context`; if the degraded row still cannot fit, the writer drops and logs it.
- Uses evaluation-time metadata to populate `first_evaluation` and `last_evaluation` rather than flush time.
- Keeps accepting the legacy `DataDogProvider(initialization_timeout=...)` keyword for compatibility, while async initialization no longer uses it.
### Evaluation Capture And Flush
```mermaid
flowchart TD
Eval["OpenFeature evaluation"] --> Metrics["FlagEvalMetricsHook\nexisting OTel metric"]
Eval --> Logging["FlagEvalEVPHook\nfinally_after"]
Logging --> Snapshot["Cheap scalar snapshot\nflag, variant, allocation,\ntargeting_key, context, eval time"]
Snapshot --> Queue["Bounded non-blocking queue\nQueue size 4096"]
Queue --> Drain["Background drain worker\n0.1s queue drain"]
Drain --> Aggregate["FlagEvaluationWriter aggregation\nfull tier then degraded tier"]
Aggregate --> Flush["Periodic flush\n10s interval"]
Flush --> EVP["Agent EVP proxy\n/evp_proxy/v2/api/v2/flagevaluation"]
```
### Payload Limit Handling
```mermaid
flowchart TD
Events["Aggregated flagEvaluations"] --> Encode["Encode each event with compact JSON"]
Encode --> FitsEvent{"Single event fits\ninside shared EVP limit?"}
FitsEvent -- "yes" --> Batch["Add to current payload"]
FitsEvent -- "no" --> Degrade["Drop targeting_key and context\nfor that event only"]
Degrade --> FitsDegraded{"Degraded event fits?"}
FitsDegraded -- "yes" --> Batch
FitsDegraded -- "no" --> Drop["Drop and count\npayload_limit"]
Batch --> FitsPayload{"Payload still <=\nDEFAULT_EVP_PAYLOAD_SIZE_LIMIT?"}
FitsPayload -- "yes" --> Continue["Continue batching"]
FitsPayload -- "no" --> Split["Close current payload\nstart next payload"]
Split --> Continue
Continue --> Post["POST each payload sequentially"]
```
## Decisions
- Keep this as a dual-path implementation: the EVP logging writer is additive and separately gated, while the existing OTel metric hook remains registered when the provider is enabled.
- Use the existing Agent EVP proxy transport and the canonical `flagevaluation` backend route instead of adding a Python-specific transport or schema.
- Keep the evaluation logging hook non-blocking on application evaluation threads: queueing is best-effort, bounded, and drop-counted under backpressure.
- Do aggregation, JSON serialization, payload splitting, and POSTs only in the background writer path.
- Split `flagEvaluations` payloads by encoded byte size using `DEFAULT_EVP_PAYLOAD_SIZE_LIMIT`; send split payloads sequentially from the writer thread.
- Use a two-tier aggregation model: full buckets retain `targeting_key` and pruned context; degraded buckets retain only lower-cardinality schema dimensions.
- Store `targeting_key` in the dedicated EVP field and remove `targetingKey` / `targeting_key` from context before building context keys or payloads.
- Do not emit OpenFeature `reason` because it is not accepted by the canonical worker schema, and do not populate `targeting_rule.key` until Python exposes real targeting-rule metadata.
- Keep `openfeature-sdk` as an application/test/lint dependency rather than adding it as a runtime dependency of `ddtrace`; the dogfooding app dependency floor is being tightened separately in https://github.com/DataDog/ffe-dogfooding/pull/88.
## Validation Evidence
### Dogfooding App
- `ffe-dogfooding` `app-python` was run with local `dd-trace-py` artifacts and reached `PROVIDER_READY`.
- The running dogfooding app reported `openfeature-sdk=0.9.0`.
- Evaluated `ffe-dogfooding-string-flag` through the Python dogfooding app with 5 evaluations per targeting key over one persistent connection:
- `python-batch-keepalive-20260623T004034Z-alpha`
- `python-batch-keepalive-20260623T004034Z-bravo`
- `python-batch-keepalive-20260623T004034Z-charlie`
- App-side result: all evaluations returned `variant_2` with allocation `allocation-override-392dd7c149f8`.
### System Tests
- Companion draft PR: DataDog/system-tests#7183
### Staging End-To-End
- Dogfooding ran without the local mock-intake EVP tee/proxy, so the Agent sent EVP traffic through the normal backend path.
- Retriever staging query returned 3 backend `flagevaluation` rows for the exact targeting keys above.
- Each targeting key returned one row with `evaluation_count=5`:
- `python-batch-keepalive-20260623T004034Z-alpha`
- `python-batch-keepalive-20260623T004034Z-bravo`
- `python-batch-keepalive-20260623T004034Z-charlie`
- All rows had `flag.key=ffe-dogfooding-string-flag`, `variant.key=variant_2`, and `allocation.key=allocation-override-392dd7c149f8`.
Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>
Motivation
Server-side feature-flag evaluations need to be observable in the same backend track as the rest of the cross-SDK rollout so APM can judge rollout smoothness and time-to-land from real evaluation counts, not only app-local behavior or metric-side approximations. This contribution gives Python services backend-visible
flagevaluationtelemetry with bounded cardinality and batching, so customers and internal dogfooding can verify which flags evaluated, for which targeting keys, and how often, while preserving the existingfeature_flag.evaluationsmetric path.Changes
flagevaluationwriter behindDD_FLAGGING_EVALUATION_COUNTS_ENABLED; existing OTelfeature_flag.evaluationsmetrics remain unchanged.finally_afterlogging hook that captures a cheap evaluation snapshot and hands it to a background writer.FlagEvalEVPHook/_flag_eval_evp_hook.pyso it is distinct fromFlagEvalMetricsHook./evp_proxy/v2/api/v2/flagevaluationwithX-Datadog-EVP-Subdomain: event-platform-intake.flagEvaluationsrequest bodies under the shared EVP payload limit before POSTing.targeting_keyandcontext; if the degraded row still cannot fit, the writer drops and logs it.first_evaluationandlast_evaluationrather than flush time.DataDogProvider(initialization_timeout=...)keyword for compatibility, while async initialization no longer uses it.Evaluation Capture And Flush
Payload Limit Handling
flowchart TD Events["Aggregated flagEvaluations"] --> Encode["Encode each event with compact JSON"] Encode --> FitsEvent{"Single event fits\ninside shared EVP limit?"} FitsEvent -- "yes" --> Batch["Add to current payload"] FitsEvent -- "no" --> Degrade["Drop targeting_key and context\nfor that event only"] Degrade --> FitsDegraded{"Degraded event fits?"} FitsDegraded -- "yes" --> Batch FitsDegraded -- "no" --> Drop["Drop and count\npayload_limit"] Batch --> FitsPayload{"Payload still <=\nDEFAULT_EVP_PAYLOAD_SIZE_LIMIT?"} FitsPayload -- "yes" --> Continue["Continue batching"] FitsPayload -- "no" --> Split["Close current payload\nstart next payload"] Split --> Continue Continue --> Post["POST each payload sequentially"]Decisions
flagevaluationbackend route instead of adding a Python-specific transport or schema.flagEvaluationspayloads by encoded byte size usingDEFAULT_EVP_PAYLOAD_SIZE_LIMIT; send split payloads sequentially from the writer thread.targeting_keyand pruned context; degraded buckets retain only lower-cardinality schema dimensions.targeting_keyin the dedicated EVP field and removetargetingKey/targeting_keyfrom context before building context keys or payloads.reasonbecause it is not accepted by the canonical worker schema, and do not populatetargeting_rule.keyuntil Python exposes real targeting-rule metadata.openfeature-sdkas an application/test/lint dependency rather than adding it as a runtime dependency ofddtrace; the dogfooding app dependency floor is being tightened separately in https://github.com/DataDog/ffe-dogfooding/pull/88.Validation Evidence
Dogfooding App
ffe-dogfoodingapp-pythonwas run with localdd-trace-pyartifacts and reachedPROVIDER_READY.openfeature-sdk=0.9.0.ffe-dogfooding-string-flagthrough the Python dogfooding app with 5 evaluations per targeting key over one persistent connection:python-batch-keepalive-20260623T004034Z-alphapython-batch-keepalive-20260623T004034Z-bravopython-batch-keepalive-20260623T004034Z-charlievariant_2with allocationallocation-override-392dd7c149f8.System Tests
Staging End-To-End
flagevaluationrows for the exact targeting keys above.evaluation_count=5:python-batch-keepalive-20260623T004034Z-alphapython-batch-keepalive-20260623T004034Z-bravopython-batch-keepalive-20260623T004034Z-charlieflag.key=ffe-dogfooding-string-flag,variant.key=variant_2, andallocation.key=allocation-override-392dd7c149f8.