Repository navigation
Enable EVP flagevaluation system tests for Python - #7183
Draft
leoromanovsky wants to merge 4 commits into
Draft
leoromanovsky wants to merge 4 commits into
leoromanovsky wants to merge 4 commits into
Conversation
Contributor
|
|
|
|
Base automatically changed from
leo.romanovsky/ffe-evp-flagevaluation-contract-fix
to
main
June 22, 2026 19:40
…p-flagevaluation-enable-python-clean # Conflicts: # tests/ffe/test_flag_eval_evp.py
…p-flagevaluation-enable-python-clean
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
to DataDog/dd-trace-py
that referenced
this pull request
Jul 14, 2026
## Motivation
Server-side feature-flag evaluations need to be observable in the same backend track as the rest of the cross-SDK rollout so APM can judge rollout smoothness and time-to-land from real evaluation counts, not only app-local behavior or metric-side approximations. This contribution gives Python services backend-visible `flagevaluation` telemetry with bounded cardinality and batching, so customers and internal dogfooding can verify which flags evaluated, for which targeting keys, and how often, while preserving the existing `feature_flag.evaluations` metric path.
## Changes
- Adds a Python EVP `flagevaluation` writer behind `DD_FLAGGING_EVALUATION_COUNTS_ENABLED`; existing OTel `feature_flag.evaluations` metrics remain unchanged.
- Registers an OpenFeature `finally_after` logging hook that captures a cheap evaluation snapshot and hands it to a background writer.
- Renames the capture hook to `FlagEvalEVPHook` / `_flag_eval_evp_hook.py` so it is distinct from `FlagEvalMetricsHook`.
- Posts batched payloads to the Agent EVP proxy route `/evp_proxy/v2/api/v2/flagevaluation` with `X-Datadog-EVP-Subdomain: event-platform-intake`.
- Aggregates evaluations by schema-visible dimensions: flag key, variant key, allocation key, runtime-default state, error message, targeting key, and pruned evaluation context.
- Flattens and prunes evaluation context before enqueue with deterministic 256-field / 256-character bounds.
- Splits encoded `flagEvaluations` request bodies under the shared EVP payload limit before POSTing.
- Degrades a single oversized full-tier row by dropping `targeting_key` and `context`; if the degraded row still cannot fit, the writer drops and logs it.
- Uses evaluation-time metadata to populate `first_evaluation` and `last_evaluation` rather than flush time.
- Keeps accepting the legacy `DataDogProvider(initialization_timeout=...)` keyword for compatibility, while async initialization no longer uses it.
### Evaluation Capture And Flush
```mermaid
flowchart TD
Eval["OpenFeature evaluation"] --> Metrics["FlagEvalMetricsHook\nexisting OTel metric"]
Eval --> Logging["FlagEvalEVPHook\nfinally_after"]
Logging --> Snapshot["Cheap scalar snapshot\nflag, variant, allocation,\ntargeting_key, context, eval time"]
Snapshot --> Queue["Bounded non-blocking queue\nQueue size 4096"]
Queue --> Drain["Background drain worker\n0.1s queue drain"]
Drain --> Aggregate["FlagEvaluationWriter aggregation\nfull tier then degraded tier"]
Aggregate --> Flush["Periodic flush\n10s interval"]
Flush --> EVP["Agent EVP proxy\n/evp_proxy/v2/api/v2/flagevaluation"]
```
### Payload Limit Handling
```mermaid
flowchart TD
Events["Aggregated flagEvaluations"] --> Encode["Encode each event with compact JSON"]
Encode --> FitsEvent{"Single event fits\ninside shared EVP limit?"}
FitsEvent -- "yes" --> Batch["Add to current payload"]
FitsEvent -- "no" --> Degrade["Drop targeting_key and context\nfor that event only"]
Degrade --> FitsDegraded{"Degraded event fits?"}
FitsDegraded -- "yes" --> Batch
FitsDegraded -- "no" --> Drop["Drop and count\npayload_limit"]
Batch --> FitsPayload{"Payload still <=\nDEFAULT_EVP_PAYLOAD_SIZE_LIMIT?"}
FitsPayload -- "yes" --> Continue["Continue batching"]
FitsPayload -- "no" --> Split["Close current payload\nstart next payload"]
Split --> Continue
Continue --> Post["POST each payload sequentially"]
```
## Decisions
- Keep this as a dual-path implementation: the EVP logging writer is additive and separately gated, while the existing OTel metric hook remains registered when the provider is enabled.
- Use the existing Agent EVP proxy transport and the canonical `flagevaluation` backend route instead of adding a Python-specific transport or schema.
- Keep the evaluation logging hook non-blocking on application evaluation threads: queueing is best-effort, bounded, and drop-counted under backpressure.
- Do aggregation, JSON serialization, payload splitting, and POSTs only in the background writer path.
- Split `flagEvaluations` payloads by encoded byte size using `DEFAULT_EVP_PAYLOAD_SIZE_LIMIT`; send split payloads sequentially from the writer thread.
- Use a two-tier aggregation model: full buckets retain `targeting_key` and pruned context; degraded buckets retain only lower-cardinality schema dimensions.
- Store `targeting_key` in the dedicated EVP field and remove `targetingKey` / `targeting_key` from context before building context keys or payloads.
- Do not emit OpenFeature `reason` because it is not accepted by the canonical worker schema, and do not populate `targeting_rule.key` until Python exposes real targeting-rule metadata.
- Keep `openfeature-sdk` as an application/test/lint dependency rather than adding it as a runtime dependency of `ddtrace`; the dogfooding app dependency floor is being tightened separately in https://github.com/DataDog/ffe-dogfooding/pull/88.
## Validation Evidence
### Dogfooding App
- `ffe-dogfooding` `app-python` was run with local `dd-trace-py` artifacts and reached `PROVIDER_READY`.
- The running dogfooding app reported `openfeature-sdk=0.9.0`.
- Evaluated `ffe-dogfooding-string-flag` through the Python dogfooding app with 5 evaluations per targeting key over one persistent connection:
- `python-batch-keepalive-20260623T004034Z-alpha`
- `python-batch-keepalive-20260623T004034Z-bravo`
- `python-batch-keepalive-20260623T004034Z-charlie`
- App-side result: all evaluations returned `variant_2` with allocation `allocation-override-392dd7c149f8`.
### System Tests
- Companion draft PR: DataDog/system-tests#7183
### Staging End-To-End
- Dogfooding ran without the local mock-intake EVP tee/proxy, so the Agent sent EVP traffic through the normal backend path.
- Retriever staging query returned 3 backend `flagevaluation` rows for the exact targeting keys above.
- Each targeting key returned one row with `evaluation_count=5`:
- `python-batch-keepalive-20260623T004034Z-alpha`
- `python-batch-keepalive-20260623T004034Z-bravo`
- `python-batch-keepalive-20260623T004034Z-charlie`
- All rows had `flag.key=ffe-dogfooding-string-flag`, `variant.key=variant_2`, and `allocation.key=allocation-override-392dd7c149f8`.
Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>
brettlangdon
pushed a commit
to DataDog/dd-trace-py
that referenced
this pull request
Jul 15, 2026
## Motivation
Server-side feature-flag evaluations need to be observable in the same backend track as the rest of the cross-SDK rollout so APM can judge rollout smoothness and time-to-land from real evaluation counts, not only app-local behavior or metric-side approximations. This contribution gives Python services backend-visible `flagevaluation` telemetry with bounded cardinality and batching, so customers and internal dogfooding can verify which flags evaluated, for which targeting keys, and how often, while preserving the existing `feature_flag.evaluations` metric path.
## Changes
- Adds a Python EVP `flagevaluation` writer behind `DD_FLAGGING_EVALUATION_COUNTS_ENABLED`; existing OTel `feature_flag.evaluations` metrics remain unchanged.
- Registers an OpenFeature `finally_after` logging hook that captures a cheap evaluation snapshot and hands it to a background writer.
- Renames the capture hook to `FlagEvalEVPHook` / `_flag_eval_evp_hook.py` so it is distinct from `FlagEvalMetricsHook`.
- Posts batched payloads to the Agent EVP proxy route `/evp_proxy/v2/api/v2/flagevaluation` with `X-Datadog-EVP-Subdomain: event-platform-intake`.
- Aggregates evaluations by schema-visible dimensions: flag key, variant key, allocation key, runtime-default state, error message, targeting key, and pruned evaluation context.
- Flattens and prunes evaluation context before enqueue with deterministic 256-field / 256-character bounds.
- Splits encoded `flagEvaluations` request bodies under the shared EVP payload limit before POSTing.
- Degrades a single oversized full-tier row by dropping `targeting_key` and `context`; if the degraded row still cannot fit, the writer drops and logs it.
- Uses evaluation-time metadata to populate `first_evaluation` and `last_evaluation` rather than flush time.
- Keeps accepting the legacy `DataDogProvider(initialization_timeout=...)` keyword for compatibility, while async initialization no longer uses it.
### Evaluation Capture And Flush
```mermaid
flowchart TD
Eval["OpenFeature evaluation"] --> Metrics["FlagEvalMetricsHook\nexisting OTel metric"]
Eval --> Logging["FlagEvalEVPHook\nfinally_after"]
Logging --> Snapshot["Cheap scalar snapshot\nflag, variant, allocation,\ntargeting_key, context, eval time"]
Snapshot --> Queue["Bounded non-blocking queue\nQueue size 4096"]
Queue --> Drain["Background drain worker\n0.1s queue drain"]
Drain --> Aggregate["FlagEvaluationWriter aggregation\nfull tier then degraded tier"]
Aggregate --> Flush["Periodic flush\n10s interval"]
Flush --> EVP["Agent EVP proxy\n/evp_proxy/v2/api/v2/flagevaluation"]
```
### Payload Limit Handling
```mermaid
flowchart TD
Events["Aggregated flagEvaluations"] --> Encode["Encode each event with compact JSON"]
Encode --> FitsEvent{"Single event fits\ninside shared EVP limit?"}
FitsEvent -- "yes" --> Batch["Add to current payload"]
FitsEvent -- "no" --> Degrade["Drop targeting_key and context\nfor that event only"]
Degrade --> FitsDegraded{"Degraded event fits?"}
FitsDegraded -- "yes" --> Batch
FitsDegraded -- "no" --> Drop["Drop and count\npayload_limit"]
Batch --> FitsPayload{"Payload still <=\nDEFAULT_EVP_PAYLOAD_SIZE_LIMIT?"}
FitsPayload -- "yes" --> Continue["Continue batching"]
FitsPayload -- "no" --> Split["Close current payload\nstart next payload"]
Split --> Continue
Continue --> Post["POST each payload sequentially"]
```
## Decisions
- Keep this as a dual-path implementation: the EVP logging writer is additive and separately gated, while the existing OTel metric hook remains registered when the provider is enabled.
- Use the existing Agent EVP proxy transport and the canonical `flagevaluation` backend route instead of adding a Python-specific transport or schema.
- Keep the evaluation logging hook non-blocking on application evaluation threads: queueing is best-effort, bounded, and drop-counted under backpressure.
- Do aggregation, JSON serialization, payload splitting, and POSTs only in the background writer path.
- Split `flagEvaluations` payloads by encoded byte size using `DEFAULT_EVP_PAYLOAD_SIZE_LIMIT`; send split payloads sequentially from the writer thread.
- Use a two-tier aggregation model: full buckets retain `targeting_key` and pruned context; degraded buckets retain only lower-cardinality schema dimensions.
- Store `targeting_key` in the dedicated EVP field and remove `targetingKey` / `targeting_key` from context before building context keys or payloads.
- Do not emit OpenFeature `reason` because it is not accepted by the canonical worker schema, and do not populate `targeting_rule.key` until Python exposes real targeting-rule metadata.
- Keep `openfeature-sdk` as an application/test/lint dependency rather than adding it as a runtime dependency of `ddtrace`; the dogfooding app dependency floor is being tightened separately in https://github.com/DataDog/ffe-dogfooding/pull/88.
## Validation Evidence
### Dogfooding App
- `ffe-dogfooding` `app-python` was run with local `dd-trace-py` artifacts and reached `PROVIDER_READY`.
- The running dogfooding app reported `openfeature-sdk=0.9.0`.
- Evaluated `ffe-dogfooding-string-flag` through the Python dogfooding app with 5 evaluations per targeting key over one persistent connection:
- `python-batch-keepalive-20260623T004034Z-alpha`
- `python-batch-keepalive-20260623T004034Z-bravo`
- `python-batch-keepalive-20260623T004034Z-charlie`
- App-side result: all evaluations returned `variant_2` with allocation `allocation-override-392dd7c149f8`.
### System Tests
- Companion draft PR: DataDog/system-tests#7183
### Staging End-To-End
- Dogfooding ran without the local mock-intake EVP tee/proxy, so the Agent sent EVP traffic through the normal backend path.
- Retriever staging query returned 3 backend `flagevaluation` rows for the exact targeting keys above.
- Each targeting key returned one row with `evaluation_count=5`:
- `python-batch-keepalive-20260623T004034Z-alpha`
- `python-batch-keepalive-20260623T004034Z-bravo`
- `python-batch-keepalive-20260623T004034Z-charlie`
- All rows had `flag.key=ffe-dogfooding-string-flag`, `variant.key=variant_2`, and `allocation.key=allocation-override-392dd7c149f8`.
Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>
vlad-scherbich
pushed a commit
to DataDog/dd-trace-py
that referenced
this pull request
Jul 15, 2026
## Motivation
Server-side feature-flag evaluations need to be observable in the same backend track as the rest of the cross-SDK rollout so APM can judge rollout smoothness and time-to-land from real evaluation counts, not only app-local behavior or metric-side approximations. This contribution gives Python services backend-visible `flagevaluation` telemetry with bounded cardinality and batching, so customers and internal dogfooding can verify which flags evaluated, for which targeting keys, and how often, while preserving the existing `feature_flag.evaluations` metric path.
## Changes
- Adds a Python EVP `flagevaluation` writer behind `DD_FLAGGING_EVALUATION_COUNTS_ENABLED`; existing OTel `feature_flag.evaluations` metrics remain unchanged.
- Registers an OpenFeature `finally_after` logging hook that captures a cheap evaluation snapshot and hands it to a background writer.
- Renames the capture hook to `FlagEvalEVPHook` / `_flag_eval_evp_hook.py` so it is distinct from `FlagEvalMetricsHook`.
- Posts batched payloads to the Agent EVP proxy route `/evp_proxy/v2/api/v2/flagevaluation` with `X-Datadog-EVP-Subdomain: event-platform-intake`.
- Aggregates evaluations by schema-visible dimensions: flag key, variant key, allocation key, runtime-default state, error message, targeting key, and pruned evaluation context.
- Flattens and prunes evaluation context before enqueue with deterministic 256-field / 256-character bounds.
- Splits encoded `flagEvaluations` request bodies under the shared EVP payload limit before POSTing.
- Degrades a single oversized full-tier row by dropping `targeting_key` and `context`; if the degraded row still cannot fit, the writer drops and logs it.
- Uses evaluation-time metadata to populate `first_evaluation` and `last_evaluation` rather than flush time.
- Keeps accepting the legacy `DataDogProvider(initialization_timeout=...)` keyword for compatibility, while async initialization no longer uses it.
### Evaluation Capture And Flush
```mermaid
flowchart TD
Eval["OpenFeature evaluation"] --> Metrics["FlagEvalMetricsHook\nexisting OTel metric"]
Eval --> Logging["FlagEvalEVPHook\nfinally_after"]
Logging --> Snapshot["Cheap scalar snapshot\nflag, variant, allocation,\ntargeting_key, context, eval time"]
Snapshot --> Queue["Bounded non-blocking queue\nQueue size 4096"]
Queue --> Drain["Background drain worker\n0.1s queue drain"]
Drain --> Aggregate["FlagEvaluationWriter aggregation\nfull tier then degraded tier"]
Aggregate --> Flush["Periodic flush\n10s interval"]
Flush --> EVP["Agent EVP proxy\n/evp_proxy/v2/api/v2/flagevaluation"]
```
### Payload Limit Handling
```mermaid
flowchart TD
Events["Aggregated flagEvaluations"] --> Encode["Encode each event with compact JSON"]
Encode --> FitsEvent{"Single event fits\ninside shared EVP limit?"}
FitsEvent -- "yes" --> Batch["Add to current payload"]
FitsEvent -- "no" --> Degrade["Drop targeting_key and context\nfor that event only"]
Degrade --> FitsDegraded{"Degraded event fits?"}
FitsDegraded -- "yes" --> Batch
FitsDegraded -- "no" --> Drop["Drop and count\npayload_limit"]
Batch --> FitsPayload{"Payload still <=\nDEFAULT_EVP_PAYLOAD_SIZE_LIMIT?"}
FitsPayload -- "yes" --> Continue["Continue batching"]
FitsPayload -- "no" --> Split["Close current payload\nstart next payload"]
Split --> Continue
Continue --> Post["POST each payload sequentially"]
```
## Decisions
- Keep this as a dual-path implementation: the EVP logging writer is additive and separately gated, while the existing OTel metric hook remains registered when the provider is enabled.
- Use the existing Agent EVP proxy transport and the canonical `flagevaluation` backend route instead of adding a Python-specific transport or schema.
- Keep the evaluation logging hook non-blocking on application evaluation threads: queueing is best-effort, bounded, and drop-counted under backpressure.
- Do aggregation, JSON serialization, payload splitting, and POSTs only in the background writer path.
- Split `flagEvaluations` payloads by encoded byte size using `DEFAULT_EVP_PAYLOAD_SIZE_LIMIT`; send split payloads sequentially from the writer thread.
- Use a two-tier aggregation model: full buckets retain `targeting_key` and pruned context; degraded buckets retain only lower-cardinality schema dimensions.
- Store `targeting_key` in the dedicated EVP field and remove `targetingKey` / `targeting_key` from context before building context keys or payloads.
- Do not emit OpenFeature `reason` because it is not accepted by the canonical worker schema, and do not populate `targeting_rule.key` until Python exposes real targeting-rule metadata.
- Keep `openfeature-sdk` as an application/test/lint dependency rather than adding it as a runtime dependency of `ddtrace`; the dogfooding app dependency floor is being tightened separately in https://github.com/DataDog/ffe-dogfooding/pull/88.
## Validation Evidence
### Dogfooding App
- `ffe-dogfooding` `app-python` was run with local `dd-trace-py` artifacts and reached `PROVIDER_READY`.
- The running dogfooding app reported `openfeature-sdk=0.9.0`.
- Evaluated `ffe-dogfooding-string-flag` through the Python dogfooding app with 5 evaluations per targeting key over one persistent connection:
- `python-batch-keepalive-20260623T004034Z-alpha`
- `python-batch-keepalive-20260623T004034Z-bravo`
- `python-batch-keepalive-20260623T004034Z-charlie`
- App-side result: all evaluations returned `variant_2` with allocation `allocation-override-392dd7c149f8`.
### System Tests
- Companion draft PR: DataDog/system-tests#7183
### Staging End-To-End
- Dogfooding ran without the local mock-intake EVP tee/proxy, so the Agent sent EVP traffic through the normal backend path.
- Retriever staging query returned 3 backend `flagevaluation` rows for the exact targeting keys above.
- Each targeting key returned one row with `evaluation_count=5`:
- `python-batch-keepalive-20260623T004034Z-alpha`
- `python-batch-keepalive-20260623T004034Z-bravo`
- `python-batch-keepalive-20260623T004034Z-charlie`
- All rows had `flag.key=ffe-dogfooding-string-flag`, `variant.key=variant_2`, and `allocation.key=allocation-override-392dd7c149f8`.
Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Python needs the shared server-side EVP
flagevaluationsystem-test contract enabled so APM reviewers can validate that Python evaluations produce the same backend-visible signal as the other SDKs. This PR provides the public system-tests companion for the Python SDK fanout.Changes
tests/ffe/test_flag_eval_evp.pyinmanifests/python.ymlfrommissing_feature (FFL-2446)tov4.12.0rc1.system-testsmain.Decisions
DataDog/dd-trace-py#18613.tests/ffe/test_flag_eval_metrics.py; this PR is only for EVPflagevaluation.Validation Evidence
System Tests
flagevaluationvalidation.