Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
141 changes: 98 additions & 43 deletions manifests/python.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,30 @@ manifest:
- weblog_declaration:
"*": irrelevant (just one weblog is enough to test the SDK)
"flask-poc": v4.13.0-dev
tests/ai_guard/test_ai_guard_sdk.py::Test_AIGuardStandalone_APMDisabledMarker: missing_feature
tests/ai_guard/test_ai_guard_sdk.py::Test_AIGuardTelemetryRedacted: bug (APPSEC-69627)
tests/ai_guard/test_ai_guard_sdk.py::Test_AIGuardTelemetryRequests: bug (APPSEC-69627)
tests/ai_guard/test_ai_guard_sdk.py::Test_AIGuardTelemetryTruncated: bug (APPSEC-69627)
tests/ai_guard/test_ai_guard_sdk.py::Test_AIGuardStandalone_APMDisabledMarker: # TODO: a lower version might be supported
- weblog_declaration:
'*': missing_feature
flask-poc: '>=4.15.2'
uwsgi-poc: '>=4.15.2'
uds-flask: '>=4.15.2'
tests/ai_guard/test_ai_guard_sdk.py::Test_AIGuardTelemetryRedacted: # TODO: a lower version might be supported
- declaration: bug (APPSEC-69627)
component_version: <4.15.2
weblog: [flask-poc, uwsgi-poc, uds-flask]
- declaration: bug (APPSEC-69627)
excluded_weblog: [flask-poc, uwsgi-poc, uds-flask]
tests/ai_guard/test_ai_guard_sdk.py::Test_AIGuardTelemetryRequests: # TODO: a lower version might be supported
- declaration: bug (APPSEC-69627)
component_version: <4.15.2
weblog: [flask-poc, uwsgi-poc, uds-flask]
- declaration: bug (APPSEC-69627)
excluded_weblog: [flask-poc, uwsgi-poc, uds-flask]
tests/ai_guard/test_ai_guard_sdk.py::Test_AIGuardTelemetryTruncated: # TODO: a lower version might be supported
- declaration: bug (APPSEC-69627)
component_version: <4.15.2
weblog: [flask-poc, uwsgi-poc, uds-flask]
- declaration: bug (APPSEC-69627)
excluded_weblog: [flask-poc, uwsgi-poc, uds-flask]
tests/ai_guard/test_ai_guard_sdk.py::Test_AnomalyDetectionTags:
- weblog_declaration:
"*": irrelevant (just one weblog is enough to test the SDK)
Expand Down Expand Up @@ -57,7 +77,12 @@ manifest:
- weblog_declaration:
"*": irrelevant (just one weblog is enough to test the SDK)
"flask-poc": v4.14.0-dev
tests/ai_guard/test_ai_guard_sdk.py::Test_RedactionDisabledTelemetry: bug (APPSEC-69627)
tests/ai_guard/test_ai_guard_sdk.py::Test_RedactionDisabledTelemetry: # TODO: a lower version might be supported
- declaration: bug (APPSEC-69627)
component_version: <4.15.2
weblog: [flask-poc, uwsgi-poc, uds-flask]
- declaration: bug (APPSEC-69627)
excluded_weblog: [flask-poc, uwsgi-poc, uds-flask]
tests/ai_guard/test_ai_guard_sdk.py::Test_RedactionFailSafe:
- weblog_declaration:
"*": irrelevant (just one weblog is enough to test the SDK)
Expand Down Expand Up @@ -499,13 +524,13 @@ manifest:
tests/appsec/iast/sink/test_unvalidated_redirect.py:
- weblog_declaration:
tornado: missing_feature
tests/appsec/iast/sink/test_unvalidated_redirect.py::TestUnvalidatedHeader:
tests/appsec/iast/sink/test_unvalidated_redirect.py::TestUnvalidatedHeader: # Easy win for django-poc, django-py3.13, fastapi, flask-poc, python3.12, uds-flask, uwsgi-poc and version 4.15.2
- weblog_declaration:
"*": v4.16.0-rc1 (implemented in v3.9.0.dev, previously flaky due to APPSEC-57817 and APPSEC-61861)
tests/appsec/iast/sink/test_unvalidated_redirect.py::TestUnvalidatedHeader::test_secure:
- weblog_declaration:
"*": v4.12.0-rc1 (implemented in v3.9.0.dev)
tests/appsec/iast/sink/test_unvalidated_redirect.py::TestUnvalidatedHeader_ExtendedLocation:
tests/appsec/iast/sink/test_unvalidated_redirect.py::TestUnvalidatedHeader_ExtendedLocation: # Easy win for django-poc, django-py3.13, fastapi, flask-poc, python3.12, uds-flask, uwsgi-poc and version 4.15.2
- weblog_declaration:
"*": v4.16.0-rc1 (implemented in v3.9.0.dev, previously flaky due to APPSEC-57817 and APPSEC-61861)
tests/appsec/iast/sink/test_unvalidated_redirect.py::TestUnvalidatedHeader_StackTrace: # Easy win for django-poc, django-py3.13, fastapi, flask-poc, python3.12, uds-flask, uwsgi-poc and version 4.6.4
Expand All @@ -514,13 +539,13 @@ manifest:
tests/appsec/iast/sink/test_unvalidated_redirect.py::TestUnvalidatedRedirect: # Easy win for django-poc, django-py3.13, fastapi, flask-poc, python3.12, uds-flask, uwsgi-poc and version 4.6.4
- weblog_declaration:
"*": v4.7.0-rc1 (implemented in v3.9.0.dev, but there was the APPSEC-57817 and APPSEC-61861 bugs)
tests/appsec/iast/sink/test_unvalidated_redirect.py::TestUnvalidatedRedirect::test_secure:
tests/appsec/iast/sink/test_unvalidated_redirect.py::TestUnvalidatedRedirect::test_secure: # Easy win for django-poc, django-py3.13, fastapi, flask-poc, python3.12 and version 4.15.2
- weblog_declaration:
"*": v4.16.0-rc1 (implemented in v3.9.0.dev, previously flaky due to APPSEC-61861)
tests/appsec/iast/sink/test_unvalidated_redirect.py::TestUnvalidatedRedirect_ExtendedLocation:
- weblog_declaration:
"*": v4.12.0-rc1 (implemented in v3.9.0.dev)
tests/appsec/iast/sink/test_unvalidated_redirect.py::TestUnvalidatedRedirect_ExtendedLocation::test_extended_location_data: v4.16.0-rc1 (implemented in v3.9.0.dev, previously flaky due to APPSEC-62470)
tests/appsec/iast/sink/test_unvalidated_redirect.py::TestUnvalidatedRedirect_ExtendedLocation::test_extended_location_data: v4.16.0-rc1 (implemented in v3.9.0.dev, previously flaky due to APPSEC-62470) # Easy win for django-poc, django-py3.13, fastapi, flask-poc, python3.12, uds-flask, uwsgi-poc and version 4.15.2
tests/appsec/iast/sink/test_unvalidated_redirect.py::TestUnvalidatedRedirect_StackTrace: # Easy win for django-poc, django-py3.13, fastapi, flask-poc, python3.12, uds-flask, uwsgi-poc and version 4.6.4
- weblog_declaration:
"*": v4.7.0-rc1 (implemented in v3.9.0.dev, but there was the APPSEC-57817 and APPSEC-61861 bugs)
Expand Down Expand Up @@ -1474,14 +1499,39 @@ manifest:
tests/ffe/test_exposure_egress.py::Test_FFE_Exposure_Egress_Agentless_Direct_Shutdown: missing_feature (Not yet implemented)
tests/ffe/test_exposure_egress.py::Test_FFE_Exposure_Egress_Agentless_Sidecar: missing_feature (Not yet implemented)
tests/ffe/test_exposures_datadog_agent.py: v4.2.0-dev
tests/ffe/test_exposures_datadog_agent.py::Test_FFE_Exposure_Caching_Serial_Id_Appears: missing_feature (EX-3413)
tests/ffe/test_exposures_datadog_agent.py::Test_FFE_Exposure_Caching_Serial_Id_Cycle: missing_feature (EX-3413)
tests/ffe/test_exposures_datadog_agent.py::Test_FFE_Exposure_Caching_Serial_Id_Disappears: missing_feature (EX-3413)
tests/ffe/test_exposures_datadog_agent.py::Test_FFE_Exposure_Caching_Serial_Id_Appears: # TODO: a lower version might be supported
- declaration: missing_feature (EX-3413)
component_version: <4.15.2
weblog: [fastapi, django-poc, uwsgi-poc, django-py3.13, python3.12, flask-poc, uds-flask, tornado]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restrict FFE activation to implementations of /ffe

This activation list includes django-poc, django-py3.13, and python3.12, which are no longer marked missing at Python 4.15.2. Those images copy the Django weblog, and a repository-wide search of utils/build/docker/python finds /ffe implemented only by the Flask, FastAPI, and Tornado weblogs. The newly enabled serial-ID tests POST to /ffe (for example, Test_FFE_Exposure_Serial_Id.setup_ffe_exposure_serial_id), so these Django variants will return 404 and fail; remove the Django variants from the repeated FFE activation lists (including the file-level EVP entry) until that endpoint exists.

Useful? React with 👍 / 👎.

- declaration: missing_feature (EX-3413)
excluded_weblog: [fastapi, django-poc, uwsgi-poc, django-py3.13, python3.12, flask-poc, uds-flask, tornado]
tests/ffe/test_exposures_datadog_agent.py::Test_FFE_Exposure_Caching_Serial_Id_Cycle: # TODO: a lower version might be supported
- declaration: missing_feature (EX-3413)
component_version: <4.15.2
weblog: [fastapi, django-poc, uwsgi-poc, django-py3.13, python3.12, flask-poc, uds-flask, tornado]
- declaration: missing_feature (EX-3413)
excluded_weblog: [fastapi, django-poc, uwsgi-poc, django-py3.13, python3.12, flask-poc, uds-flask, tornado]
tests/ffe/test_exposures_datadog_agent.py::Test_FFE_Exposure_Caching_Serial_Id_Disappears: # TODO: a lower version might be supported
- declaration: missing_feature (EX-3413)
component_version: <4.15.2
weblog: [fastapi, django-poc, uwsgi-poc, django-py3.13, python3.12, flask-poc, uds-flask, tornado]
- declaration: missing_feature (EX-3413)
excluded_weblog: [fastapi, django-poc, uwsgi-poc, django-py3.13, python3.12, flask-poc, uds-flask, tornado]
tests/ffe/test_exposures_datadog_agent.py::Test_FFE_Exposure_DoLog_False::test_ffe_exposure_dolog_false:
- declaration: flaky (FFL-3293)
component_version: 4.14.1
tests/ffe/test_exposures_datadog_agent.py::Test_FFE_Exposure_Serial_Id: missing_feature (EX-3413)
tests/ffe/test_flag_eval_evp.py: missing_feature (FFL-2446)
tests/ffe/test_exposures_datadog_agent.py::Test_FFE_Exposure_Serial_Id: # TODO: a lower version might be supported
- declaration: missing_feature (EX-3413)
component_version: <4.15.2
weblog: [fastapi, django-poc, uwsgi-poc, django-py3.13, python3.12, flask-poc, uds-flask, tornado]
- declaration: missing_feature (EX-3413)
excluded_weblog: [fastapi, django-poc, uwsgi-poc, django-py3.13, python3.12, flask-poc, uds-flask, tornado]
tests/ffe/test_flag_eval_evp.py: # TODO: a lower version might be supported
- declaration: missing_feature (FFL-2446)
component_version: <4.15.2
weblog: [fastapi, django-poc, uwsgi-poc, django-py3.13, python3.12, flask-poc, uds-flask, tornado]
- declaration: missing_feature (FFL-2446)
excluded_weblog: [fastapi, django-poc, uwsgi-poc, django-py3.13, python3.12, flask-poc, uds-flask, tornado]
Comment on lines +1529 to +1534

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep agentless EVP contracts disabled outside Flask

At Python 4.15.2 this file-level rule removes the sole FFL-2446 declaration from both Test_FFE_EVP_Flagevaluation_Egress_Agentless_* classes, so the direct and serverless agentless scenarios will run their @scenario_crash contracts for every listed weblog. The adjacent agentless-configuration test is explicitly scoped to flask-poc at manifests/python.yml:1491-1494, and the parallel direct/serverless exposure contracts remain missing_feature (Not yet implemented) at manifests/python.yml:1498-1500. A passing agent-backed FFE run cannot enable these separate scenario topologies; retain class-specific skips until they are supported, or express a genuinely scenario-specific condition with a decorator. .cursor/rules/test-activation.mdcL33-L35

Useful? React with 👍 / 👎.

tests/ffe/test_flag_eval_metrics.py: v4.7.0
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Nested_Attributes_Ignored: irrelevant (FFL-1980)
tests/integration_frameworks/llm/anthropic/test_anthropic_ai_guard.py::TestAnthropicAiGuard: v4.11.0
Expand Down Expand Up @@ -1596,8 +1646,8 @@ manifest:
uds-flask: missing_feature
tests/integrations/test_db_integrations_sql.py::Test_MsSql::test_sql_success:
- weblog_declaration:
flask-poc: bug (APMAPI-1058)
uds-flask: bug (APMAPI-1058)
flask-poc: v4.15.2 # TODO: a lower version might be supported
uds-flask: v4.15.2 # TODO: a lower version might be supported
tests/integrations/test_db_integrations_sql.py::Test_MySql:
- weblog_declaration:
"*": missing_feature
Expand Down Expand Up @@ -1715,24 +1765,24 @@ manifest:
uwsgi-poc: v2.9.0
tests/integrations/test_dbm.py::Test_Dbm::test_trace_payload_full:
- weblog_declaration:
flask-poc: bug (APMAPI-1058)
uds-flask: bug (APMAPI-1058)
flask-poc: v4.15.2 # TODO: a lower version might be supported
uds-flask: v4.15.2 # TODO: a lower version might be supported
tests/integrations/test_dbm.py::Test_Dbm_Comment_Batch_Python_Aiomysql:
- weblog_declaration:
"*": missing_feature (Missing on weblog)
*flask : v2.9.0
tests/integrations/test_dbm.py::Test_Dbm_Comment_Batch_Python_Aiomysql::test_dbm_comment:
- weblog_declaration:
flask-poc: bug (APMAPI-1058)
uds-flask: bug (APMAPI-1058)
flask-poc: v4.15.2 # TODO: a lower version might be supported
uds-flask: v4.15.2 # TODO: a lower version might be supported
tests/integrations/test_dbm.py::Test_Dbm_Comment_Batch_Python_MysqlConnector:
- weblog_declaration:
"*": missing_feature (Missing on weblog)
*flask : v2.9.0
tests/integrations/test_dbm.py::Test_Dbm_Comment_Batch_Python_MysqlConnector::test_dbm_comment:
- weblog_declaration:
flask-poc: bug (APMAPI-1058)
uds-flask: bug (APMAPI-1058)
flask-poc: v4.15.2 # TODO: a lower version might be supported
uds-flask: v4.15.2 # TODO: a lower version might be supported
tests/integrations/test_dbm.py::Test_Dbm_Comment_Batch_Python_Mysqldb:
- weblog_declaration:
"*": missing_feature (Missing on weblog)
Expand Down Expand Up @@ -1772,24 +1822,24 @@ manifest:
*flask : v2.9.0
tests/integrations/test_dbm.py::Test_Dbm_Comment_Python_Aiomysql::test_dbm_comment:
- weblog_declaration:
flask-poc: bug (APMAPI-1058)
uds-flask: bug (APMAPI-1058)
flask-poc: v4.15.2 # TODO: a lower version might be supported
uds-flask: v4.15.2 # TODO: a lower version might be supported
tests/integrations/test_dbm.py::Test_Dbm_Comment_Python_Asyncpg:
- weblog_declaration:
"*": missing_feature (Missing on weblog)
*flask : v2.9.0
tests/integrations/test_dbm.py::Test_Dbm_Comment_Python_Asyncpg::test_dbm_comment:
- weblog_declaration:
flask-poc: bug (APMAPI-1058)
uds-flask: bug (APMAPI-1058)
flask-poc: v4.15.2 # TODO: a lower version might be supported
uds-flask: v4.15.2 # TODO: a lower version might be supported
tests/integrations/test_dbm.py::Test_Dbm_Comment_Python_MysqlConnector:
- weblog_declaration:
"*": missing_feature (Missing on weblog)
*flask : v2.9.0
tests/integrations/test_dbm.py::Test_Dbm_Comment_Python_MysqlConnector::test_dbm_comment:
- weblog_declaration:
flask-poc: bug (APMAPI-1058)
uds-flask: bug (APMAPI-1058)
flask-poc: v4.15.2 # TODO: a lower version might be supported
uds-flask: v4.15.2 # TODO: a lower version might be supported
tests/integrations/test_dbm.py::Test_Dbm_Comment_Python_Mysqldb:
- weblog_declaration:
"*": missing_feature (Missing on weblog)
Expand All @@ -1807,8 +1857,8 @@ manifest:
uwsgi-poc: v2.9.0
tests/integrations/test_dbm.py::Test_Dbm_Comment_Python_Psycopg::test_dbm_comment:
- weblog_declaration:
flask-poc: bug (APMAPI-1058)
uds-flask: bug (APMAPI-1058)
flask-poc: v4.15.2 # TODO: a lower version might be supported
uds-flask: v4.15.2 # TODO: a lower version might be supported
tests/integrations/test_dbm.py::Test_Dbm_Comment_Python_Pymysql:
- weblog_declaration:
"*": missing_feature (Missing on weblog)
Expand All @@ -1818,7 +1868,12 @@ manifest:
- weblog_declaration:
flask-poc: bug (APMAPI-1058)
uds-flask: bug (APMAPI-1058)
tests/integrations/test_dbm.py::Test_Dbm_DynamicService_Postgres: missing_feature
tests/integrations/test_dbm.py::Test_Dbm_DynamicService_Postgres: # TODO: a lower version might be supported
- weblog_declaration:
'*': missing_feature
flask-poc: '>=4.15.2'
uwsgi-poc: '>=4.15.2'
uds-flask: '>=4.15.2'
tests/integrations/test_dbm.py::_BaseDbmComment::test_dbm_comment:
- weblog_declaration:
flask-poc: bug (APMAPI-1058)
Expand All @@ -1838,7 +1893,7 @@ manifest:
flask-poc: v2.16.0
tests/integrations/test_dsm.py::Test_DsmKafka::test_dsm_kafka:
- weblog_declaration:
flask-poc: bug (APMAPI-1058)
flask-poc: v4.15.2 # TODO: a lower version might be supported
uds-flask: bug (APMAPI-1058)
tests/integrations/test_dsm.py::Test_DsmKafka::test_dsm_kafka_without_cluster_id: irrelevant
tests/integrations/test_dsm.py::Test_DsmKinesis:
Expand Down Expand Up @@ -2300,15 +2355,15 @@ manifest:
component_version: <=2.16.0
tests/parametric/test_telemetry.py::Test_TelemetrySSIConfigs: v3.11.0
tests/parametric/test_trace_filters.py::Test_Trace_Filters: '>=4.12.2' # TODO: a lower version might be supported
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_ignore_resources: missing_feature
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_reject: missing_feature
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_reject_no_value: missing_feature
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_reject_regex: missing_feature
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_require_missing: missing_feature
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_require_regex_missing: missing_feature
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_resource_normalization: missing_feature
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_tag_normalization_env: missing_feature
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_trim: missing_feature
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_ignore_resources: '>=4.15.2' # TODO: a lower version might be supported
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_reject: '>=4.15.2' # TODO: a lower version might be supported
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_reject_no_value: '>=4.15.2' # TODO: a lower version might be supported
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_reject_regex: '>=4.15.2' # TODO: a lower version might be supported
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_require_missing: '>=4.15.2' # TODO: a lower version might be supported
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_require_regex_missing: '>=4.15.2' # TODO: a lower version might be supported
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_resource_normalization: '>=4.15.2' # TODO: a lower version might be supported
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_tag_normalization_env: '>=4.15.2' # TODO: a lower version might be supported
tests/parametric/test_trace_filters.py::Test_Trace_Filters::test_trace_filters_trim: '>=4.15.2' # TODO: a lower version might be supported
tests/parametric/test_trace_sampling.py::Test_Trace_Sampling_Basic: v1.9.0 # actual version unknown
tests/parametric/test_trace_sampling.py::Test_Trace_Sampling_Globs: v2.8.0
tests/parametric/test_trace_sampling.py::Test_Trace_Sampling_Globs_Feb2024_Revision: v2.8.0
Expand Down Expand Up @@ -2541,8 +2596,8 @@ manifest:
tests/test_otel_http_semantics.py::Test_OtelSemantics_OTLP_TraceMetrics::test_trace_metric_agrees_with_client_success: missing_feature (libdatadog status.code fix is not merged)
tests/test_otel_http_semantics.py::Test_OtelSemantics_OTLP_TraceMetrics::test_trace_metric_agrees_with_server_success: missing_feature (libdatadog status.code fix is not merged)
tests/test_otel_http_semantics.py::Test_OtelSemantics_SamplingRules: 4.15.0-dev
tests/test_otel_http_semantics.py::Test_OtelSemantics_Spans_Http_Client: 4.15.0-dev
tests/test_otel_http_semantics.py::Test_OtelSemantics_Spans_Http_Server: 4.15.0-dev
tests/test_otel_http_semantics.py::Test_OtelSemantics_Spans_Http_Client: 4.15.0-dev # Easy win for django-poc, django-py3.13, fastapi, flask-poc, python3.12, uds-flask, uwsgi-poc and version 4.15.2
tests/test_otel_http_semantics.py::Test_OtelSemantics_Spans_Http_Server: 4.15.0-dev # Easy win for django-poc, django-py3.13, fastapi, flask-poc, python3.12, tornado, uds-flask, uwsgi-poc and version 4.15.2
tests/test_otel_http_semantics.py::Test_OtelSemantics_Spans_Http_Server::test_http_endpoint_retained_from_appsec: irrelevant (The AppSec endpoint retention request is specific to .NET)
tests/test_otel_tracestate_sampling.py::Test_EmitOtOnProbabilityDecision_Rate0_5: v4.15.0-dev
tests/test_otel_tracestate_sampling.py::Test_ForceKeepClearsTh: v4.15.0-dev
Expand Down
Loading