[Bug] Add synthetic properties check to remote ESQL validation

[eric-forte-elastic]

Pull Request

Issue link(s): #5306

Summary - What I changed

While #5256 fixed an issue where non-ecs schemas were not being loaded in to the proper indexes, this can cause issues where on certain indexs where the merging of non-ecs and the integration schema can cause artificial conflicts.

For instance in the example run:

What was once a non-nested flattened field from the intgration:

"azure.signinlogs.properties.authentication_details": "flattened"

gets merged with:
"azure.signinlogs.properties.authentication_details.authentication_method": "keyword",

which then causes it to become a nested field. As such, pruning needs to happen again after merging. However, the pruning logic also needed to be updated to handle cases where flattened fields can have keywords which can occur as the result of an artificial merge of non-ecs and the integration in the index. While having properties on flattened fields is supported (see docs), typically (so far as we've encountered) integrations use the .fields parameter instead, so we were only checking for the .fields paramter to determine if a nested field had "fields" instead of checking for both .fields and .properties.

Example Run Error.

Note: this only will occur if a rule with a specific index pattern that can cause this is used, as the schema is filtered beforehand by index.

How To Test

Run the python -m detection_rules dev test esql-remote-validation command with a local/remote stack and observe no errors

❯ python -m detection_rules dev test esql-remote-validation
Loaded config file: /home/forteea1/Code/dac_demo/dac_clean/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

ESQL rules loaded: 67
Total rules: 67
Failed rules: 0
Failed rules written to failed_rules.log

Also pull in the rule from #5305 and then re-run the test. (included below)
defense_evasion_masquerading_as_svchost.toml.txt

❯ python -m detection_rules dev test esql-remote-validation
Loaded config file: /home/forteea1/Code/dac_demo/dac_clean/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

ESQL rules loaded: 68
Total rules: 68
Failed rules: 0
Failed rules written to failed_rules.log

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Bug - Guidelines

These guidelines serve as a reminder set of considerations when addressing a bug in the code.

Documentation and Context

• Provide detailed documentation (description, screenshots, reproducing the bug, etc.) of the bug if not already documented in an issue.
• Include additional context or details about the problem.
• Ensure the fix includes necessary updates to the release documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the bug fix or edge cases.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and detecting the bug fix (e.g., test logs, screenshots).
• Validate that any rules affected by the bug are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Checks

• Verify that the bug fix works across all relevant environments (e.g., different OS versions).
• Confirm that the proper version label is applied to the PR patch, minor, major.

[Mikaayenson]

Might be good to add a test case with the exact scenario from the PR description:

• Integration schema: "azure.signinlogs.properties.authentication_details": "flattened"
• Non-ECS schema: "azure.signinlogs.properties.authentication_details.authentication_method": "keyword"
• Verify that after merging and pruning, the flattened field's properties key is correctly removed.

What about edge cases:

• Fields with .fields. in the path
• Fields at different nesting levels
• Multiple flattened fields in the same schema

Also, can we verify delete_nested_key_from_dict works correctly with the constructed paths above.

[eric-forte-elastic]

Added unit test for conflicts between schema. In this case, we only need to test for integration schema and non_ecs conflicts. The parsing/pruning logic has been updated such that these are the only two logical paths where pruning occurs.

1. Integrations schema is combined with non_ecs, ecs, and custom schema then filtered to an index level and as part of that filtering the schema is pruned of known ES|QL unsupported schema situations.
2. The non_ecs schema file is used directly in cases where there is a conflict between an integration and non_ecs (like that of this PR), and is pruned directly to remove conflicts.

The reason for the complexity for this kind of validation support is to support the field type not supported errors like this:

E               detection_rules.esql_errors.EsqlUnsupportedTypeError: BadRequestError(400, 'verification_exception', 'Found 1 problem\nline 6:13: Cannot use field [azure.signinlogs.properties.authentication_details] with unsupported type [flattened]')

Which required the proper index mapping to be built and checked independent from what is specified in non_ecs, as this error is specific to ES|QL rules. We need the index mapping to be built and pushed even if ES|QL does not support it so that we can get the specific type error (e.g. unsupported type [flattened] as opposed to the field returning "not found" which would be the alternative).

[eric-forte-elastic]

Updated test results with the prune removal/re-structure and list comprehension:

❯ python -m detection_rules dev test esql-remote-validation
Loaded config file: /home/forteea1/Code/dac_demo/dac_clean/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

ESQL rules loaded: 68
Total rules: 68
Failed rules: 0
Failed rules written to failed_rules.log

[eric-forte-elastic]

Updated Unit Test with new unit Test:

detection-rules on  5306-bug-esql-schema-field-pruning [$!?] is  v1.5.9 via  v3.12.12 (detection-rules-build) on  eric.forte
❯ python -m pytest tests/test_rules_remote.py::TestRemoteRules
============================================================================================================= test session starts ==============================================================================================================
platform linux -- Python 3.12.12, pytest-8.4.0, pluggy-1.6.0
rootdir: /home/forteea1/Code/dac_demo/dac_clean/detection-rules
configfile: pyproject.toml
plugins: typeguard-4.4.4
collected 12 items                                                                                                                                                                                                                             

tests/test_rules_remote.py ............                                                                                                                                                                                                  [100%]

=============================================================================================================== warnings summary ===============================================================================================================
tests/test_rules_remote.py: 13 warnings
  /home/forteea1/Code/dac_demo/dac_clean/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/elasticsearch/_sync/client/__init__.py:399: SecurityWarning: Connecting to 'https://localhost:9200' using TLS with verify_certs=False is insecure
    _transport = transport_class(

tests/test_rules_remote.py::TestRemoteRules::test_esql_endpoint_alerts_index
tests/test_rules_remote.py::TestRemoteRules::test_esql_endpoint_alerts_index
tests/test_rules_remote.py::TestRemoteRules::test_esql_endpoint_alerts_index_endpoint_fields
tests/test_rules_remote.py::TestRemoteRules::test_esql_filtered_index
tests/test_rules_remote.py::TestRemoteRules::test_esql_non_dataset_package_related_integrations
tests/test_rules_remote.py::TestRemoteRules::test_esql_non_ecs_schema_conflict_resolution
tests/test_rules_remote.py::TestRemoteRules::test_esql_related_integrations
  /home/forteea1/Code/dac_demo/dac_clean/detection-rules/detection_rules/index_mappings.py:367: ElasticsearchWarning: No limit defined, adding default limit of [1000]
    response = elastic_client.esql.query(query=query)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================================================================= 12 passed, 20 warnings in 10.72s =======================================================================================================

Unit test running on main without ES|QL rules (tests that just the unit test can catch the error and not dependent on a specific rule)

detection-rules on  main [$!?] is  v1.5.8 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ python -m pytest tests/test_rules_remote.py::TestRemoteRules
============================================================================================================= test session starts ==============================================================================================================
platform linux -- Python 3.12.12, pytest-8.4.0, pluggy-1.6.0
rootdir: /home/forteea1/Code/dac_demo/dac_clean/detection-rules
configfile: pyproject.toml
plugins: typeguard-4.4.4
collected 12 items                                                                                                                                                                                                                             

tests/test_rules_remote.py ........F...   

...

=========================================================================================================== short test summary info ============================================================================================================
FAILED tests/test_rules_remote.py::TestRemoteRules::test_esql_non_ecs_schema_conflict_resolution - detection_rules.esql_errors.EsqlSchemaError: BadRequestError(400, 'mapper_parsing_exception', 'unknown parameter [properties] on mapper [authentication_details] of type [flattened]')
================================================================================================== 1 failed, 11 passed, 19 warnings in 10.22s ==================================================================================================

[shashank-elastic]

Manually Checked. Things are good

detection-rules on  5306-bug-esql-schema-field-pruning [$?] is 📦 v1.5.9 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 6s 
❯  python -m detection_rules dev test esql-remote-validation

Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

/Users/shashankks/elastic_workspace/detection-rules/detection_rules/index_mappings.py:367: ElasticsearchWarning: No limit defined, adding default limit of [1000]
  response = elastic_client.esql.query(query=query)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/index_mappings.py:367: ElasticsearchWarning: Line 57:5: Field 'Esql.azure_signinlogs_result_description_values' shadowed by field at line 64:5
  response = elastic_client.esql.query(query=query)
ESQL rules loaded: 67
Total rules: 67
Failed rules: 0
Failed rules written to failed_rules.log

[eric-forte-elastic]

Backport Checks 🟢

Details

detection-rules on  5306-bug-esql-schema-field-pruning is  v1.5.9 via   on  eric.forte 
❯ git checkout 8.19 -- rules_building_block

detection-rules on  5306-bug-esql-schema-field-pruning is  v1.5.9 via   on  eric.forte 
❯ git checkout 8.19 -- rules

detection-rules on  5306-bug-esql-schema-field-pruning [+] is  v1.5.9 via   on  eric.forte 
❯ source env/detection-rules-build/bin/activate.fish

detection-rules on  5306-bug-esql-schema-field-pruning [+] is  v1.5.9 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ export DR_REMOTE_ESQL_VALIDATION=True

detection-rules on  5306-bug-esql-schema-field-pruning [+] is  v1.5.9 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ python -m detection_rules dev test esql-remote-validation
Loaded config file: /tmp/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Total rules: 67
Failed rules: 0
Failed rules written to failed_rules.log

In branches 9.0 and onward, there are no rule differences in the packages so the test from main is equivalent:

detection-rules on  5306-bug-esql-schema-field-pruning is  v1.5.9 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ git checkout 9.0 -- rules_building_block

detection-rules on  5306-bug-esql-schema-field-pruning is  v1.5.9 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ git checkout 9.0 -- rules

detection-rules on  5306-bug-esql-schema-field-pruning is  v1.5.9 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ git status
On branch 5306-bug-esql-schema-field-pruning
Your branch is up to date with 'origin/5306-bug-esql-schema-field-pruning'.

nothing to commit, working tree clean

detection-rules on  5306-bug-esql-schema-field-pruning is  v1.5.9 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ git checkout 9.1 -- rules_building_block

detection-rules on  5306-bug-esql-schema-field-pruning is  v1.5.9 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ git checkout 9.1 -- rules

detection-rules on  5306-bug-esql-schema-field-pruning is  v1.5.9 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ git status
On branch 5306-bug-esql-schema-field-pruning
Your branch is up to date with 'origin/5306-bug-esql-schema-field-pruning'.

nothing to commit, working tree clean

detection-rules on  5306-bug-esql-schema-field-pruning is  v1.5.9 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ git checkout 9.2 -- rules_building_block

detection-rules on  5306-bug-esql-schema-field-pruning is  v1.5.9 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ git checkout 9.2 -- rules

detection-rules on  5306-bug-esql-schema-field-pruning is  v1.5.9 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ git status
On branch 5306-bug-esql-schema-field-pruning
Your branch is up to date with 'origin/5306-bug-esql-schema-field-pruning'.

nothing to commit, working tree clean

detection-rules on  5306-bug-esql-schema-field-pruning is  v1.5.9 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ 

[Mikaayenson]

🚢