[Bug] New Terms Rule Importing Failures

[eric-forte-elastic]

Describe the bug
Related to #3266

When exporting a New Terms Rule type and attempting to import it, the rule import does not successfully import the rule, it appears to be an issue with importing the new terms fields. It is expected to be an issue around lines 658-681 of rule.py for the NewTermsRuleData(QueryRuleData): class.

To Reproduce
Steps to reproduce the behavior:

1. Use detection rules to import a New Terms Rule rules_export_new_terms.ndjson.txt

2. Supply the new terms "host.id","winlog.event_data.SubjectUserName","winlog.event_data.CallerProcessName"

3. See marshmallow.exceptions.ValidationError

Full Error Ouput

detection-rules on  main [$!?] is  v0.1.0 via  v3.12.2 (detection-rules-build) on  eric.forte
❯ python -m detection_rules import-rules rules_export_new_terms.ndjson

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building rule for /home/forteea1/Code/clean_mains/detection-rules/rules/enumeration_of_privileged_local_groups_membership_duplicate.toml
new_terms (required): "host.id","winlog.event_data.SubjectUserName","winlog.event_data.CallerProcessName"
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/__main__.py", line 34, in <module>
    main()
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/__main__.py", line 31, in main
    root(prog_name="detection_rules")
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/main.py", line 118, in import_rules
    rule_prompt(rule_path, required_only=True, save=True, verbose=True, additional_required=additional, **contents)
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/cli_utils.py", line 187, in rule_prompt
    rule = TOMLRule(path=Path(path), contents=TOMLRuleContents.from_dict({'rule': contents, 'metadata': meta}))
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/mixins.py", line 109, in from_dict
    return schema.load(obj)
           ^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow_dataclass/__init__.py", line 768, in load
    all_loaded = super().load(data, many=many, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 719, in load
    return self._do_load(
           ^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 901, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'rule': [ValidationError({'language': ['Must be equal to eql.'], 'type': ['Must be equal to eql.'], 'name': ['String does not match expected pattern.'], 'new_terms': ['Unknown field.']}), ValidationError({'language': ['Must be equal to esql.'], 'type': ['Must be equal to esql.'], 'name': ['String does not match expected pattern.'], 'new_terms': ['Unknown field.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.'], 'name': ['String does not match expected pattern.'], 'new_terms': ['Unknown field.']}), ValidationError({'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.'], 'type': ['Must be equal to threat_match.'], 'name': ['String does not match expected pattern.'], 'new_terms': ['Unknown field.']}), ValidationError({'anomaly_threshold': ['Missing data for required field.'], 'type': ['Must be equal to machine_learning.'], 'machine_learning_job_id': ['Missing data for required field.'], 'name': ['String does not match expected pattern.'], 'new_terms': ['Unknown field.'], 'query': ['Unknown field.'], 'index': ['Unknown field.'], 'language': ['Unknown field.']}), ValidationError({'type': ['Must be equal to query.'], 'name': ['String does not match expected pattern.'], 'new_terms': ['Unknown field.']}), ValidationError({'name': ['String does not match expected pattern.'], 'new_terms': {'_schema': ['Invalid input type.']}})]}

Expected behavior
It is expected that when import the New Terms Rule that the rule imports successfully.

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu
• Version: 22.04 LTS

[Mikaayenson]

@eric-forte-elastic TBH there were several things wrong with the way the NewTerms class was originally implemented:

1. The main issue was that importing ndjson is not supported since the schema field names do not match the fields exported from Kibana. It does some ad-hoc transforming to convert the fields to the format expected by Kibana. There exporting still works.
2. Based on what is exported and in the UI, history_window_start: List[HistoryWindowStart] should not be a List. It's most likely why there was some list indexing within the transform method.

So where does this leave us???

We could wait until 8.15 when we have schema forgiveness and fix the schemas properly but that's a long time to wait for users that want to leverage our CLI to import/export rules re:DaC.

My recommendation is to do a small patch on the importer and then later update the schemas properly.

Here is a patch that reflects what we could do.

Patch rule_prompt

diff --git a/detection_rules/cli_utils.py b/detection_rules/cli_utils.py
index e7b0b4853..42ad2f85e 100644
--- a/detection_rules/cli_utils.py
+++ b/detection_rules/cli_utils.py
@@ -170,7 +170,19 @@ def rule_prompt(path=None, rule_type=None, required_only=True, save=True, verbos
             contents[name] = schema_prompt(name, value=kwargs.pop(name))
             continue
 
-        result = schema_prompt(name, is_required=name in required_fields, **options.copy())
+        if name == "new_terms":
+            # patch to allow new_term imports
+            result = {"field": "new_terms_fields"}
+            if "new_terms_fields" in kwargs:
+                result["value"] = schema_prompt("new_terms_fields", value=kwargs.pop("new_terms_fields"))
+            if "history_window_start" in kwargs:
+                result["history_window_start"] = [
+                    {"field": "history_window_start",
+                     "value": schema_prompt("history_window_start", value=kwargs.pop("history_window_start"))}
+                ]
+
+        else:
+            result = schema_prompt(name, is_required=name in required_fields, **options.copy())
 
         if result:
             if name not in required_fields and result == options.get('default', ''):

cc. @brokensound77