[8.18] API - AmsiScanBuffer events and new final_hook_module fields

[jdu2600]

Change Summary

• Add new AmsiScanBuffer event fields
• Add new final_hook_module fields
• Add new unique_key_v2 field

Sample values

{
  "event": {
    "provider": "Microsoft-Antimalware-Scan-Interface",
    "category": [
      "api"
    ],
    "type": [
      "allowed"
    ],
    "outcome": "unknown"
  },
  "message": "Endpoint API event - AmsiScanBuffer",
  "process": {
    "name": "cscript.exe",
    "command_line": "\"C:\\Windows\\system32\\cscript.exe\" //nologo \"C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\Health Service State\\Monitoring Host Temporary Files 1\\4870\\SecureChannel.vbs\"",
    "pid": 8660,
    "Ext": {
      "api": {
        "summary": "AmsiScanBuffer( VBScript, C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\Health Service State\\Monitoring Host Temporary Files 1\\4870\\SecureChannel.vbs, 3c933cd9f5ae963ecd674a38d0ab6764c0ea615dbe41fda4f642710e47220c34 )",
        "metadata": {
          "return_value": 1
        },
        "name": "AmsiScanBuffer",
        "parameters": {
          "content_name": "C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\Health Service State\\Monitoring Host Temporary Files 1\\4870\\SecureChannel.vbs",
          "app_name": "VBScript",
          "size": 626,
          "buffer": "IMOMScriptAPI.CreatePropertyBag();\r\nISWbemServicesEx.ExecQuery(\"Select * from Win32_ComputerSystem\", \"Unsupported parameter type 0000000a\", \"48\");\r\nISWbemObjectSet._NewEnum();\r\nISWbemObjectEx._01800001();\r\nISWbemObjectEx._01800002();\r\nIWshShell3.Run(\"cmd.exe /c nltest /sc_reset:intern.i-sone.no\", \"0\", \"true\");\r\n"
        }
      }
    }
  }
}

"call_stack_final_hook_module": {
  "path": "c:\\program files (x86)\\microsoft\\edgewebview\\application\\131.0.2903.112\\msedgewebview2.exe",
  "code_signature": [
    {
      "trusted": true,
      "subject_name": "Microsoft Corporation",
      "exists": true,
      "status": "trusted"
    }
  ],
  "hash": {
    "sha256": "4b93ebf3336c2873585ba9bc828fd602a3f97b8d653fe1abb2c103518b49c54e"
  }
}

"Memory_protection": {
      "unique_key_v2": "3204c83b1c8f72aa5da6a5c059c9b4aef13dcf930cf84dd647654bcae7b33d97",
    },

Release Target

8.18

For mapping changes:

• I ran make after making the schema changes, and committed all changes

[pzl]

The quality of the PR is good, fields added look fine

But I need to look into some problems we may have with the alert index. We have run into field limits in the past, and had to remove some unused ones to get back to a reasonable number.

And that was for total field count. We have also run into nested field limits. I will need to dig up the context to see what headroom we have, and what we did about the limits.

[pzl]

A few quick things:

this looks like its adding ~40 fields to the alerts index, not counting the enabled: false fields. Are they all necessary? Is it possible to trim the mapping down to the only necessary ones?

Do we have a specific reason for the ones that are mapped, but not enabled? I will have to look into how those are counted against the limits.

We should remove the * and explicitly put the fields we want. So if any can be omitted under that group, should be. And that prevents accidental inclusions in the future on rebuilds.

[jdu2600]

this looks like its adding ~40 fields to the alerts index

There should only be 13 new fields if I'm counting correctly.

• process.Ext.parameters.app_name
• process.Ext.parameters.buffer
• process.Ext.parameters.content_name
• process.Ext.metadata.amsi_filenames
• process.Ext.metadata.amsi_logs.entries
• process.Ext.metadata.amsi_logs.type
• process.thread.Ext.call_stack.call_stack_final_hook_module.path
• process.thread.Ext.call_stack.call_stack_final_hook_module.code_signature.exists
• process.thread.Ext.call_stack.call_stack_final_hook_module.code_signature.status
• process.thread.Ext.call_stack.call_stack_final_hook_module.code_signature.subject_name
• process.thread.Ext.call_stack.call_stack_final_hook_module.code_signature.trusted
• process.thread.Ext.call_stack.call_stack_final_hook_module.hash.sha256
• Memory_protection.unique_key_v2

292f75c makes it look like more, but but the majority of those fields already exist in the alert index - they're just defined in alerts/memory_protection_event.yaml and/or alerts/malware_event.yaml

I didn't want to modify those other files though since the new fields can technically only currently appear in rule_detection_event.yaml alert variants.

[jdu2600]

Are they all necessary? Is it possible to trim the mapping down to the only necessary ones?

Do we have a specific reason for the ones that are mapped, but not enabled? I will have to look into how those are counted against the limits.

We should remove the * and explicitly put the fields we want. So if any can be omitted under that group, should be. And that prevents accidental inclusions in the future on rebuilds.

I can't answer this. The limit of my knowledge is how to add new fields such that they exist in the schema (so that my EAF tests pass).

I don't really understand the impact to the end user on mappings etc (or lack thereof) so I flagged this PR with a few folks in slack yesterday.

[pzl]

makes it look like more, but but the majority of those fields already exist in the alert index

yes, you're right. The end result on the mapping is much smaller. 13 or so fields is much more manageable. But we should still avoid any * usage going forward.

We may want to have a larger discussion soon about schema/EAF checking. Many of the fields added in the last few releases mentioned just trying to get EAF tests passing, and and agreeing schema. But there are impacts to having things mapped (here, in this repo) outside of just having a matching schema. For most indices it doesn't matter, but we need to be stingier with alerts due to its size.

I'm guessing we may need to separate the goals of full matching schemas, and which fields require being searched and filtered on

cc @ferullo probably, when you're back

[jdu2600]

Thanks @pzl. That makes sense.
Is this PR okay to merge, or do you want me to make changes?

@nfritts - for visibility re: future larger discussion.

[jdu2600]

@pzl I removed some of the problematic alert mappings in eac91f1

[pzl]

👍 thank you

and thank you for the discussion