Only load and scan evtx files based on loaded rules

[hitenkoku]

refs: #1317

Updated by Zach:
For clarification, for example, if a user chooses to scan with only a single rule that searches for events that have Channel: Security, then we should not load evtx files that have a Channel that is not Security. In most scenarios, there will not be multiple channels in a single .evtx file (unless the file was artificially created), so Hayabusa can check what channel is being used in the .evtx file by checking the first record's Channel field. In order to provide the ability to search all .evtx files regardless of rules, we will need to add a --scan-all-evtx-files option.
Determining what Channel a rule uses should be based on the Channel field defined in the rule (if there is one).
One thing we need to be careful about though, is that some rules do not have a service or category defined, and therefore do not have the Channel field defined on purpose because they are designed to scan all logs. There are currently only 2 rules that do this.

• https://github.com/SigmaHQ/sigma/blob/master/rules/windows/builtin/win_alert_mimikatz_keywords.yml
• https://github.com/Yamato-Security/hayabusa-rules/blob/main/hayabusa/builtin/UnkwnChannEID_Med_PossibleHiddenShellcode.yml

The first rule looks in all .evtx files for keywords that are related to Mimikatz and the second one searches all logs for hidden shellcode.

If these rules are enabled, then I think we should not add any .evtx files to load based on them or else all .evtx files would have to be loaded and it would not be possible to implement this issue. So for example, if a user ran the -r option and specified only one of these rules and did NOT also specify the --scan-all-evtx-files option, then Hayabusa should not load any .evtx files and the scan should finish immediately with no results.

This issue is to be implemented in csv-timeline and json-timeline commands.

Here is the flow of what I am thinking of.
+-----------------------+
| Load Sigma Rules |
| Extract Channels |
+-----------------------+
|
V
+-----------------------+
| Scan .evtx Directories|
| Read 1st Record of |
| Each .evtx for Channel|
+-----------------------+
|
V
+-----------------------+
| Match .evtx Channels |
| with Sigma Channels |
+-----------------------+
|
V
+-----------------------+
| Load Matched .evtx |
| Files Only |
+-----------------------+
|
V
+-----------------------+
| Filter Sigma Rules |
| Based on .evtx |
| File Channels |
+-----------------------+
|
V
+-----------------------+
| Process Scans Using |
| Applicable Sigma Rules|
+-----------------------+

This issue should be implemented before the issue of enabling only applicable sigma rules. (#1317)

I also want to inform the user about how many .evtx files are actually being scanned and how many .yml rules are actually being enabled.

Before:

Noisy rules: 12 (Disabled)

Stable rules: 107 (6.16%)
Test rules: 1,629 (93.84%)

Hayabusa rules: 19
Sigma rules: 1,717
Total enabled detection rules: 1,736

Output profile: standard

Scanning in progress. Please wait.

After:

Noisy rules: 12 (Disabled)

Stable rules: 107 (6.16%)
Test rules: 1,629 (93.84%)

Hayabusa rules: 19
Sigma rules: 1,717
Total specified detection rules: 1,736

Evtx files loaded after channel filter: XXX
Detection rules loaded after channel filter: XXX

Output profile: standard

Scanning in progress. Please wait.

• The .evtx files shown in the progress bar should change depending on whether the Channel filter is enabled or not.
• If --enable-all-rules is enabled, then we do not display Detection rules loaded after channel filter: xxxx
• If --scan-all-evtx-files is enabled, then we do not display Evtx files loaded after channel filter: xxx