feat: add support for Data[x] notation and Provider_Name(field mapping feature's config)

[fukusuket]

What Changed

• Closed Support for Provider_name and Data[x] notation to the field mapping #1350
  • added support Provider_Name in data mapping config(Optional)
  • added support multiple Provider_Name in data mapping config(Optional)
  • added support Data[x] notation in data mapping config.

I would appreciate it if you could check it out when you have time🙏

[fukusuket]

Provider_Name(field_data mapping) / Data[x] notation

I used following mapping/rule.

csv-timeline

I confirmed data conversion works as follows.(However, there seems to be some data where Data[3] is 0 or Null.)

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx --include-eid 1022,1033 -C -q -w
...
2016-09-02 05:46:35.000 +09:00 · MSI Install · info · IE10Win7 · App · 1033 · 1739 · App: EMET 5.51 ¦ Ver: 5.51 ¦ Lang: English ¦ StatusCode: 1603 ¦ Vendor: Microsoft Corporation · Binary: 7B41383630384530462D353642382D343635432D413736322D3836443638464634464337327D3030303065313034636234326138383837393837373364376662393066336263643466343030303030393034 ¦ Data:  ¦ Data: (NULL)
2016-09-02 05:49:36.000 +09:00 · MSI Install · info · IE10Win7 · App · 1033 · 1780 · App: Microsoft .NET Framework 4.5 ¦ Ver: 4.5.50709 ¦ Lang: 0 ¦ StatusCode: 0 ¦ Vendor: Microsoft Corporation · Binary: 7B39463631323432392D344130302D334434342D383843462D3134364441324545314639327D3030303039363430393865343862643534343434643364333032386533613730346261653030303030303030 ¦ Data:  ¦ Data: (NULL)
...

json-timeline

I confirmed data conversion works as follows.

% ./hayabusa json-timeline -d ../hayabusa-sample-evtx --include-eid 1022,1033 -C -q -w
...

{
    "Timestamp": "2016-09-02 05:50:28.000 +09:00",
    "RuleTitle": "MSI Install",
    "Level": "info",
    "Computer": "IE10Win7",
    "Channel": "App",
    "EventID": 1033,
    "RecordID": 1795,
    "Details": {
        "App": "EMET 5.51",
        "Ver": "5.51",
        "Lang": "English",
        "StatusCode": 0,
        "Vendor": "Microsoft Corporation"
    },
    "ExtraFieldInfo": {
        "Binary": "7B41383630384530462D353642382D343635432D413736322D3836443638464634464337327D3030303065313034636234326138383837393837373364376662393066336263643466343030303030393034",
        "Data": ["", "(NULL)"]
    }
}

[fukusuket]

hayabusa-sample-evtx

I compared result with main and there is no diff as follows.

csv

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -D -n -u -q -C -o new.csv
% ./hayabusa-main csv-timeline -d ../hayabusa-sample-evtx -w -D -n -u -q -C -o old.csv
% diff new.csv old.csv
%

json

% ./hayabusa json-timeline -d ../hayabusa-sample-evtx -w -D -n -u -q -C -o new.json
% ./hayabusa-main json-timeline -d ../hayabusa-sample-evtx -w -D -n -u -q -C -o old.json
% diff new.json old.json
%

integration-test

it completed successfully as follows.
https://github.com/Yamato-Security/hayabusa/actions/runs/9207376537

[fukusuket]

Multiple Provider_Name(field_data mapping)

I put following multiple Provider_Name data config.

Title: 'TEST'
Channel: Application
EventID: 1
Provider_Name:
    - 'Microsoft-Windows-Audit-CVE'
    - 'Audit-CVE'
RewriteFieldData:
    CVEID:
        - 'CVE': '**CONVERTED!!**'

and confirmed field data conversion(CVEID 's value)works as follows.

% ./hayabusa json-timeline -f ../hayabusa-sample-evtx/YamatoSecurity/Vulnerabilities/App_1_CVE-Detected.evtx -w -q -p super-verbose
...

{
    "Timestamp": "2020-01-19 03:14:29.831 +09:00",
    "RuleTitle": "Audit CVE Event",
    "Level": "crit",
    "Computer": "Isaac",
    "Channel": "App",
    "EventID": 1,
    "RuleAuthor": "Florian Roth (Nextron Systems), Zach Mathis",
    "RuleModifiedDate": "2022/10/22",
    "Status": "test",
    "RecordID": 19156,
    "Details": {
    },
    "ExtraFieldInfo": {
        "AdditionalDetails": "CA: <USERTrust ECC Certification Authority> sha1: C01B8463C8619676BA102EEBF0C30CDCED9A942B para: 06052B81040022 otherPara: 30820157020101303C06072A8648CE3D0101023100FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEFFFFFFFF0000000000000000FFFFFFFF307B0430FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEFFFFFFFF0000000000000000FFFFFFFC0430B3312FA7E23EE7E4988E056BE3F82D19181D9C6EFE8141120314088F5013875AC656398D8A2ED19D2A85C8EDD3EC2AEF031500A335926AA319A27A1D00896A6773A4827ACDAC73046104D6EB7932E25502AE5151A3609384C13041904BE25B912B77FE4333C0B1486FB0CD815522FF79F8E85A9045E8DFDA490E706BD2FC38276D0A998962A1DFA1E0361AE82AEFC24BDECD9BD856CDC2FDADB54DEEFF6ACDFC0B2BF5ABEBB21C5705D4023100FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFC7634D81F4372DDF581A0DB248B0A77AECEC196ACCC52973020101",
        "CVEID": "[**CONVERTED!!**-2020-0601] cert validation"
    },
    "MitreTactics": [
        "Exec", "PrivEsc", "Evas", "CredAccess", "LatMov", "Impact"
    ],
    "MitreTags": [
        "T1203", "T1068", "T1211", "T1212", "T1210", "T1499.004"
    ],
    "Provider": "MS-Win-Audit-CVE",
    "RuleCreationDate": "2020/01/15",
    "RuleFile": "win_audit_cve.yml",
    "EvtxFile": "../hayabusa-sample-evtx/YamatoSecurity/Vulnerabilities/App_1_CVE-Detected.evtx"
}

[fukusuket]

all-evtx

I confirmed that there were no performance regressions.

rev	elasped time	mem	Events with hits / Total events
main	00:10:55.2197	16.0 GiB	5,969,347 / 6,463,018
This PR	00:10:45.2180	16.0 GiB	5,969,347 / 6,463,018

[YamatoSecurity]

@fukusuket LGTM! Thanks so much!

[hitenkoku]

LGTM