Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: extract PowerShell Classic data fields option #1220

Merged
merged 11 commits into from
Nov 26, 2023

Conversation

fukusuket
Copy link
Collaborator

@fukusuket fukusuket commented Nov 23, 2023

What Changed

  • feat: Extract fields from PwSh classic logs #1201
    • extracts the following data in PowerShell Classic Data. The following specifications were created based on the Elastic beats settings.
      • EID: 400, 403, 600 : Event.EventData.Data[2] field
      • EID: 800 : Event.EventData.Data[1] field
    • Extract fields from Event.EventData.Data using the following logic.
      • field_split: \n\t
      • value_split: =(only first =)
    • added --no-pwsh-field-extraction option

Evidence

Enviroment

  • OS: macOS Sonoma version 14.0

Test(basic extraction)

Test EID:400

./hayabusa json-timeline -w -d ../data -r test.yml --include-eid 400 -o 400.json --field-data-extraction -C

{
    "Timestamp": "2023-10-12 16:29:06.591 +09:00",
    "Computer": "MyComputer",
    "Channel": "PwShClassic",
    "EventID": 400,
    "Level": "info",
    "RecordID": 7,
    "RuleTitle": "PwSh Engine Started",
    "Details": {
    },
    "ExtraFieldInfo": {
        "CommandLine": "",
        "CommandName": "",
        "CommandPath": "",
        "CommandType": "",
        "Data": "Available",
        "Data": "None",
        "EngineVersion": "5.1.22621.1778",
        "HostApplication": "powershell -ExecutionPolicy Bypass -windowstyle hidden -command C:\\mcj\\tool\\win10\\ssd_fw_update\\ssd_fw_update.ps1",
        "HostId": "bafb59f7-0327-4c95-9b52-e515e2c033ff",
        "HostName": "ConsoleHost",
        "HostVersion": "5.1.22621.1778",
        "NewEngineState": "Available",
        "PipelineId": "",
        "PreviousEngineState": "None",
        "RunspaceId": "8d3b7b41-5bb3-4bb3-abe4-2c75c231a021",
        "ScriptName": "",
        "SequenceNumber": 13
    }
}

Test EID:403

% ./hayabusa json-timeline -w -d ../data -r test.yml --include-eid 403 -o 403.json --field-data-extraction

{
    "Timestamp": "2023-10-12 16:29:09.432 +09:00",
    "Computer": "MyComputer",
    "Channel": "PwShClassic",
    "EventID": 403,
    "Level": "info",
    "RecordID": 8,
    "RuleTitle": "PwSh Engine Started",
    "Details": {
    },
    "ExtraFieldInfo": {
        "CommandLine": "",
        "CommandName": "",
        "CommandPath": "",
        "CommandType": "",
        "Data": "Available",
        "Data": "Stopped",
        "EngineVersion": "5.1.22621.1778",
        "HostApplication": "powershell -ExecutionPolicy Bypass -windowstyle hidden -command C:\\mcj\\tool\\win10\\ssd_fw_update\\ssd_fw_update.ps1",
        "HostId": "bafb59f7-0327-4c95-9b52-e515e2c033ff",
        "HostName": "ConsoleHost",
        "HostVersion": "5.1.22621.1778",
        "NewEngineState": "Stopped",
        "PipelineId": "",
        "PreviousEngineState": "Available",
        "RunspaceId": "8d3b7b41-5bb3-4bb3-abe4-2c75c231a021",
        "ScriptName": "",
        "SequenceNumber": 15
    }
}

Test EID:600

% ./hayabusa json-timeline -w -d ../data -r test.yml --include-eid 600 -o 600.json --field-data-extraction

{
    "Timestamp": "2023-10-12 16:29:06.435 +09:00",
    "Computer": "MyComputer",
    "Channel": "PwShClassic",
    "EventID": 600,
    "Level": "info",
    "RecordID": 1,
    "RuleTitle": "PwSh Engine Started",
    "Details": {
    },
    "ExtraFieldInfo": {
        "CommandLine": "",
        "CommandName": "",
        "CommandPath": "",
        "CommandType": "",
        "Data": "Registry",
        "Data": "Started",
        "EngineVersion": "",
        "HostApplication": "powershell -ExecutionPolicy Bypass -windowstyle hidden -command C:\\mcj\\tool\\win10\\ssd_fw_update\\ssd_fw_update.ps1",
        "HostId": "bafb59f7-0327-4c95-9b52-e515e2c033ff",
        "HostName": "ConsoleHost",
        "HostVersion": "5.1.22621.1778",
        "NewProviderState": "Started",
        "PipelineId": "",
        "ProviderName": "Registry",
        "RunspaceId": "",
        "ScriptName": "",
        "SequenceNumber": 1
    }
}

Test EID:800

% ./hayabusa json-timeline -w -d ../data -r test.yml --include-eid 800 -o 800.json --field-data-extraction

{
    "Timestamp": "2023-10-12 17:14:41.284 +09:00",
    "Computer": "MyComputer",
    "Channel": "PwShClassic",
    "EventID": 800,
    "Level": "info",
    "RecordID": 351,
    "RuleTitle": "PwSh Engine Started",
    "Details": {
    },
    "ExtraFieldInfo": {
        "CommandLine": "&{ Add-Type -AssemblyName 'PresentationFramework', 'System.Windows.Forms'; $path = 'C:\\MCJ\\OA3'; $buffer = New-Object Byte[] 1024; $items = Get-ChildItem $path -force -recurse | Where-Object { $_.Attributes -notcontains 'Directory' }; $items += Get-ChildItem $( [System.Environment]::GetFolderPath( 'Desktop' ) ) -Filter oa3mgr*.lnk; $icon = New-Object System.Windows.Forms.NotifyIcon; $icon.Icon = [System.IO.Path]::Combine( $path, 'mouse.ico' ); $icon.Text = 'OA3 ファイル削除'; $icon.Visible = $true; Write-Host '削除開始'; $icon.BalloonTipTitle = $icon.Text; $icon.BalloonTipIcon = 'Info'; $icon.BalloonTipText = '削除開始'; $icon.ShowBalloonTip( 10000 ); Start-Sleep -s 5; for( $index = 0; $index -lt $items.Count; $index++ ) { $filename = $items[ $index ].FullName; if( [System.IO.File]::Exists( $filename ) ) { [System.IO.File]::SetAttributes( $filename, [System.IO.FileAttributes]::Normal ); $file = [System.IO.File]::OpenWrite( $filename ); Write-Host $( [System.String]::Format( '{0}:ゼロクリア', $filename ) ); while( $file.Position -lt $file.Length ) { $file.Write( $buffer, 0, 1024 ); }; $file.Close(); $file.Dispose(); $icon.BalloonTipText = $( [System.String]::Format( '{0}:削除', $filename ) ); $icon.ShowBalloonTip( 10000 ); Write-Host $( [System.String]::Format( '{0}:削除', $filename ) ); [System.IO.File]::Delete( $filename ); }; }; Set-Location -Path $( [System.IO.Path]::Combine( $path, '..' ) ); Remove-Item $path -force -recurse; $window = New-Object System.Windows.Window; $window.Topmost = $true; if( [System.IO.Directory]::Exists( $path ) ) { $icon.BalloonTipIcon = 'Error'; $icon.BalloonTipText = $( [System.String]::Format( 'ファイルの削除に失敗しました。セキュリティツールなどを使用し{0}以下を完全に削除をしてください。', $path ) ); $icon.BalloonTipTitle = '削除失敗'; $icon.ShowBalloonTip( 10000 ); [void][System.Windows.MessageBox]::Show( $window, $( [System.String]::Format( 'ファイルの削除に失敗しました。セキュリティツールなどを使用し{0}以下を完全に削除をしてください。', $path ) ), '削除失敗', [System.Windows.MessageBoxButton]::OK, [System.Windows.MessageBoxImage]::Error ); } else { $icon.BalloonTipIcon = 'Info'; $icon.BalloonTipText = $( [System.String]::Format( 'ファイルを完全に削除しました。', $path ) ); $icon.BalloonTipTitle = '削除成功'; $icon.ShowBalloonTip( 10000 ); [void][System.Windows.MessageBox]::Show( $window, $( [System.String]::Format( 'ファイルを完全に削除しました。', $path ) ), '削除成功', [System.Windows.MessageBoxButton]::OK, [System.Windows.MessageBoxImage]::Information ); } $window.Close(); $icon.Visible = $false; $icon.Dispose(); }",
        "Data": "&{ Add-Type -AssemblyName 'PresentationFramework', 'System.Windows.Forms'; $path = 'C:\\MCJ\\OA3'; $buffer = New-Object Byte[] 1024; $items = Get-ChildItem $path -force -recurse | Where-Object { $_.Attributes -notcontains 'Directory' }; $items += Get-ChildItem $( [System.Environment]::GetFolderPath( 'Desktop' ) ) -Filter oa3mgr*.lnk; $icon = New-Object System.Windows.Forms.NotifyIcon; $icon.Icon = [System.IO.Path]::Combine( $path, 'mouse.ico' ); $icon.Text = 'OA3 ファイル削除'; $icon.Visible = $true; Write-Host '削除開始'; $icon.BalloonTipTitle = $icon.Text; $icon.BalloonTipIcon = 'Info'; $icon.BalloonTipText = '削除開始'; $icon.ShowBalloonTip( 10000 ); Start-Sleep -s 5; for( $index = 0; $index -lt $items.Count; $index++ ) { $filename = $items[ $index ].FullName; if( [System.IO.File]::Exists( $filename ) ) { [System.IO.File]::SetAttributes( $filename, [System.IO.FileAttributes]::Normal ); $file = [System.IO.File]::OpenWrite( $filename ); Write-Host $( [System.String]::Format( '{0}:ゼロクリア', $filename ) ); while( $file.Position -lt $file.Length ) { $file.Write( $buffer, 0, 1024 ); }; $file.Close(); $file.Dispose(); $icon.BalloonTipText = $( [System.String]::Format( '{0}:削除', $filename ) ); $icon.ShowBalloonTip( 10000 ); Write-Host $( [System.String]::Format( '{0}:削除', $filename ) ); [System.IO.File]::Delete( $filename ); }; }; Set-Location -Path $( [System.IO.Path]::Combine( $path, '..' ) ); Remove-Item $path -force -recurse; $window = New-Object System.Windows.Window; $window.Topmost = $true; if( [System.IO.Directory]::Exists( $path ) ) { $icon.BalloonTipIcon = 'Error'; $icon.BalloonTipText = $( [System.String]::Format( 'ファイルの削除に失敗しました。セキュリティツールなどを使用し{0}以下を完全に削除をしてください。', $path ) ); $icon.BalloonTipTitle = '削除失敗'; $icon.ShowBalloonTip( 10000 ); [void][System.Windows.MessageBox]::Show( $window, $( [System.String]::Format( 'ファイルの削除に失敗しました。セキュリティツールなどを使用し{0}以下を完全に削除をしてください。', $path ) ), '削除失敗', [System.Windows.MessageBoxButton]::OK, [System.Windows.MessageBoxImage]::Error ); } else { $icon.BalloonTipIcon = 'Info'; $icon.BalloonTipText = $( [System.String]::Format( 'ファイルを完全に削除しました。', $path ) ); $icon.BalloonTipTitle = '削除成功'; $icon.ShowBalloonTip( 10000 ); [void][System.Windows.MessageBox]::Show( $window, $( [System.String]::Format( 'ファイルを完全に削除しました。', $path ) ), '削除成功', [System.Windows.MessageBoxButton]::OK, [System.Windows.MessageBoxImage]::Information ); } $window.Close(); $icon.Visible = $false; $icon.Dispose(); }",
        "Data": "CommandInvocation(Add-Type): \"Add-Type\"\\r\\nパラメーター バインド(Add-Type): 名前=\"AssemblyName\"; 値=\"PresentationFramework, System.Windows.Forms\"",
        "DetailSequence": 1,
        "DetailTotal": 1,
        "EngineVersion": "5.1.22621.1778",
        "HostApplication": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\PowerShell.exe -STA -NoLogo -NoProfile -Command &{ Add-Type -AssemblyName 'PresentationFramework', 'System.Windows.Forms'; $path = 'C:\\MCJ\\OA3'; $buffer = New-Object Byte[] 1024; $items = Get-ChildItem $path -force -recurse | Where-Object { $_.Attributes -notcontains 'Directory' }; $items += Get-ChildItem $( [System.Environment]::GetFolderPath( 'Desktop' ) ) -Filter oa3mgr*.lnk; $icon = New-Object System.Windows.Forms.NotifyIcon; $icon.Icon = [System.IO.Path]::Combine( $path, 'mouse.ico' ); $icon.Text = 'OA3 ファイル削除'; $icon.Visible = $true; Write-Host '削除開始'; $icon.BalloonTipTitle = $icon.Text; $icon.BalloonTipIcon = 'Info'; $icon.BalloonTipText = '削除開始'; $icon.ShowBalloonTip( 10000 ); Start-Sleep -s 5; for( $index = 0; $index -lt $items.Count; $index++ ) { $filename = $items[ $index ].FullName; if( [System.IO.File]::Exists( $filename ) ) { [System.IO.File]::SetAttributes( $filename, [System.IO.FileAttributes]::Normal ); $file = [System.IO.File]::OpenWrite( $filename ); Write-Host $( [System.String]::Format( '{0}:ゼロクリア', $filename ) ); while( $file.Position -lt $file.Length ) { $file.Write( $buffer, 0, 1024 ); }; $file.Close(); $file.Dispose(); $icon.BalloonTipText = $( [System.String]::Format( '{0}:削除', $filename ) ); $icon.ShowBalloonTip( 10000 ); Write-Host $( [System.String]::Format( '{0}:削除', $filename ) ); [System.IO.File]::Delete( $filename ); }; }; Set-Location -Path $( [System.IO.Path]::Combine( $path, '..' ) ); Remove-Item $path -force -recurse; $window = New-Object System.Windows.Window; $window.Topmost = $true; if( [System.IO.Directory]::Exists( $path ) ) { $icon.BalloonTipIcon = 'Error'; $icon.BalloonTipText = $( [System.String]::Format( 'ファイルの削除に失敗しました。セキュリティツールなどを使用し{0}以下を完全に削除をしてください。', $path ) ); $icon.BalloonTipTitle = '削除失敗'; $icon.ShowBalloonTip( 10000 ); [void][System.Windows.MessageBox]::Show( $window, $( [System.String]::Format( 'ファイルの削除に失敗しました。セキュリティツールなどを使用し{0}以下を完全に削除をしてください。', $path ) ), '削除失敗', [System.Windows.MessageBoxButton]::OK, [System.Windows.MessageBoxImage]::Error ); } else { $icon.BalloonTipIcon = 'Info'; $icon.BalloonTipText = $( [System.String]::Format( 'ファイルを完全に削除しました。', $path ) ); $icon.BalloonTipTitle = '削除成功'; $icon.ShowBalloonTip( 10000 ); [void][System.Windows.MessageBox]::Show( $window, $( [System.String]::Format( 'ファイルを完全に削除しました。', $path ) ), '削除成功', [System.Windows.MessageBoxButton]::OK, [System.Windows.MessageBoxImage]::Information ); } $window.Close(); $icon.Visible = $false; $icon.Dispose(); }",
        "HostId": "09b8009b-26e6-4525-a5e1-ffa15269fde1",
        "HostName": "ConsoleHost",
        "HostVersion": "5.1.22621.1778",
        "PipelineId": 1,
        "RunspaceId": "5e6fd261-10f2-47b5-8235-87f4615f1850",
        "ScriptName": "",
        "SequenceNumber": 15,
        "UserId": "MYCOMPUTER\\Administrator"
    }
}

@fukusuket
Copy link
Collaborator Author

Test(default option)

When the option was not enabled, I confirmed that there were no differences with the results of 2.10.1.
csv-timline

% ./hayabusa-2.10.1-mac-arm csv-timeline -d ../hayabusa-sample-evtx -o old.csv -w -C
% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -o new.csv -w -C
% diff new.csv old.csv
%

json-timline

% ./hayabusa-2.10.1-mac-arm json-timeline -d ../hayabusa-sample-evtx -o old.json -w -C
% ./hayabusa json-timeline -d ../hayabusa-sample-evtx -o new.json -w -C
% diff old.json new.json
%

@fukusuket
Copy link
Collaborator Author

fukusuket commented Nov 23, 2023

Test(benchmark)

https://github.com/NextronSystems/evtx-baseline/releases/tag/v0.7

version option elasped time memory detection
main -w 00:05:56.057 7.0 GiB 1,601,208 / 4,817,181
This PR -w 00:05:42.214 7.0 GiB 1,601,208 / 4,817,181
This PR -w --field-data-extraction 00:05:52.556 7.0 GiB 1,601,179 / 4,817,181

(The reason there is a difference in the number of detections is because the Data field has been replaced.
Rules that were detected using the Data field will no longer be detected when this option is enabled.) 

@fukusuket
Copy link
Collaborator Author

fukusuket commented Nov 23, 2023

Known issue

The data of EID:800 of hayabusa-sample-evtx seems to contain some irregular data such as the following.
Below is the output of version 2.10.1. This PR does not support extraction of irregular cases such as the following.

Only one Data field and Some custom? field

(There are usually 3 Data fields)

{
    "Timestamp": "2021-04-23 19:09:29.971 +09:00",
    "Computer": "srvdefender01.offsec.lan",
    "Channel": "PwShClassic",
    "EventID": 800,
    "Level": "info",
    "RecordID": 7019,
    "RuleTitle": "PwSh Engine Started",
    "Details": {
    },
    "ExtraFieldInfo": {
        "Culture": "en-US",
        "Data": "Microsoft.PowerShell.Core\\Set-StrictMode -Off\\r\\n\\tDetailSequence=1\\r\\n\\tDetailTotal=1\\r\\n\\r\\n\\tSequenceNumber=617\\r\\n\\r\\n\\tUserId=OFFSEC\\admmig\\r\\n\\tHostName=ConsoleHost\\r\\n\\tHostVersion=5.1.14393.3383\\r\\n\\tHostId=d72f218b-82c7-499c-97cb-c80e944516ab\\r\\n\\tHostApplication=C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\\r\\n\\tEngineVersion=5.1.14393.3383\\r\\n\\tRunspaceId=4ecdbb14-73d6-41a0-8b4a-662d7fa34856\\r\\n\\tPipelineId=76\\r\\n\\tScriptName=C:\\Program Files\\WindowsPowerShell\\Modules\\PSReadline\\1.2\\PSReadLine.psm1\\r\\n\\tCommandLine= Microsoft.PowerShell.Core\\Set-StrictMode -Off\\r\\nCommandInvocation(Set-StrictMode): \"Set-StrictMode\"\\r\\nParameterBinding(Set-StrictMode): name=\"Off\"; value=\"True\"",
        "Keyword": "Classic",
        "Level": "Information",
        "Message": "Pipeline execution details for command line: Microsoft.PowerShell.Core\\Set-StrictMode -Off\\r\\n. \\r\\n\\r\\nContext Information: \\r\\n\\tDetailSequence=1\\r\\n\\tDetailTotal=1\\r\\n\\r\\n\\tSequenceNumber=617\\r\\n\\r\\n\\tUserId=OFFSEC\\admmig\\r\\n\\tHostName=ConsoleHost\\r\\n\\tHostVersion=5.1.14393.3383\\r\\n\\tHostId=d72f218b-82c7-499c-97cb-c80e944516ab\\r\\n\\tHostApplication=C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\\r\\n\\tEngineVersion=5.1.14393.3383\\r\\n\\tRunspaceId=4ecdbb14-73d6-41a0-8b4a-662d7fa34856\\r\\n\\tPipelineId=76\\r\\n\\tScriptName=C:\\Program Files\\WindowsPowerShell\\Modules\\PSReadline\\1.2\\PSReadLine.psm1\\r\\n\\tCommandLine= Microsoft.PowerShell.Core\\Set-StrictMode -Off\\r\\n \\r\\n\\r\\nDetails: \\r\\nCommandInvocation(Set-StrictMode): \"Set-StrictMode\"\\r\\nParameterBinding(Set-StrictMode): name=\"Off\"; value=\"True\"",
        "Opcode": "Info",
        "Task": "Pipeline Execution Details"
    }
}

Only one Data field

(There are usually 3 Data fields)

{
    "Timestamp": "2021-06-10 23:12:46.198 +09:00",
    "Computer": "fs01.offsec.lan",
    "Channel": "PwShClassic",
    "EventID": 800,
    "Level": "info",
    "RecordID": 28813,
    "RuleTitle": "PwSh Engine Started",
    "Details": {
    },
    "ExtraFieldInfo": {
        "Data": "Set-WmiInstance -Namespace root/subscription -Class __FilterToConsumerBinding -Arguments $FilterToConsumerArgs @Arguments | Out-Null\\r\\n\\tDetailSequence=1\\r\\n\\tDetailTotal=1\\r\\n\\r\\n\\tSequenceNumber=295\\r\\n\\r\\n\\tUserId=OFFSEC\\admmig\\r\\n\\tHostName=ConsoleHost\\r\\n\\tHostVersion=5.1.14393.3383\\r\\n\\tHostId=cebe54d1-4235-4f11-9fd4-5319a4253d0e\\r\\n\\tHostApplication=C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\\r\\n\\tEngineVersion=5.1.14393.3383\\r\\n\\tRunspaceId=147302b5-06ec-4bbb-a604-7772a0f40632\\r\\n\\tPipelineId=69\\r\\n\\tScriptName=C:\\Users\\admmig\\Desktop\\powerlurk.ps1\\r\\n\\tCommandLine= Set-WmiInstance -Namespace root/subscription -Class __FilterToConsumerBinding -Arguments $FilterToConsumerArgs @Arguments | Out-Null\\r\\nCommandInvocation(Set-WmiInstance): \"Set-WmiInstance\"\\r\\nParameterBinding(Set-WmiInstance): name=\"Namespace\"; value=\"root/subscription\"\\r\\nParameterBinding(Set-WmiInstance): name=\"Class\"; value=\"__FilterToConsumerBinding\"\\r\\nParameterBinding(Set-WmiInstance): name=\"Arguments\"; value=\"System.Collections.Hashtable\"\\r\\nCommandInvocation(Out-Null): \"Out-Null\"\\r\\nParameterBinding(Out-Null): name=\"InputObject\"; value=\"__FilterToConsumerBinding.Consumer=\"CommandLineEventConsumer.Name=\\\"Evil23\\\"\",Filter=\"__EventFilter.Name=\\\"Evil23\\\"\"\""
    }
}

@fukusuket fukusuket self-assigned this Nov 23, 2023
@fukusuket fukusuket added the enhancement New feature or request label Nov 23, 2023
@fukusuket fukusuket linked an issue Nov 23, 2023 that may be closed by this pull request
@fukusuket
Copy link
Collaborator Author

The option is tentatively set to field-data-extraction, but it can be changed :)
I would appreciate it if you could review when you have time🙏

@fukusuket fukusuket marked this pull request as ready for review November 23, 2023 15:48
Copy link
Collaborator

@hitenkoku hitenkoku left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@YamatoSecurity
Copy link
Collaborator

@fukusuket Look great! Thank you!
I took some benchmarks and elapsed time and memory usage only differ by about 1~2% so I think we can turn this on by default. What about having it enabled by default and be able to turn off with a --no-pwsh-field-extraction option?
For the known issue of The data of EID:800 of hayabusa-sample-evtx seems to contain some irregular data, I think that is fine to ignore for now.

Just one thing though, it would be nice if we can still detect on the original Data field. I was imagining that we still write detections for the Data field but when outputting, we extract the fields. I suppose it would be ideal to be able to do both. That is, be able to detect with Data but also with more specific extracted fields to reduce FPs. Is it possible to add detection for Data?

By the way, is it possible to specify with eventkey aliases certain fields to output in the Details field? Or is everything hardcoded to be outputted into ExtraFieldInfo?

@fukusuket
Copy link
Collaborator Author

@YamatoSecurity
Thank you so much for review :)

Is it possible to add detection for Data?

Yes, I agree that it is better! In order to be able to detect the Data field, we also need to output the Data field before extraction as shown below, but what do you think? (Although the size of the output file will be larger, I think it would be better to also output the Data field before extraction so that the user can check why it was detected.)

{
    "Timestamp": "2023-10-12 16:29:06.591 +09:00",
    "Computer": "MyComputer",
    "Channel": "PwShClassic",
    "EventID": 400,
    "Level": "info",
    "RecordID": 7,
    "RuleTitle": "PwSh Engine Started",
    "Details": {
    },
    "ExtraFieldInfo": {
        "CommandLine": "",
        "CommandName": "",
        "CommandPath": "",
        "CommandType": "",
        "Data": "Available",
        "Data": "None",
        "Data": "NewEngineState=Available\\r\\n\\tPreviousEngineState=None\\r\\n\\r\\n\\tSequenceNumber=13\\r\\n\\r\\n\\tHostName=ConsoleHost\\r\\n\\tHostVersion=5.1.22621.1778\\r\\n\\tHostId=bafb59f7-0327-4c95-9b52-e515e2c033ff\\r\\n\\tHostApplication=powershell -ExecutionPolicy Bypass -windowstyle hidden -command C:\\mcj\\tool\\win10\\ssd_fw_update\\ssd_fw_update.ps1\\r\\n\\tEngineVersion=5.1.22621.1778\\r\\n\\tRunspaceId=8d3b7b41-5bb3-4bb3-abe4-2c75c231a021\\r\\n\\tPipelineId=\\r\\n\\tCommandName=\\r\\n\\tCommandType=\\r\\n\\tScriptName=\\r\\n\\tCommandPath=\\r\\n\\tCommandLine=",
        "EngineVersion": "5.1.22621.1778",
        "HostApplication": "powershell -ExecutionPolicy Bypass -windowstyle hidden -command C:\\mcj\\tool\\win10\\ssd_fw_update\\ssd_fw_update.ps1",
        "HostId": "bafb59f7-0327-4c95-9b52-e515e2c033ff",
        "HostName": "ConsoleHost",
        "HostVersion": "5.1.22621.1778",
        "NewEngineState": "Available",
        "PipelineId": "",
        "PreviousEngineState": "None",
        "RunspaceId": "8d3b7b41-5bb3-4bb3-abe4-2c75c231a021",
        "ScriptName": "",
        "SequenceNumber": 13
    }
}

By the way, is it possible to specify with eventkey aliases certain fields to output in the Details field? Or is everything hardcoded to be outputted into ExtraFieldInfo?

Yes, you can specify it in Details (but it's not output right now, so check it...). Also, the key is only separated by "\n\t" and "=", so it is not hard coded.

Copy link

codecov bot commented Nov 24, 2023

Codecov Report

Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (919f593) 83.34% compared to head (15a1fab) 83.48%.

Files Patch % Lines
src/detections/configs.rs 86.66% 2 Missing ⚠️
src/detections/field_extract.rs 98.70% 2 Missing ⚠️
src/detections/rule/mod.rs 60.00% 2 Missing ⚠️
src/detections/utils.rs 93.75% 1 Missing ⚠️
src/main.rs 90.90% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1220      +/-   ##
==========================================
+ Coverage   83.34%   83.48%   +0.14%     
==========================================
  Files          26       27       +1     
  Lines       23835    24082     +247     
==========================================
+ Hits        19866    20106     +240     
- Misses       3969     3976       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@fukusuket
Copy link
Collaborator Author

@YamatoSecurity
I have implemented #1220 (comment)
I would appreciate it if you could review when you have time🙏

@fukusuket
Copy link
Collaborator Author

fukusuket commented Nov 24, 2023

Test(rule details extraction)

I confirmed that the field after extraction is output to details as follows using the following rules.
also confirmed that detection in the HostApplication field is also possible.

author: test
date: 2023/11/24
modified: 2023/11/24

title: 'PwSh Classic TEST'
details: 'HostVersion: %HostVersion% ¦ HostApplication: %HostApplication%'

id: ac2ae63b-83e6-4d06-aeaf-07409bda92c9
level: informational
status: test
logsource:
    product: windows
    service: powershell
detection:
    selection:
        Channel: 'Windows PowerShell'
    selection_2:
        - EventID: 400
        - EventID: 403
        - EventID: 600
        - EventID: 800
    selection_3:
        HostApplication|contains: ' -nop '
    condition: selection and selection_2 and selection_3
ruletype: Hayabusa
{
    "Timestamp": "2021-10-20 23:39:26.000 +09:00",
    "Computer": "FS03.offsec.lan",
    "Channel": "PwShClassic",
    "EventID": 400,
    "Level": "info",
    "RecordID": 132,
    "RuleTitle": "PwSh Classic TEST",
    "Details": {
        "HostApplication": "powershell.exe -NoP -C C:\\Windows\\System32\\rundll32.exe C:\\Windows\\System32\\comsvcs.dll, MiniDump (Get-Process lsass).Id \\Windows\\Temp\\vtnr8kff.dmp full;Wait-Process -Id (Get-Process rundll32).id",
        "HostVersion": "4.0"
    },
    "ExtraFieldInfo": {
        "CommandLine": "",
        "CommandName": "",
        "CommandPath": "",
        "CommandType": "",
        "Data": "Available",
        "Data": "NewEngineState=Available\\r\\n\\tPreviousEngineState=None\\r\\n\\r\\n\\tSequenceNumber=13\\r\\n\\r\\n\\tHostName=ConsoleHost\\r\\n\\tHostVersion=4.0\\r\\n\\tHostId=63c2be9c-cd2c-46d7-ac07-01a9e98f2400\\r\\n\\tHostApplication=powershell.exe -NoP -C C:\\Windows\\System32\\rundll32.exe C:\\Windows\\System32\\comsvcs.dll, MiniDump (Get-Process lsass).Id \\Windows\\Temp\\vtnr8kff.dmp full;Wait-Process -Id (Get-Process rundll32).id\\r\\n\\tEngineVersion=4.0\\r\\n\\tRunspaceId=1668d864-c13c-442c-bf70-3fe6a4dee2ca\\r\\n\\tPipelineId=\\r\\n\\tCommandName=\\r\\n\\tCommandType=\\r\\n\\tScriptName=\\r\\n\\tCommandPath=\\r\\n\\tCommandLine=",
        "Data": "None",
        "HostId": "63c2be9c-cd2c-46d7-ac07-01a9e98f2400",
        "HostName": "ConsoleHost",
        "NewEngineState": "Available",
        "PipelineId": "",
        "PreviousEngineState": "None",
        "RunspaceId": "1668d864-c13c-442c-bf70-3fe6a4dee2ca",
        "ScriptName": "",
        "SequenceNumber": 13
    }
}

@fukusuket
Copy link
Collaborator Author

fukusuket commented Nov 24, 2023

Test(benchmark)

https://github.com/NextronSystems/evtx-baseline/releases/tag/v0.8
(A new dataset was released that included Windows 11 :) Total file size: 8.8 GB)

version option elasped time memory detection filesize
main -w 00:09:09.172 12.0 GiB 2,445,615 / 6,611,184 3,101,452,854bytes
This PR -w --no-pwsh-field-extraction 00:09:01.020 12.0 GiB 2,445,615 / 6,611,184 3,101,452,854bytes
This PR -w 00:09:10.468 12.0 GiB 2,445,615 / 6,611,184 3,101,568,298bytes

Copy link
Collaborator

@YamatoSecurity YamatoSecurity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fukusuket LGTM! Thank you so much!

@YamatoSecurity YamatoSecurity merged commit f6bf6b3 into main Nov 26, 2023
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Extract fields from PwSh classic logs
3 participants