feat: embedded config files to binary #1370 #1385

hitenkoku · 2024-07-15T09:13:14Z

What Changed

embedded config files in binary

fukusuket

@hitenkoku
I confirmed csv result has no diff between before/after :)
But the progress bar is output twice as follows only when run without config directory🤔 Could you please confirm this?🙏
(When there is a config directory, the progress bar is printed only once)

% ./hayabusa-new csv-timeline -d ../hayabusa-sample-evtx -o new.csv -D -n -u -w -q -C
Start time: 2024/07/15 19:06

Total event log files: 585
Total file size: 137.2 MB

Loading detection rules. Please wait.

Excluded rules: 20
Noisy rules: 12

Deprecated rules: 209 (4.72%)
Experimental rules: 566 (12.78%)
Stable rules: 253 (5.71%)
Test rules: 3,355 (75.77%)
Unsupported rules: 45 (1.02%)

Hayabusa rules: 179
Sigma rules: 4,249
Total detection rules: 4,428

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 575
Detection rules enabled after channel filter: 4,355

Output profile: standard

Scanning in progress. Please wait.

[00:00:00] 0 / 575 ⠁ [                                        ] 0%

[00:00:05] 575 / 575   [========================================] 100%

Scanning finished. Please wait while the results are being saved.

hitenkoku · 2024-07-17T06:16:14Z

@fukusuket Thank you for your review.
I checking your point out problem.

…1370

hitenkoku · 2024-07-25T10:54:05Z

@fukusuket Sorry for late fix.

I fixed follwoing problem in
62e14f7 .

Could you review it?

@hitenkoku I confirmed csv result has no diff between before/after :) But the progress bar is output twice as follows only when run without config directory🤔 Could you please confirm this?🙏 (When there is a config directory, the progress bar is printed only once)

fukusuket

@hitenkoku
Thank you so much for fix! I confirmed #1385 (review) is fixed! LGTM!!🚀

YamatoSecurity · 2024-07-26T01:37:11Z

@hitenkoku Thanks for this. I have confirmed that if I rename config to config2 it still works. However, Hayabusa does not read the config files in the config directory by default so users cannot change the config files unless they recompile the binary. This is inconvenient for those who want to customize color, profiles, etc...
And it also breaks the set-default-profile command. Ex: ./target/release/hayabusa set-default-profile -p super-verbose
Event if the default profile is changed in default_profile.yaml, Hayabusa will read from the embedded file which does not change.
Can you change it so that Hayabusa only reads the embedded file if the config directory does not exist?

…1386

hitenkoku · 2024-07-27T09:55:06Z

@YamatoSecurity Fixed follwing problem. Please recheck it.

@hitenkoku Thanks for this. I have confirmed that if I rename config to config2 it still works. However, Hayabusa does not read the config files in the config directory by default so users cannot change the config files unless they recompile the binary. This is inconvenient for those who want to customize color, profiles, etc... And it also breaks the set-default-profile command. Ex: ./target/release/hayabusa set-default-profile -p super-verbose Event if the default profile is changed in default_profile.yaml, Hayabusa will read from the embedded file which does not change. Can you change it so that Hayabusa only reads the embedded file if the config directory does not exist?

YamatoSecurity · 2024-07-28T07:15:27Z

@hitenkoku Thanks! When i run the following it updates the config file correctly but it displays an error:

./target/release/hayabusa set-default-profile -p minimal
Successfully updated the default profile.

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Start time: 2024/07/28 16:13

Default profile cannot be set due to the absence of a config folder. Please check config folder.

Can you remove the Default profile cannot be set due to the absence of a config folder. Please check config folder. error message when there is a success?

Also, when I change the level_color.txt settings to the same colors:

level,colorcode
critical,00ff00
high,00ff00
medium,00ff00
low,00ff00

just to test, the colors do not change anymore. Can you check this?

…mmand #1370

hitenkoku · 2024-07-29T02:03:37Z

@YamatoSecurity Thanks for your review. I following problems. Please recheck it.

@hitenkoku Thanks! When i run the following it updates the config file correctly but it displays an error:
./target/release/hayabusa set-default-profile -p minimal
Successfully updated the default profile.

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Start time: 2024/07/28 16:13

Default profile cannot be set due to the absence of a config folder. Please check config folder.
Can you remove the Default profile cannot be set due to the absence of a config folder. Please check config folder. error message when there is a success?

Also, when I change the level_color.txt settings to the same colors:
level,colorcode
critical,00ff00
high,00ff00
medium,00ff00
low,00ff00
just to test, the colors do not change anymore. Can you check this?

YamatoSecurity · 2024-07-29T10:59:58Z

@hitenkoku Thank you!
I checked that the error message now does not get displayed:

Successfully updated the default profile.

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Start time: 2024/07/29 19:57

However, it is easy to miss the message as it gets displayed before the logo. Can you change it to:

./target/release/hayabusa set-default-profile -p minimal

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Successfully updated the default profile.

or just

./target/release/hayabusa set-default-profile -p minimal

Successfully updated the default profile.

I changed the colors for the levels in level_color.txt to

level,colorcode
critical,00ff00
high,00ff00
medium,00ff00
low,00ff00

but the colors still do not change. Can you check this?

YamatoSecurity · 2024-07-29T23:52:47Z

The mitre_tactics.txt settings also do not seem to be read from the config file, just the embedded config file.
To test, I changed attack.credential_access,CredAccess,08. Credential Access to attack.credential_access,CredHogeAccess,08. Credential Access and searched for CredHogeAccess after running Hayabusa with the super-verbose file but it still shows up as CredAccess.

Could you please check this as well.

…1370

hitenkoku · 2024-08-12T16:27:45Z

@YamatoSecurity Thanks for your comment.

I fixed following point out in c12a06a.

The mitre_tactics.txt settings also do not seem to be read from the config file, just the embedded config file. To test, I changed attack.credential_access,CredAccess,08. Credential Access to attack.credential_access,CredHogeAccess,08. Credential Access and searched for CredHogeAccess after running Hayabusa with the super-verbose file but it still shows up as CredAccess.

Could you please check this as well.

I fixed following point out in 562c117.

@hitenkoku Thank you! I checked that the error message now does not get displayed:

Successfully updated the default profile.

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Start time: 2024/07/29 19:57

However, it is easy to miss the message as it gets displayed before the logo. Can you change it to:

./target/release/hayabusa set-default-profile -p minimal

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Successfully updated the default profile.

or just

./target/release/hayabusa set-default-profile -p minimal

Successfully updated the default profile.

I fixed following your point out in dbd0417.

I changed the colors for the levels in level_color.txt to
level,colorcode
critical,00ff00
high,00ff00
medium,00ff00
low,00ff00
but the colors still do not change. Can you check this?

YamatoSecurity

@hitenkoku I checked that everything works on my environment. Thanks so much!

feat: embedded config files to binary #1370

ee9e93c

hitenkoku added the enhancement New feature or request label Jul 15, 2024

hitenkoku requested review from fukusuket and YamatoSecurity July 15, 2024 09:13

hitenkoku self-assigned this Jul 15, 2024

hitenkoku linked an issue Jul 15, 2024 that may be closed by this pull request

Embed config files in binary #1370

Closed

chore: removed debug code #1370

f0a53a0

fukusuket reviewed Jul 15, 2024

View reviewed changes

fix: fixed twice output progress bar bug when no config folder exist #…

62e14f7

…1370

hitenkoku requested a review from fukusuket July 25, 2024 10:54

refactor: fixed clippy error #1370

4f2f8ba

fukusuket approved these changes Jul 25, 2024

View reviewed changes

hitenkoku added 2 commits July 27, 2024 18:42

fix(profile): fixed no exist error when config folder was no exist #1386

0386909

feat(main): added message in no config folder of set-default-profile #…

0166429

…1386

YamatoSecurity added 2 commits July 28, 2024 14:46

update msg

80ded0a

update changelog

fa3e75a

YamatoSecurity and others added 3 commits July 28, 2024 16:16

Merge branch 'main' into 1370-embed-config-files-in-binary

1398f49

fix(afterfact): fixed not changed output color #1370

831b068

fix(main): removed failed message on successed set-default-profile co…

b49867f

…mmand #1370

hitenkoku added 3 commits August 12, 2024 22:44

UI(profile): fixed set default profile update message output position #…

562c117

…1370

chore: applied point out #1385 (comment)

c12a06a

fix(profile): fixed error in test #1370

f1b3aac

hitenkoku added 2 commits August 13, 2024 00:00

style(profile): fixed clippy warning #1370

83fbf8e

fix(afterfact): fixed level_color.txt load #1370

dbd0417

YamatoSecurity approved these changes Aug 14, 2024

View reviewed changes

YamatoSecurity merged commit 0ff1d5a into main Aug 14, 2024
5 checks passed

YamatoSecurity deleted the 1370-embed-config-files-in-binary branch August 14, 2024 01:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: embedded config files to binary #1370 #1385

feat: embedded config files to binary #1370 #1385

hitenkoku commented Jul 15, 2024

fukusuket left a comment •

edited

Loading

hitenkoku commented Jul 17, 2024

hitenkoku commented Jul 25, 2024

fukusuket left a comment

YamatoSecurity commented Jul 26, 2024

hitenkoku commented Jul 27, 2024

YamatoSecurity commented Jul 28, 2024

hitenkoku commented Jul 29, 2024

YamatoSecurity commented Jul 29, 2024

YamatoSecurity commented Jul 29, 2024

hitenkoku commented Aug 12, 2024

YamatoSecurity left a comment

feat: embedded config files to binary #1370 #1385

feat: embedded config files to binary #1370 #1385

Conversation

hitenkoku commented Jul 15, 2024

What Changed

fukusuket left a comment • edited Loading

Choose a reason for hiding this comment

hitenkoku commented Jul 17, 2024

hitenkoku commented Jul 25, 2024

fukusuket left a comment

Choose a reason for hiding this comment

YamatoSecurity commented Jul 26, 2024

hitenkoku commented Jul 27, 2024

YamatoSecurity commented Jul 28, 2024

hitenkoku commented Jul 29, 2024

YamatoSecurity commented Jul 29, 2024

YamatoSecurity commented Jul 29, 2024

hitenkoku commented Aug 12, 2024

YamatoSecurity left a comment

Choose a reason for hiding this comment

fukusuket left a comment •

edited

Loading