Merge pull request #1180 from Yamato-Security/finalize-2.9.0

finalize 2.9.0

CHANGELOG-Japanese.md

@@ -1,49 +1,29 @@
 # 変更点
 
-## 2.10.0 [2023/XX/XX] "xxx Release"
-
-**新機能:**
-
-- XXX
-
-**改善:**
-
-- XXX
-
-**バグ修正:**
-
-- XXX
-
-**その他:**
-
-- CIを通すためにtimeクレートを利用している更新されていないhhmmssクレートを除外した。 (#1181) (@hitenkoku)
-
-## 2.9.0 [2023/XX/XX] "xxx Release"
-
-**新機能:**
-
-- XXX
+## 2.9.0 [2023/09/22] "Autumn Rain Release"
 
 **改善:**
 
 - ディレクトリパスの指定にバックスラッシュを使用すべきではないことを示すエラーメッセージを追加した。 (#1166) (@hitenkoku, 提案者: @joswr1ght)
 - 一度に読み込むレコード数の最適化。(#1175) (@yamatosecurity)
 - プログレスバー内にあるバックスラッシュの表示をスラッシュに変更した。 (#1172) (@hitenkoku)
+- JSON形式で出力する際に、`count`ルールの`Details`フィールドを文字列にし、パースしやすくした。(#1179) (@hitenkoku)
+- デフォルトのスレッド数をCPU数から、プログラムが使用すべきデフォルトの並列度の推定値(`std::thread::available_parallelism`)に変更した。(#1182) (@hitenkoku)
 
 **バグ修正:**
 
-- XXX
+- まれにJSONフィールドが正しくパースされない状態を修正した。(#1145) (@hitenkoku)
 
 **その他:**
 
-- XXX
+- CIを通すために`time`クレートを利用している更新されていない`hhmmss`クレートを除外した。 (#1181) (@hitenkoku)
 
 ## 2.8.0 [2023/09/01] "Double X Release"
 
 **新機能:**
 
 - フィールドマッピング設定に16進数値を10進数に変換する`HexToDecimal`機能に対応した。 (元の16進数のプロセスIDを変換するのに便利。) (#1133) (@fukusuket)
-- `csv-timeline`と`json-timeline`に`-x, --recover-records`オプションを追加し、空ページのファイルカービングによってevtxレコードを復元できるようにした。(#952) (@hitenkoku) (Evtxカービング機能は@forensicmattに実装された。)
+- `csv-timeline`と`json-timeline`に`-x, --recover-records`オプションを追加し、evtxのスラックスペースのファイルカービングによってevtxレコードを復元できるようにした。(#952) (@hitenkoku) (Evtxカービング機能は@forensicmattに実装された。)
 - `csv-timeline`と`json-timeline`に`-X, --remove-duplicate-detections`オプションを追加した。(`-x`を使用する場合、重複データのあるバックアップログを含める場合などに便利。) (#1157) (@fukusuket)
 - `csv-timeline`、`json-timeline`、`logon-summary`、`eid-metrics`、`pivot-keywords-list`、`search`コマンドに、直近のイベントだけをスキャンするための`--timeline-offset`オプションを追加した。 (#1159) (@hitenkoku)
 - `search`コマンドに`-a, --and-logic`オプションを追加し、複数のキーワードをAND条件で検索できるようにした。 (#1162) (@hitenkoku)

CHANGELOG.md

@@ -1,57 +1,29 @@
 # Changes
 
-## 2.10.0 [2023/XX/XX] "xxx Release"
-
-**New Features:**
-
-- XXX
-
-**New Features:**
-
-- XXX
-
-**Enhancements:**
-
-- XXX
-
-**Bug Fixes:**
-
-- XXX
-
-**Other:**
-
-- Excluded unupdated hhmmss crates that use time crates to pass CI. (#1181) (@hitenkoku)
-
-## 2.9.0 [2023/XX/XX] "xxx Release"
-
-**New Features:**
-
-- XXX
-
-**New Features:**
-
-- XXX
+## 2.9.0 [2023/09/22] "Autumn Rain Release"
 
 **Enhancements:**
 
 - Added an error message to indicate that when you can't load evtx files in Windows due to specifying a directory path with spaces in it, you need to remove the trailing backslash. (#1166) (@hitenkoku, thanks for the suggestion from @joswr1ght)
-- Optimized the number of records to load at a time. (#1175) (@yamatosecurity)
-- Replaced double backslashes in paths in under the progress bar on Windows systems with single forward slashes. (#1172) (@hitenkoku)
+- Optimized the number of records to load at a time for performance. (#1175) (@yamatosecurity)
+- Replaced double backslashes in paths under the progress bar on Windows systems with single forward slashes. (#1172) (@hitenkoku)
+- Made the `Details` field for `count` rules a string in the JSON output for easier parsing. (#1179) (@hitenkoku)
+- Changed the default number of threads from number of CPUs to the estimate of the default amount of parallelism a program should use (`std::thread::available_parallelism`). (#1182) (@hitenkoku)
 
 **Bug Fixes:**
 
-- XXX
+- Fixed JSON fields would not be correctly parsed in rare cases. (#1145) (@hitenkoku)
 
 **Other:**
 
-- XXX
+- Removed the unmaintained `hhmmss` crate that uses an old `time` crate in order to pass the code coverage CI checks. (#1181) (@hitenkoku)
 
 ## 2.8.0 [2023/09/01] "Double X Release"
 
 **New Features:**
 
 - Added support for `HexToDecimal` in the field mapping configuration files to convert hex values to decimal. (Useful for converting the original process IDs from hex to decimal.) (#1133) (@fukusuket)
-- Added `-x, --recover-records` option to `csv-timeline` and `json-timeline` to recover evtx records through file carving on empty pages. (#952) (@hitenkoku) (Evtx carving feature is thanks to @forensicmatt)
+- Added `-x, --recover-records` option to `csv-timeline` and `json-timeline` to recover evtx records through file carving in evtx slack space. (#952) (@hitenkoku) (Evtx carving feature is thanks to @forensicmatt)
 - Added `-X, --remove-duplicate-detections` option to `csv-timeline` and `json-timeline` to not output any duplicate detection entries. (Useful when you use `-x`, include backup logs or logs extracted from VSS with duplicate data, etc...)
 - Added a `--timeline-offset` option to `csv-timeline`, `json-timeline`, `logon-summary`, `eid-metrics`, `pivot-keywords-list` and `search` commands to scan just recent events based on a offset of years, months, days, hours, etc... (#1159) (@hitenkoku)
 - Added a `-a, --and-logic` option in the `search` command to search keywords with AND logic. (#1162) (@hitenkoku)

Cargo.toml

@@ -1,17 +1,17 @@
 [package]
 name = "hayabusa"
-version = "2.9.0-dev"
+version = "2.9.0"
 repository = "https://github.com/Yamato-Security/hayabusa"
 authors = ["Yamato Security @SecurityYamato"]
 edition = "2021"
-rust-version = "1.71.0"
+rust-version = "1.72.1"
 include = ["src/**/*", "LICENSE.txt", "README.md", "CHANGELOG.md"]
 
 [dependencies]
 itertools = "*"
 dashmap = "*"
 clap = { version = "4.*", features = ["derive", "cargo", "color"]}
-evtx = { git = "https://github.com/Yamato-Security/hayabusa-evtx.git" , features = ["fast-alloc"] , rev = "c8391f1" } # 0.8.7 2023/08/30 update
+evtx = { git = "https://github.com/Yamato-Security/hayabusa-evtx.git" , features = ["fast-alloc"] , rev = "d7af285" } # 0.8.7 2023/09/22 update
 quick-xml = {version = "0.*", features = ["serialize"] }
 serde = { version = "1.*", features = ["derive"] }
 serde_json = { version = "1.0"}

README.md

@@ -248,7 +248,7 @@ You can learn how to analyze JSON-formatted results with `jq` [here](doc/Analysi
 * Log enrichment by adding GeoIP (ASN, city, country) information to IP addresses.
 * Search all events for keywords or regular expressions.
 * Field data mapping. (Ex: `0xc0000234` -> `ACCOUNT LOCKED`)
-* Evtx record carving from empty space.
+* Evtx record carving from evtx slack space.
 * Event de-duplication when outputting. (Useful when recovery records is enabled or when you include backed up evtx files, evtx files from VSS, etc...)
 
 # Downloads
@@ -475,7 +475,7 @@ Input:
   -f, --file <FILE>      File path to one .evtx file
   -l, --live-analysis    Analyze the local C:\Windows\System32\winevt\Logs folder
   -J, --JSON-input       Scan JSON formatted logs instead of .evtx (.json or .jsonl)
-  -x, --recover-records  Carve evtx records from empty pages (default: disabled)
+  -x, --recover-records  Carve evtx records from slack space (default: disabled)
 
 Filtering:
       --timeline-offset <OFFSET>  Scan recent events based on an offset (ex: 1y, 3M, 30d, 24h, 30m)
@@ -514,7 +514,7 @@ Input:
   -f, --file <FILE>      File path to one .evtx file
   -l, --live-analysis    Analyze the local C:\Windows\System32\winevt\Logs folder
   -J, --JSON-input       Scan JSON formatted logs instead of .evtx (.json or .jsonl)
-  -x, --recover-records  Carve evtx records from empty pages (default: disabled)
+  -x, --recover-records  Carve evtx records from slack space (default: disabled)
 
 Filtering:
       --exclude-computer <COMPUTER...>  Do not scan specified computer names (ex: ComputerA) (ex: ComputerA,ComputerB)
@@ -579,7 +579,7 @@ Input:
   -f, --file <FILE>      File path to one .evtx file
   -l, --live-analysis    Analyze the local C:\Windows\System32\winevt\Logs folder
   -J, --JSON-input       Scan JSON formatted logs instead of .evtx (.json or .jsonl)
-  -x, --recover-records  Carve evtx records from empty pages (default: disabled)
+  -x, --recover-records  Carve evtx records from slack space (default: disabled)
 
 Filtering:
       --exclude-computer <COMPUTER...>  Do not scan specified computer names (ex: ComputerA) (ex: ComputerA,ComputerB)
@@ -635,7 +635,7 @@ Input:
   -f, --file <FILE>      File path to one .evtx file
   -l, --live-analysis    Analyze the local C:\Windows\System32\winevt\Logs folder
   -J, --JSON-input       Scan JSON formatted logs instead of .evtx (.json or .jsonl)
-  -x, --recover-records  Carve evtx records from empty pages (default: disabled)
+  -x, --recover-records  Carve evtx records from slack space (default: disabled)
 
 Filtering:
   -E, --EID-filter                      Scan only common EIDs for faster speed (./rules/config/target_event_IDs.txt)
@@ -705,7 +705,7 @@ Input:
   -d, --directory <DIR>  Directory of multiple .evtx files
   -f, --file <FILE>      File path to one .evtx file
   -l, --live-analysis    Analyze the local C:\Windows\System32\winevt\Logs folder
-  -x, --recover-records  Carve evtx records from empty pages (default: disabled)
+  -x, --recover-records  Carve evtx records from slack space (default: disabled)
 
 Filtering:
   -a, --and-logic                 Search keywords with AND logic (default: OR)
@@ -793,7 +793,7 @@ Input:
   -f, --file <FILE>      File path to one .evtx file
   -l, --live-analysis    Analyze the local C:\Windows\System32\winevt\Logs folder
   -J, --JSON-input       Scan JSON formatted logs instead of .evtx (.json or .jsonl)
-  -x, --recover-records  Carve evtx records from empty pages (default: disabled)
+  -x, --recover-records  Carve evtx records from slack space (default: disabled)
 
 Filtering:
   -E, --EID-filter                      Scan only common EIDs for faster speed (./rules/config/target_event_IDs.txt)
@@ -1051,7 +1051,7 @@ Input:
   -f, --file <FILE>      File path to one .evtx file
   -l, --live-analysis    Analyze the local C:\Windows\System32\winevt\Logs folder
   -J, --JSON-input       Scan JSON formatted logs instead of .evtx (.json or .jsonl)
-  -x, --recover-records  Carve evtx records from empty pages (default: disabled)
+  -x, --recover-records  Carve evtx records from slack space (default: disabled)
 
 Filtering:
   -E, --EID-filter                      Scan only common EIDs for faster speed (./rules/config/target_event_IDs.txt)