Support encoded rules to avoid AV false positives #1369
Comments
|
Update: The encrypted |
|
@hitenkoku Please wait on this issue. I recently tried uncompressing the hayabusa rules without encryption and it took quite a long time so I think it will slow down the scan quite a bit if we used an encrypted zip. The only other solution I can think of is by merging all of the rules into a single file and doing a simple XOR or something on it to bypass anti-virus signatures. |
Prototype memoMerge YAMLs and XORuse std::fs::{self, File};
use std::io::{self, Read, Write};
use std::path::Path;
use yaml_rust::{YamlLoader, YamlEmitter};
fn list_yaml_files(dir: &Path) -> io::Result<Vec<String>> {
let mut yaml_files = Vec::new();
for entry in fs::read_dir(dir)? {
let entry = entry?;
let path = entry.path();
if path.is_dir() {
yaml_files.extend(list_yaml_files(&path)?);
} else if path.extension().and_then(|s| s.to_str()) == Some("yml") {
yaml_files.push(path.to_string_lossy().to_string());
}
}
Ok(yaml_files)
}
fn merge_yaml_files(files: Vec<String>) -> Result<String, Box<dyn std::error::Error>> {
let mut merged_yaml = Vec::new();
for file in files {
let mut content = String::new();
File::open(&file)?.read_to_string(&mut content)?;
let docs = YamlLoader::load_from_str(&content)?;
merged_yaml.extend(docs);
}
let mut out_str = String::new();
for (i, doc) in merged_yaml.iter().enumerate() {
if i > 0 {
out_str.push_str("\n"); // Add separator between documents
}
{
let mut emitter = YamlEmitter::new(&mut out_str);
emitter.dump(doc)?;
}
}
Ok(out_str)
}
fn xor_encrypt(data: &str, key: u8) -> Vec<u8> {
data.bytes().map(|b| b ^ key).collect()
}
fn xor_decrypt(data: &[u8], key: u8) -> Vec<u8> {
data.iter().map(|&b| b ^ key ).collect()
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
// encrypt
let dir = Path::new("/Users/fukusuke/Hayabusa/hayabusa-2.16.0-mac-arm/rules");
let yaml_files = list_yaml_files(dir)?;
let merged_yaml = merge_yaml_files(yaml_files)?;
let encrypted_yaml = xor_encrypt(&merged_yaml, 0xAA); // Example key: 0xAA
let mut output_file = File::create("encrypted_output.yml")?;
output_file.write_all(&encrypted_yaml)?;
// decrypt
let encrypted_file_path = Path::new("encrypted_output.yml");
let mut encrypted_file = File::open(encrypted_file_path)?;
let mut encrypted_content = Vec::new();
encrypted_file.read_to_end(&mut encrypted_content)?;
let decrypted_content = xor_decrypt(&encrypted_content, 0xAA); // Example key: 0xAA
println!("Decrypted YAML content:\n{}", decrypted_content);
Ok(())
}hayabusa sidehttps://github.com/Yamato-Security/hayabusa/blob/main/src/yaml.rs#L148 pub fn read_encrypted_file(path: &PathBuf) -> Result<String, String> {
let mut fr = fs::File::open(path)
.map(BufReader::new)
.map_err(|e| e.to_string())?;
let mut encrypted_content = Vec::new();
let _ = fr.read_to_end(&mut encrypted_content);
let decrypted_content = encrypted_content.iter().map(|&b| b ^ 0xAA ).collect(); // Example key: 0xAA
let decrypted_string = String::from_utf8(decrypted_content).expect("Invalid UTF-8 sequence");
Ok(decrypted_string)
}
...Test Perspective
Test Resultcurrent ruleencrypted and merged ruleencrypted rule's result
|
|
@YamatoSecurity @hitenkoku
Based on the above results, merge of all Yaml files into one file with XOR seems to be good method :) |
|
@fukusuket Oh great! thanks so much! |
|
I will implement it so that XOR files are also uploaded to https://github.com/Yamato-Security/hayabusa-encrypted-rules ! |
|
@fukusuket Thanks! I renamed the repository to https://github.com/Yamato-Security/hayabusa-encoded-rules since we are going to encode instead of encrypt them. Sorry for the trouble! |
|
I've implemented GitHub Actions for XOR file auto commit!💪 |
|
@fukusuket Super quick! Thanks so much! |
YamatoSecurity
changed the title
|
@fukusuket I checked in https://github.com/Yamato-Security/hayabusa-encoded-rules/blob/main/rules.zip occured error when xor processed bytes convert utf8 string. yml file is not xor converted in rules.zip. I am very sorry for taking so long and not being able to do this. Could you please confirm this data? |
|
@hitenkoku |
|
I understand! Here is the file to read!(The file name is encoded_rules, not rules.zip) |
|
@fukusuket Thanks for your quick reply. |
|
@hitenkoku Sorry the description in this issue is old and we changed our plans because it would be too slow to de-compress and decrypt the rules. Instead of compressing/decompressing and encryption/decryption, we are going to gather all of the YML rule information into one single file and XOR encode/decode it. So in the source code as well, please change the wording of |
|
@YamatoSecurity Thanks for your comment. The current version of hayabusa is based on the assumption that files are separated and rule counts are handled separately, so significant modifications are required. |
|
@hitenkoku @YamatoSecurity main: prototype: The difference between the 4 rules is that the encoded_rule includes the yaml that Windows Defender detects |
|
I apologize for the inconvenience. |
|
@hitenkoku If it is ok with you, can I assign @fukusuket to this since he already made this prototype so might be easier this way? |
|
@hitenkoku @YamatoSecurity |
|
@YamatoSecurity I think that many use cases will not use both If you want to use encoded_rules, just use the zip included with The internal logic of what do you think? |
|
@YamatoSecurity |
|
@YamatoSecurity
It could probably be tied to yml (only if there is a regular yml), but the implementation would probably be complex, so I would like to allow for the above constraints if possible🤔 what do you think? |
In order to prevent Windows defender from alerting on false positives on yml rules and to minimize the amount of files we need to save to the system, Hayabusa will have the option to use encoded rules from a single file hosted at https://github.com/Yamato-Security/hayabusa-encoded-rules
We originally planned to encrypt and zip the rules and config files in the hayabusa-rules repository but that would greatly affect performance so we are now planning on using a file contains all rules and rules config files in an XOR-encoded form.
The text was updated successfully, but these errors were encountered: