[FR] Add Documentation on the available custom rules config options

docs/core_component_syncing_rules_and_data_from_elastic_security_to_vcs.md

@@ -84,7 +84,7 @@ jobs:
 
     - name: Export Rules from Elastic Security and Import Rules to TOML
       run: |
-        python -m detection_rules kibana --space "${{ github.event.inputs.space }}" export-rules --directory ${{ env.CUSTOM_RULES_DIR }}/rules/
+        python -m detection_rules kibana --space "${{ github.event.inputs.space }}" export-rules --directory ${{ env.CUSTOM_RULES_DIR }}/rules/ -e -ac -s
       env:
         DR_CLOUD_ID: ${{ secrets.ELASTIC_CLOUD_ID }}
         DR_KIBANA_USER: ${{ secrets.ELASTIC_USERNAME }}
@@ -163,3 +163,47 @@ Setting up a CI/CD workflow that periodically triggers, such as through a cron j
 1. Establish a cron job within your CI/CD platform to periodically trigger rule exports.
 1. Automate the extraction and formatting of rules from Elastic Security using the Detection Engine API.
 1. Commit the updated rules to VCS and optionally open PRs for team review.
+
+
+## Sub-Component 3 (optional): Unit Testing Rules via CI/CD
+
+This is an optional step you may want to configure to setup custom unit testing that runs on your PRs. This is optional and just an example means of accomplishing this.
+
+```yaml
+name: Custom Unit Tests
+
+on:
+  push:
+    branches: [ "main", "7.*", "8.*" ]
+  pull_request:
+    branches: [ "*" ]
+
+jobs:
+  tests:
+
+    runs-on: ubuntu-latest
+    env:
+      CUSTOM_RULES_DIR: ${{ secrets.CUSTOM_RULES_DIR }}
+
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Set up Python 3.12
+      uses: actions/setup-python@v5
+      with:
+        python-version: '3.12'
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip cache purge
+        pip install .[dev]
+
+    - name: Python Lint
+      run: |
+        python -m flake8 tests detection_rules --ignore D203,N815 --max-line-length 120
+
+    - name: Unit tests
+      run: |
+        python -m detection_rules test
+```

docs/core_component_syncing_rules_and_data_from_vcs_to_elastic_security.md

@@ -69,7 +69,7 @@ Currently supported arguments:
 3. Once authenticated, you can use [API calls](https://www.elastic.co/guide/en/security/current/security-apis.html#_api_calls) directly. If using the detection-rules CLI, the following DAC commands are available. Deploy the latest set of rules by running the following command:
 
 ```bash
-(detection-rules-build) ➜  detection-rules git:(DAC-feature) ✗ python -m detection_rules -h
+(detection-rules-build) ➜  detection-rules git:(main) ✗ python -m detection_rules -h
 Loaded config file: /Users/stryker/workspace/ElasticGitHub/detection-rules/.detection-rules-cfg.json
 
 █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
@@ -605,7 +605,7 @@ jobs:
         git fetch origin ${{ github.base_ref }}
         git diff --name-only origin/${{ github.base_ref }} '*.toml' | while read rule_file; do
           echo "Importing rule from file $rule_file"
-          python -m detection_rules kibana --space "dev" import-rules --rule-file "$rule_file"  --overwrite
+          python -m detection_rules kibana --space "dev" import-rules --rule-file "$rule_file"  --overwrite -e -ac
         done
       env:
         DR_CLOUD_ID: ${{ secrets.ELASTIC_CLOUD_ID }}

docs/etoe_reference_example.md

@@ -5,7 +5,7 @@
 
 Check out the [Quick Start Slides](./_static/DaC_Rolling_your_own_Detections_as_Code.pdf) to an overview of DaC and how to get started. It provides some example CLI commands using the detection-rules repo.
 
-⚠️ Note: The alpha detection-rules [branch](https://github.com/elastic/detection-rules/tree/DAC-feature), content within these slides, and this reference guide are subject to change. Once we finally migrate the changes to the `main` branch, we will update the content accordingly.
+⚠️ Note: The Detection Rules DaC feature is still in Beta. This phase includes continued testing to include edge cases and feedback from the community. This reference guide and associated content are subject to change. 
 
 If you’re starting from scratch and would like to get quickly started, here are the high-level instructions. These steps assume you are familiar with the considerations associated with the various steps and elect to follow the detection-rules VCS approach to manage rules. It also assumes advanced configurations are not applied.
 
@@ -18,6 +18,79 @@ Take a look at an example of how you can use some of our DaC features. The repo
     <iframe src="https://drive.google.com/file/d/1XMPSdgjZipa94xufv_4byVrMm-0XaKZh/preview" width="640" height="480" allow="autoplay"></iframe>
 </div>
 
+### Quick Start Example Detection Rules CLI Commands
+
+**Steps:**
+
+1. Clone the detection rules repo and install python requirements, see [getting started](https://github.com/elastic/detection-rules?tab=readme-ov-file#getting-started).
+
+1. Initialize a custom rules directory via the setup config command. 
+
+    `python -m detection_rules custom-rules setup-config dac_custom_rules_dir`
+
+1. Edit the `_config` file in your `dac_custom_rules_dir` directory. 
+
+    Most users will want to add these additional parameters for ease of use:
+
+    ```yaml
+    bypass_version_lock: True
+    normalize_kql_keywords: True
+    auto_gen_schema_file: "etc/schemas/auto_gen.json"
+    bypass_optional_elastic_validation: True
+    ```
+
+1. Edit the `test_config.yaml` in your `dac_custom_rules_dir/etc` directory. 
+
+    For unit testing, most users will also want to bypass the following Elastic unit tests.
+
+    ```yaml
+    unit_tests:
+        bypass:
+        - tests.test_gh_workflows.TestWorkflows.test_matrix_to_lock_version_defaults
+        - tests.test_schemas.TestVersionLockSchema.test_version_lock_has_nested_previous
+        - tests.test_packages.TestRegistryPackage.test_registry_package_config
+        - tests.test_all_rules.TestValidRules.test_schema_and_dupes
+        - tests.test_all_rules.TestRuleMetadata.test_invalid_queries
+        - tests.test_all_rules.TestValidRules.test_bbr_validation
+        - tests.test_all_rules.TestValidRules.test_rule_type_changes
+        - tests.test_schemas.TestSchemas.test_eql_validation
+    ```
+
+1. Set your environment variable to use the custom configuration and rules directory you just made.
+
+    `export CUSTOM_RULES_DIR="<full_path_to_dac_custom_rules_dir>"`
+
+1. If running locally, set your ``.detection-rules-cfg.json` in the root of the detection rules directory. See [setup a config file](https://github.com/elastic/detection-rules/blob/main/CLI.md#setup-a-config-file) for more details.
+
+    ```json
+    {
+        "cloud_id": "example:example",
+        "api_key": "example",
+    }
+    ```
+
+1. Export custom rules and related exceptions to a Kibana Instance, overwriting existing, stripping version, and skipping errors.
+
+    `python -m detection_rules kibana export-rules -s -sv -e -ac`
+
+1. Run local unit tests.
+
+   `make test`
+
+1. Import custom rules and related exceptions to a Kibana Instance, overwriting existing.
+
+    `python -m detection_rules kibana import-rules --overwrite -e -ac`
+
+1. Or if you prefer to import and export using ndjson file(s) instead of the Kibana API use the following:
+
+    For moving rules from a Kibana rules export to the repo
+    `python -m detection_rules import-rules-to-repo <ndjson_file> --required-only -e -da DefaultAuthor -ske -ac`
+
+    For moving rules from the repo to an ndjson file that is compatible with Kibana rule import.
+    ` python -m detection_rules export-rules-from-repo -ac -e`
+
+For more information on these CLI commands please see CLI.md and docs/custom-rules.md in Detection Rules.
+
 ### Locally
 
 **Steps:**

docs/faq.md

@@ -3,31 +3,155 @@
 Here are some frequently asked questions about Detections as Code (DaC). There are known [limitations](./known_limitations.md), so please review if you do not see your question answered here. More questions and answers will be added over time to address common queries and concerns.
 
 **Q1:** What is Detections as Code (DaC)?
+
 **A1**: Detections as Code is a methodology that applies software development practices to the creation, management, and deployment of security detection rules. By treating detection rules as code, it enables version control, automated testing, and deployment processes, enhancing collaboration, consistency, and agility in response to threats.
 
 **Q2**: How does DaC benefit my security team?
+
 **A2**: DaC streamlines the detection rule lifecycle, ensuring high-quality detections through peer reviews and automated tests. It facilitates a scalable approach to managing an ever-growing number of detection rules and adapting quickly to the evolving threat landscape. This methodology supports compliance with change management requirements and fosters a mature security posture.
 
 **Q3**: Can I integrate DaC with my existing CI/CD pipelines?
+
 **A3**: Yes, DaC is designed to integrate seamlessly with existing CI/CD pipelines. By leveraging the detection-rules repository and its tooling within your CI/CD workflows, you can automate the testing, validation, and deployment of detection rules directly into Elastic Security or other SIEM platforms.
 
 **Q4**: What are the prerequisites for adopting DaC with Elastic?
+
 **A4**: To adopt DaC, you'll need access to the Elastic detection-rules repository, a version control system (like GitHub), and familiarity with CI/CD principles and tools. Basic knowledge of Elastic Security and its API is also beneficial for customizing and deploying rules.
 
 **Q5**: How do I start using DaC in my organization?
+
 **A5**: Begin by exploring the detection-rules repository and familiarizing yourself with its structure and tooling. Consider your governance model (VCS as authoritative, Elastic Security as authoritative, or dual sync) and plan your workflow accordingly. Experiment with the alpha branch to test new features and configurations that suit your needs.
 
 **Q6**: Are there any resources available for learning more about DaC?
+
 **A6**: Elastic provides comprehensive documentation, reference guides, and slide decks to help you understand and implement DaC. Joining the Elastic community forums and Slack channels, such as #security-rules-dac, can also provide valuable insights and support from fellow users and Elastic experts.
 
 **Q7**: How can I contribute to the DaC methodology or detection-rules repo?
+
 **A7**: Contributions are welcome! You can contribute by testing the alpha features, reporting issues, suggesting enhancements, or sharing your own use cases and scripts. Contributions help improve DaC for everyone and are a great way to give back to the community.
 
 **Q8**: How should exception & action files be deployed to Kibana?
+
 **A8**: Currently, there two approaches.
 
 1. You could directly add the correct exception or action item in the toml files themselves. We have two fields available: `actions: Optional[list]` and `exceptions_list: Optional[list]`. This approach is a bit difficult because we do not define the entire schema for these fields that are expected by Kibana. It's not really a feature we use within our prebuilt ruleset, so its left open to the user to properly supply the correct fields and format.
 2. The second approach to define the actions and /or exceptions directories in the `_config.yaml`. This will load action list toml files or exception list toml files defined in their folders. These toml files will have the `rule id`  mapping so you know which list are associated to the rule. When you upload rules to kibana, it will automatically bring the defined exception/action lists.
 
 Note this is one way at the moment (toml --> kibana). So you can not export lists and import them into toml files.
 
+
+**Q9**: Is there a way to run the unit tests only on `custom_rules_dir`?
+
+**A9**: Yes, there are a few options you have when it comes to doing this. For more reference material, see our [custom-rules.md](https://github.com/elastic/detection-rules/blob/main/docs/custom-rules.md).
+
+If you already have your custom rules directory for reference and appropriate config files setup you can skip this step.
+First, generate a base set of configs and custom rules directory using our helper command
+`python -m detection_rules custom-rules setup-config <new_directory_name>`
+
+Set the environment variable `CUSTOM_RULES_DIR` to the path of you custom rules directory. At this point the rule loading will pull from the rules folders specified in the `<new_directory_name>_config.yaml`.
+
+The default behavior now is for the unit tests to be run on only the rules directories specified in that config file.
+You can further customize the unit tests configuration (ignoring specific tests, etc.) in the `<new_directory_name>/etc/test_config.yaml`
+
+
+**Q10**: I want to use `export-rules-from-repo`in CI/CD to convert a new/modified rule from TOML to JSON and push the JSON to Kibana API. What is the best way to do that?
+
+**A10**: You may be able to use `import-rules` to simplify this process. 
+
+The `export-rules-from-repo` command is not the required method for uploading rules to Kibana, just another option for those who may wish to use ndjson.
+For instance, in this setup I have a local directory specific in my `CUSTOM_RULES_DIR` config. 
+
+If I want to modify it and upload it to Kibana, I can directly use the `import-rules` command with the `--overwrite` flag to trigger the update from TOML directly into Kibana.
+Command: `python -m detection_rules kibana import-rules --overwrite`. You can also specify a directory or specific file here too in addition to the config.
+
+ <img src="_static/rule_update_example.gif"  alt="Import Rules --overwrite Example" id="figure4"/>
+
+
+**Q11**: When using custom schemas, do I have to declare all fields or only those that are not part of the ecs?
+
+**A11**: You do not have to declare all the fields, just the ones not part of the ecs.
+
+Custom Schema fields are treated as additional non-ecs fields, in effect this means your custom schema(s) values are added as additional available options that should be considered as valid. Here is an example that might be useful.
+
+I want to add 2 new keyword fields to process that I want to be considered as valid for a specific index. First I need to modify my `stack-schema-map.yaml` to specify a custom schema file I want to add as additional fields to validate against my target stack version (in this case 8.14.0).
+
+```
+8.14.0:
+  beats: 8.12.2
+  ecs: 8.11.0
+  endgame: 8.4.0
+  custom: schemas/custom-schema.json
+```
+
+Now I need to populate `schemas/custom-schema.json` with my desired keywords for my index. Here is an example of what this would look like:
+
+```
+{
+    "custom-index*": {
+      "process.NewValue": "keyword",
+      "process.AnotherValue": "keyword"
+    }
+}
+```
+
+With this custom schema defined I can now make an example rule that I would like to validate. The quickest way to do this is to create the toml rule file and run the `view-rule` command which parse the rule file and validate it against the specified schemas (including custom schemas specified in your config)
+
+Here is an example rule using the above schema:
+```
+[metadata]
+creation_date = "2023/08/23"
+integration = ["endpoint"]
+maturity = "production"
+min_stack_comments = "New fields added: required_fields, related_integrations, setup"
+min_stack_version = "8.3.0"
+updated_date = "2024/03/08"
+
+[rule]
+author = ["Elastic"]
+description = """
+Test Rule
+"""
+from = "now-9m"
+index = ["logs-endpoint.events.*", "custom-index*"]
+language = "eql"
+license = "Elastic License v2"
+name = "TEST Potential Protocol Tunneling via Chisel Client"
+note = """Test Note"""
+references = [
+    "https://blog.bitsadmin.com/living-off-the-foreign-land-windows-as-offensive-platform",
+    "https://book.hacktricks.xyz/generic-methodologies-and-resources/tunneling-and-port-forwarding"
+    ]
+risk_score = 47
+rule_id = "e8e3af2a-11b8-4ab7-9ca1-c6db621ea89d"
+setup = """Test Setup"""
+severity = "medium"
+tags = [
+        "Domain: Endpoint",
+        "OS: Linux",
+        "Use Case: Threat Detection",
+        "Tactic: Command and Control",
+        "Data Source: Elastic Defend"
+        ]
+type = "eql"
+query = '''
+process where host.os.type == "linux" and process.NewValue == "GoodValue"
+'''
+timestamp_override = "event.ingested"
+
+[[rule.threat]]
+framework = "MITRE ATT&CK"
+
+[[rule.threat.technique]]
+id = "T1572"
+name = "Protocol Tunneling"
+reference = "https://attack.mitre.org/techniques/T1572/"
+
+[rule.threat.tactic]
+id = "TA0011"
+name = "Command and Control"
+reference = "https://attack.mitre.org/tactics/TA0011/"
+```
+
+Running `view-rule` on this rule should look similar to the following. When the custom schema is specified, the schema validation should pass and the contents should be printed. If the custom schema is removed, the validation should fail and an error message should appear.
+
+ <img src="_static/byos_test.gif"  alt="View Rule Custom Schema" id="figure5"/>

docs/index.md

@@ -6,7 +6,7 @@
 <span style="font-size:em;">_______________</span><br>
 <span style="font-size:1.5em;">Managing Elastic Security Security Detection Rules Using DaC</span><br><br><br>
 
-⚠️ Note: The alpha detection-rules [DAC-feature](https://github.com/elastic/detection-rules/tree/DAC-feature) branch, content within these [slides](./_static/DaC_Rolling_your_own_Detections_as_Code.pdf), and this reference guide are subject to change. Once we finally migrate the changes to the `main` branch, we will update the content accordingly.
+⚠️ Note: The Detection Rules DaC feature is still in Beta. This phase includes continued testing to include edge cases and feedback from the community. This reference guide, content within these [slides](./_static/DaC_Rolling_your_own_Detections_as_Code.pdf), and associated content are subject to change. 
 
 ```{toctree}
 :maxdepth: 2