[Discussions] DaC Community Feedback

[Mikaayenson]

Summary

From the beginning of the Elastic detection-rules repo, it not only contained the Elastic prebuilt detection rules files, but also additional tooling for detection rule management like a suite of tests, CLI commands, and automation scripts used by the Elastic Threat Research and Detection Engineering (TRADE) team.

Elastic TRADE team has been following Detections as Code (DaC) practices for years, supporting development and release processes for Elastic prebuilt detection and endpoint rules.

With DaC becoming more mainstream and continuing with our commitment to openness, we are working on making it easier for users to kick-start their own DaC process using Elastic detection-rules repo for their rules management. This DaC expansion will build upon prior detection rules features to provide an end-to-end experience for detection engineers.

As we continue planning and organizing upcoming detections as code sprints, we've started capturing feedback from the community. This issue is designed to organize any subsequent feedback in a common place. Your feedback is important as it will help us prioritize and shape how we refactor our detection-rules DaC experience.

What to Expect

We will be posting updates about work happening on the DaC topic in this issue, so make sure to subscribe to get those updates, try out the changes, and let us know what you think.

How to Contribute

1. 👋 If you would like to chime in, please feel free to add a comment below!
2. Check ⬇️ the existing issues and 👍 if they address your need.
3. If you feel anything is missing, feel free to open an issue and let us know so that we can track appropriately!

Existing Related Issues

• [FR] Make rules/ directory configurable #1342
• [FR] option to users to be able to edit detection-rules #1917
• [FR] Run the unit tests on a specific folder/files #3129
• [FR] Manage Exceptions in an as-code way #1738
• [FR] Better way to work with non-ecs fields #3262
• [Bug] CLI Command import-rules Prompts Threshold Related Values to be Submitted #3266
• [FR] git synchronize #362
• [FR] preserver rule_id in kibana_upload #612

[leandrojmp]

Just some feedback as we tried to use the detection rules script to automate the management of our custom rules, but ended up writing our own tool because it didn't work for our use cases.

The main issue was that we do not use index patterns for our data, we use data views which is the recommended way now, and since the index pattern API is deprecated we assumed that using index patterns is also depracted.

But the detection-rules script does not work with data views, it uses index patterns only.

Another big issue is that we have a lot non-ecs fields, which also does not work with detection-rules,

So because of those two main issues, we needed to write a similar tool to automate the deploy of our custom rules.

[brokensound77]

Thank you @leandrojmp - this is great feedback. We will be updating the rule schema to allow data_view_id shortly, which allows use of index or data views in all rules.

As far as the non-ecs, rules are allowed to used any field values, we just validate stricter for our built in rules based on the corresponding datasource schema. This is what you are likely referring to. We are also in the process of decoupling the validation and schemas of rules to make this simpler.

Today, you can get around this by maintaining the non-ecs-schema, however, with the upcoming changes, user rules will be completely decoupled, allowing more granular control of what validations occur and what schemas are used (in case you want to bring your own)

[brokensound77]

here is the PR to support data views: #3510

[brokensound77]

👋 just dropping in to provide an update on progress to supporting DAC more seamlessly.

• Refresh Kibana module with API updates #3466
• [FR] Refactor to more seamlessly support multiple DAC approaches #3407
• https://dac-reference.readthedocs.io/en/latest/dac_concept_and_workflows.html
• https://br0k3nlab.com/presentations/files/dac-bsidesok-2024.pdf

Updates

In #3466, the kibana module was updated to use the newer APIs for rule management. There were also 2 commands added for simplicity. This merged to main

In #3407, there was a lot introduced, including decoupling significant components of the repo for easier adoptability. This was merged to a feature branch (DAC-feature). We consider this early alpha and ready for users to start testing.

We realize a significant outcome of this is simply conveying our findings and best practices for implementing DAC on these rules, particularly with the built-in components of this repo. So, we started work on the DAC reference. As with any as code, there is tons of variation on viable approaches - this reference aims to enable users to implement DAC with, without, or as a hybrid partially using the repo.

@Mikaayenson and I also recently gave a talk at BSidesOK about leveraging this, where we released the reference. Here are the slides.

Next steps

As stated, we consider this in early alpha and are open to feedback from early adopters. Additionally, we have other smaller snippets of planned work upcoming to further simplify this.

• [FR][DAC] add support for custom-schemas (BYOS) #3618
• [FR][DAC] further decouple reliance on default rule dir locations #3619
• [FR][DAC] Add *LIMITED* support for version and revision to BaseRuleData #3620
• [FR][DAC] raise a better exception for missing contents in config files #3621
• [FR] normalize use of yaml extension (vs yml) #3622
• [FR][DAC] Add validation on exceptions values #3623
• [FR][DAC] Consideration: expose kql parse parameters for custom rules validation #3624
• [FR][DAC] Consideration: expose a callback function within kibana export-rules to organize the output #3625
• [FR][DAC] Consideration: DAC related CI/CD (GH actions) for syncing with Kibana #3626

If you have any feedback or questions along the way, the best place to reach us is in the public slack channel for security-rules-dac

[ar3diu]

I think dac-reference is the most detailed documentation on this subject that I have seen so far, which is great because for many teams the concept and implementation methods were not always clear. Thanks for putting this together!

[brokensound77]

I think dac-reference is the most detailed documentation on this subject that I have seen so far, which is great because for many teams the concept and implementation methods were not always clear. Thanks for putting this together!

Thanks for the awesome feedback! That was our hope all along, as we realized through this process that just understanding is likely more valuable than any code support we could add.

Keep an eye on it as we will continue to expand on it as we build this out