[META] - Security Analytics in OpenSearch (MVP)

[getsaurabh02]

Is your feature request related to a problem?
The current OpenSearch customers have no easy way to run security related rules on their data, out of the box, without having to clone their data into a dedicated security specific solution. Over time these OpenSearch customers pay a high cost to deploy and operate off the shelf security solutions, due to the data volume-based pricing plans from most vendors. Due to increasing volumes of data related to enterprise security, customers are facing increases in the costs of software licenses and infrastructure required for data processing. Increasing costs are harder to justify for many customers who are using only a subset of the product features.

Also, often times these customers need custom features for which they aren’t able to easily modify or adapt the existing products to their use case. This especially include tailoring custom rules which are more suited to their workload and data. Since missing features reflect the needs of the business in a specific industry, the vendor is driven by the size of that market and demand for that particular feature, sometimes resulting in a long wait for customers. In addition, customers need to employ significantly large teams of hard to find security experts who need to be trained to use multiple security products.

What solution would you like?
By offering OpenSearch customers Security Analytics solution in form of an extensible plugin, with pre-built functionality, will address the above problems. It will enable customers to use their existing OpenSearch cluster for detecting security threats in their business data. It will also allow them to modify/tailor the default solution and develop the necessary proprietary changes on top, while using the low level components and uniform interfaces offered by the plugin. Customers will not have to pay hefty licensing fees that grow with their data volumes, while also using the familiar OpenSearch software. As an open source platform with potential contributions from a broader community, this would enable customers to move fast and build their custom solutions quicker.

In the Security Analytics plugin, OpenSearch will add several advanced tools and capabilities for the security use-cases including IP reputation, provide in-built rules for the most common type of attacks, integration points with external threat intel feeds, threat correlation techniques for deeper insights, external storage archival. This will enable security analysts to go beyond the ad-hoc analysis and find potential threats in real time with minimal effort as well as expedite forensic analysis of incidents.

Do you have any additional context?
More detailed design covered as part of the RFC : #2

Breaking the changes further into more granular issues for P0 items as below

• 1. [META] Rule curation layer(P0) & Rule ingestion layer(P0) #5 External Rule Curation Layer (for Sigma) - Sigma Rule Transformers, Converter and Aggregator
• 2. Rule Ingestion Layer - APIs to create/update/delete rules and persist in form of Rule Index
• 3. Index Mapping for Rules - Field Aliases Creation and its Management
• 4. Rule Orchestrator Layer - Detector Lifecycle Management, Percolate Query Execution, Parallel Pipeline Execution, Rule Executiuon State Mgmt
• 5. Event Persistence for Findings - Findings index common schema, Findings Index Creation and Lifecycle Management
• 6. Event Alerting Layer - Alert Index common schema, Alert Index Creation, Trigger Condition Handling, Grouping, Alerts Workflow Management
• 7. Event Visualization Framework - Dashboard experience for Findings and Alerts
• 8. Security Analytics Dashboard - Admin Page, Event/Alert Discovery and Threat Document Discovery
• 9. Feature Testing - Functional and Integration
• 10. Performance - Load, Memory Allocation, CPU Profiling, Heap Dumps and Leaks Analysis

P1 Items

• Rule Editor Framework - Rule visualization experience, editor, validator and version support
• Event Correlation Framework - Background job forAggregation and Filtering, Event deduplication, normalization and Correlation Visualization.
• External Store Support - Data, Findings and Alert Index Archival, On Demand Hot Reload, On Demand Rule Execution and Correlation
• External Rules Adapters - Data Adapters for Importing Rules From Existing SIEM or security solution
• External Data Adapters - Data Connectors for Importing data and Comparison Visualization

[elfisher]

I'd love to understand what the generalizable components are versus things that only are relevant to security analytics.

[brijos]

@getsaurabh02 what all in the list is getting launched in 2.3?

[getsaurabh02]

I'd love to understand what the generalizable components are versus things that only are relevant to security analytics.

Thanks @elfisher and @brijos for reviewing the details here. Among the above listed the components under P0, the ones related to rule curation and ingestion layer (such as Item 1 and 2) will be more relevant to security analytics (SA) given the responsibilities of these layers.

Item 3 for Index Mapping will initially serve the SA use case, but later can also be evolved into a more generic component to be used in other places for index schema matching. This seems to be common problem for customers using several other features, such as Anomaly Detection (AD), where they would need custom mapping on top of their indices, along with the ability to dry-run (validate) and discover mapping gaps.

In addition, as part of the SA effort we are also planning to extend the Alerting Plugin capabilities in form of a generic framework for executing rules and return the results. This will allow other plugins, such as SA, to use the Alerting Plugins ability to execute (document level) monitors and return the results back. Hence some of the heavy lifting required for executing the rules on the incoming documents, generate findings and triggering alerts (Item 4-6) can be harnessed from the Alerting Plugin. SA will need to add the instrumentations, state machines and life-cycle management around tasks being offloaded to Alerting.

We are currently in the process of generating more clarity on the UX front (Item 7 &8), however, SA will have its own dashboard, tabbed (linked) from the common OS dashboard, for dedicated experience. We could leverage components from the existing dashboards (such as Alerting) to keep the consistency of experience for customers. Similarly any new component type built up as part of the SA dashboard experience, will be contributed to the core dashboards, to make them available for other dashboard plugins to consume.

What all in the list is getting launched in 2.3?

@brijos We are targeting the SA MVP for 2.3 release, which will need to have all the P0 components (item 1-10) listed above. I am planning to add specific issues for each of them to be able to capture more task-level details.

[sandervandegeijn]

Are you conforming to ECS for the data ingest?

[Naarcha-AWS]

I'll follow up on this to locate some additional POCs.

[getsaurabh02]

@ict-one-nl Yes we are for the pre-canned rules. Also we are shipping some mapping templates, which users can configure as part the threat-detector setup, which will allow them create index aliases to be able to run the pre-canned rules, even if their existing data sources are not conforming to the common schema format.

[sandervandegeijn]

Cool thanks! :) So, the format will be based on ECS?

I'm also looking at the Elastic side of this equation, with filebeat. It could be very usable to use this to index the data to opensearch (maybe through Kafka, but the main problem is dat they are using ingest pipelines to process the data. Would the solution that is proposed solve this?