Custom SIEM

A declarative engine for SIEM-style analytics - rules are data and not complex code.

SIEM engine being a tool used for SoC analysis which collects data from different systems and looks from security threats. Proprietary solutions usually collect, aggregate and analyze data from organize wide applications, devices, server and users in real time. They usually provide a unified approach for the Organization's security posture allowing to detect, investigate and respond to the security incidents swiftly and effectively. SIEM

For a traditional SIEM Engine relies on a proprietary model structure which involves complex licensing cost. With this project proposing a custom SIEM language build on top of opensource pattern which can modularized and deployed as a (JSON/ YAML document). The module then interprets the data against pandas DataFrame execution layer backed by opensource execution layer (MongoDB which can be extended in future to support more data providers). Adding a new detection would ideally involve wrting configuration and not complex deployment code.

Status Specification complete -- docs/REFERENCE.md for full schema. The implementation is in python which scripts provided to containerize and orchestrate the same.

How it helps?

Traditional SIEM query languages are expressive but tie detection logic to a specific backend and licensing model. Custom SIEM keeps the parts of this model while keeping a composible grammar which is easily extensible and run natively.

Architecture

flowchart LR
    Manager["Manager <br/> shared collection configuration, time windows etc"] --> Header["Detection Header<br/>joins detections, routes results"]
    Header --> D1["Worker: simple(Single Source)"]
    Header --> D2["Worker: complex (Multiple Sources)"]
    Header --> D3["Worker: complex-select (Multiple Sources with an ability to load data on the fly)"]
    D1 --> Fn["Auxillary Functions<br/> aggregate| anomaly | arithematic | basic computation"]
    D2 --> Fn
    D3 --> Fn
    Fn --> Out[("Results/alerts collection/data results")]

Loading

• Manager - Defines the common data sources that feeds the pipeline. This logical unit defines data sources, filtering based on time, specific fields. This acts as a central orchestrator which can manage more than one Headers just based on a single shared configuration
• Header - one addressable unit of worker. In terms of orchestration it can weave together the results from more than one Worker using basic construct such as (inner/ left / right / outer, horizontal or vertical)
• Worker It is the core atomic layer of this model. Each worker is a pipeline of matching/transforming steps (==, !=, <. >). The Engine further has an addition for auxillary functions which is extensible and perform complex aggregation operations, anomaly detection, and has support for invoking custom models for performong machine learning operations.

Core Capabilities

• Declarative detection Use of json/yaml based configuration build on top of simple functions; reduces the coding overheads
• Correlate different data Easily correlate different; detections and automate the extraction process
• Build in anomaly Detection Model supports unsupverised machine learning models which are all config selectable, not hardcoded per detection.
• Easy Extension Engine can be extended to custom function, ML models making it easily to extend and customise as per needs

Example

The full grammar lives in docs/Reference.md; A simple step to find the anomaly checks over hourly login volume. The shape of the config would look similat to

{
    "login_volume_outliers": 
    {
        "isolation_forest": 
        {
            "source_cols": ["event_count_per_hour"],
            "contamination": 0.02,
            "n_estimators": 200,
            "return_anomaly": True,
        }
    }
}

A full walkthrough - Manager, Header, and Worker wired end to end is explained in detail in the reference implementation

Stack

Python · pandas · MongoDB · scikit-learn· docker · Orchestration