Skip to content

[Security Solution][Alerts] Use dynamic mappings to create smaller templates & mappings #100884

Open

Description

The Fleet and Elasticsearch teams have optimized the data ingestion mappings to use dynamic mappings as much as possible: elastic/elasticsearch#64978 Here is the current version of the template: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/core/src/main/resources/data-streams-mappings.json

We should consider doing the same for alerts as data.

The idea is to take advantage of the following observations:

  • most ECS fields are keywords, so if we default to the keyword type, the template size can be dramatically reduced
  • ECS has conventions for IP fields (*.ip) and text fields (*.message)
  • the rest of the fields can show up explicitly in the mapping.

In addition to the benefit of having smaller templates, this means that all the fields that don't show up explicitly (keywords) are zero-cost when they are not used. This reduces the cost of assuming the whole of ECS is the base of the Alert schema.

We need to be careful, though, to not cause mapping conflicts, because we're copying data from disparate indices that can be in conflict with one another. I think the following approach would be feasible:

  • The Alert mappings use dynamic fields for keywords and other special cases
  • The alert.original_event field is set to be not indexed at all or flattened
  • The Kibana code that creates the Alert documents puts all the ECS + well known alerting fields at the top level, and any other field under alert.original_event.
  • When copying the ECS + well-known fields, the code must verify that the fields have the correct types (string vs object vs number) and ignore the fields that don't match. This ensures that the resulting mapping is consistent with ECS.
  • Every other field goes under alert.original_event where it can't cause conflicts.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions