ECS Ingest processor

[ruflin]

With ECS we know the exact structure of some fields. Based on this some common processing happens. A few examples:

source.ip -> geoip processor to enrich with geo information
user_agent.original -> user_agent enrichement

Since Elasticsearch 6.5 it is possible to have a pipeline that calls an other pipeline: https://www.elastic.co/guide/en/elasticsearch/reference/6.5/pipeline-processor.html We could provide an ECS pipeline that does all this default processing. All that users would have to do is adding it to their ingest pipeline.

Over time we could add more processing in such a pipeline. For example if we have a convention that all http.request.method should be upper case the Uppercase Processor could be applied to this field and similar things.

[webmat]

It just occurred to me that you proposed this on the ECS repo, not Beats.

So you're thinking of reference implementations (more or less), correct? They can be used for Beats, but they would be published here independent of Beats implementation details, and useable by anyone with a need for them?

I love the idea.

I think we should experiment concretely with them on Beats first, before building too much "infrastructure" to support this here, however. WDYT?

[ruflin]

The idea came definitively from the duplicated code we have in the ingest pipelines of Beats but it was meant to be more general. I don't think it's something we should do right now but I need a place to write down the idea and get other thoughts on it.

[webmat]

Here's some ideas that have been occurring to me while working on the Beats migrations. They're broken up in very small ideas, but we could likely join a few together to create these reference pipelines:

• Categorize private/public IP addresses
  • Replace nginx module's, that doesn't support IPv6
• Extract source IP out of x-forwarded-for
  • See nginx module implementation, that gets the first public IP in the list, not just the first IP... (apparently there's some subtlety to this?)
• GeoIP
• User Agent parsing
  • Perform all necessary field renames
  • consider populating user_agent.version and user_agent.os.version with full version strings. Right now we only have the numbers broken out in separate fields.
• Break apart / reconstruct URL?

[webmat]

Other ideas

• populate related.ip automatically based on expected ECS ip fields
• calculate network.community_id based on expected ECS ip fields

[webmat]

• Extract file type from URL file names and file names. Idea from Allow searching and aggregations on the nginx.access.url field beats#6349

[webmat]

• Extract subdomain and determine "registerable domain" based on domain & public suffix list. Consider the Painless DomainSplit function

cc @andrewkroh

[webmat]

• Perform the extraction of IP address and domain from .address fields

[webmat]

• Save original @timestamp value in event.created, prior to parsing an event's timestamp

[webmat]

• Fallback pipeline to populate message based on a few key fields

[webmat]

• Pipeline that performs a batch of field mappings from old field to new field, including mapping values. See this discussion and this code https://github.com/elastic/beats/pull/10176/files/026d9042ba4fe96b4e72b72444b97bcaadb46f69#r251149547 for an example.
  • Mappings would be passed via a temp variable that's deleted by the pipeline.

[webmat]

• calculate a directional geo_shape when both ends of a network connection have a geo_point. See Update module ingest node templates to include geo_shapes when possible beats#11702 for details