Detect ip addresses as part of dynamic field mappings

[ruflin]

The default Elasticsearch index templates, all templates used for ECS and our packages have a dynamic field mappings for strings to map to keywords by default if it is not a number or date. Unfortunately this converts all ip addresses to keywords too. It would be great if Elasticsearch could also detect the format of ip addresses (ipv4, ipv6) and automatically map it.

This solves a problem we currently have with the Elastic Agent. Based on which environment the Agent runs in, additional metadata is attached to the events. This can be information about k8s, docker, host, cloud and is used to identify where the event originated. All this fields are keywords with a few ip addresses. We know the exact fields that will show up and could map everything in advance but means we bloat the mapping for each single dataset with all possible options even though in most cases, only 1 is needed.

If Elasticsearch would detect ip addresses, we can skip all these mappings and just let Elasticsearch put the correct mappings into place when they are needed. Like this, each dataset only contains the mappings that are required and not for all metadata.

This is how the dynamic_template looks today:

"dynamic_templates": [
  {
    "strings_as_keyword": {
      "mapping": {
        "ignore_above": 1024,
        "type": "keyword"
      },
      "match_mapping_type": "string"
    }
  }
],

[elasticmachine]

Pinging @elastic/es-search (:Search/Mapping)

[ruflin]

I opened #64978 to have a short term solution that just matches on naming conventions.

[javanna]

We discussed this as part of #85692. I will paste here the relevant part of the discussion for reference.

We are open to adding auto-detection for ip addresses, it would work similarly to how date auto-detection works. This is not a particularly popular feature request (surprisingly?). If we introduced it we may not want to enable it by default for backwards compatibility concerns, but then few users would benefit from it if it's opt-in, which is not ideal either. It was mentioned that ip auto-detection would solve a big part of the problems around applying ECS conventions in Elasticsearch, but we could not exactly understand why: ip fields need to be mapped manually, the disadvantage is that incoming documents will hold only one or two of those while your mappings need to define all the possible ip fields to ensure that if they get indexed they get the right type. How many are there in total, our of curiosity? Would it be possible to define them in a dynamic template? Is there some common naming convention for ip fields?