Universal fields API

[stu-elastic]

Rather than requiring users to use standard list and map objects to deal with field access, update, etc. We should expose an API that facilitates these operations while making it easy to handle fields that do not currently exist.

Users should be able to use the API to access fields in either doc values or source, where the implementation can pick the most performant option (doc values).

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Scripting)

[elasticmachine]

Pinging @elastic/es-search (:Search/Mapping)

[jpountz]

Users should be able to use the API to access fields in either doc values or source, where the implementation can pick the most performant option (doc values).

I was actually hoping to go the other way around. With runtime fields coming, there's much less incentive to leave fields unmapped and I hope the more common practice will be to dynamically map fields as runtime fields, e.g. with a dynamic template:

PUT myindex
{
  "mappings": {
    "dynamic_templates": [{
      "runtime_by_default": {
        "match_mapping_type": "*",
        "mapping": {
          "type": "runtime",
          "runtime_type": "{dynamic_type}"
        }
      }
    }],
    ...
  }
}

If we do that, then we won't even need _source as the responsibility of reading data from the _source and exposing it the right way will be the responsibility of the mappings rather than the responsibility of scripts. And _source will only be ever needed when the fact that doc flattens the JSON structure proves problematic, which hopefully means rarely.

[rjernst]

@jpountz How do you envision that working for nested documents? Currently nested docs cannot be accessed through doc values.

[jpountz]

I don't know how that could work but I wonder if this is really a scripting problem as queries and aggregations have the same limitation: you can't leverage nested fields unless you enter a nested context via the nested query or the nested aggregation. So I'd be keen to think about the nested problem more broadly.

[rjernst]

It's a problem in scripting for ingest, where a user may want to manipulate nested docs in source. I agree on the search side nested queries/aggs should be used.