Kibana generates requests with too many docvalue_fields, causing Elasticsearch to throw errors

[mightyguava]

Describe the feature:

Kibana automatically queries all date fields in the index patterns as doc_values. There should be a way to choose which fields it requests as doc values. Or, better, it should request as doc values only fields actually relevant to the query.

Describe a specific use case for the feature:

This request has context coming from https://discuss.elastic.co/t/kibana-requesting-too-many-doc-values/147760

We have an elasticsearch cluster that indexes some events that flow through our system for debugging use. These events have pretty widely varying formats, so they end up generating a lot of different fields. Kibana maps about 2000 fields for the indexes. We don't configure these indexes manually, and just let Elasticsearch automatically generate indexes based on the data.

Performance has never been a problem. Type conflicts are pretty rare and haven't been problematic enough to warrant any action.

One day, I refreshed field mappings, and all search queries are breaking with the error:

{
  "responses": [
    {
      "_shards": {
        "failed": 95,
        "failures": [
          {
            "index": "tracer--2018-09-07",
            "node": "rMepPe8BS1m2ILlUDDQFmg",
            "reason": {
              "reason": "Trying to retrieve too many docvalue_fields. Must be less than or equal to: [100] but was [101]. This limit can be set by changing the [index.max_docvalue_fields_search] index level setting.",
              "type": "illegal_argument_exception"
            },
            "shard": 0
          }
        ],
        "skipped": 5600,
        "successful": 5600,
        "total": 5695
      },
      "hits": {
        "hits": [],
        "max_score": 0.0,
        "total": 0
      },
      "status": 200,
      "timed_out": false,
      "took": 2480
    }
  ]
}

It looks like the indexes now have 101 different date type fields. Kibana seems to automatically request every date fields as docvalue fields in every single request.

These are for "Discover" requests, and we don't ever sort/aggregate on any of these fields. I've actually never had a need sort/aggregate on any of these date fields.

Rather than having to go and manually map these to strings, Kibana should be clever enough to not request them as doc values by examining the query. If that's not possible, there should at least be some way to configure the index template to restrict which fields are used as doc values.

[robin-anil]

is there a way to stop this behavior at the Kibana layer -- a date field doesn't mean we need to send a doc value request for it

[timroes]

A short explanation here, why this is happening: Elasticsearch accepts a potential way wider range on date formats that you can inject in a document that you insert, than Kibana can in the end parse and apply it's date field formatter on. That's why we can't use the date fields from _source, since they would potentially be unparseable by us. So we request them as docvalues, since we can make sure that the format returned is parseable by us.

Since you can expand every document in discover also, and we'll need to apply the date field formatters to what's shown, we need to load all those doc_values. The only thing I could see us preventing this, if we would make a second request for the full document once it's expended in Discover, and only load the fields shown in the table before that (and thus only the date fields shown as columns via docvalues).

That has the downside of having quiet some overhead every time you want to expand a document in discover, and from our experience users are navigating rather quickly through different documents in Discover, which wouldn't be good possible anymore in that case.

Maybe we could in the future add this as a configuration option whether you want the one or the other behavior (everything loaded immediately, or just the data shown in the table and full doc only loaded once expanded).

cc @kertal