Remove _field_stats endpoint

[jimczi]

_field_stats endpoint has been deprecated in 5 and should be removed in 6.

[jimczi]

@Bargs @spalger can you confirm that Kibana 6 will not use _field_stats at all and rely only on _field_caps and ES request cache for aggs ?

[Bargs]

We no longer use field_stats to get field info at index pattern creation time, but we do still use it prior to executing searches for some index patterns. There are plans to remove that option (it's already deprecated) but it does not look like that's happened yet. I'm not sure if it's targeted for 6.0. @epixa ?

[spalger]

Pretty sure we are still planning to ship the "expand indices" option with K 6.0 (where we convert index* into an index list using the _field_stats API). IIRC it was intended as a backup plan in case the wildcard optimizations that ES has implemented aren't sufficient for some users.

If we are confident that removing this optimization from Kibana won't result in worse, or specifically unusable, performance then I'm not opposed to removing it now.

[jimczi]

If we are confident that removing this optimization from Kibana won't result in worse, or specifically unusable, performance then I'm not opposed to removing it now.

If the optim is to check if an index matches document in a specific timeframe with _field_stats then I think it's safe to remove. The only difference with the _field_stats solution is that the optim will happen locally on each shard rather than on Kibana side. If you have 2000 shards you'll send 2000 queries that should return very fast if the min/max timeframe is empty for them.
@colings86 should confirm but the min/max rewriting for time range queries plus the request cache should be enough to have decent perf even when doing a gazillion shard query.

[colings86]

@colings86 should confirm but the min/max rewriting for time range queries plus the request cache should be enough to have decent perf even when doing a gazillion shard query.

I agree

[trevan]

With _field_stats gone, how does one easily calculate "old indices"? https://www.elastic.co/blog/managing-time-based-indices-efficiently has us using _field_stats and I don't think a min/max aggregation will be performant against lots of indices. I had mentioned this at #23914 (comment) as well but no one seemed to respond.

A problem with the min/max rewriting that I've just noticed is that it appears to require some warming up that is painful on our cluster. Our Kibana currently uses the _field_stats for index calculation and requests usually are <10 seconds. But if I run the same query without the index subset, the first time takes several minutes. After that, they are <10 seconds (though a few seconds slower). This warm up seems to happen if I haven't run a query in an hour and it increases the load average in my cluster fairly substantially.

[spalger]

@trevan sounds like you would benefit from #14835

[trevan]

@spalger, thanks for pointing that out. Unfortunately, we do have deletions and updates. Not a lot (<5% of the data), though.

[jimczi]

@trevan the "expand time based indices" feature should be done on ES side transparently. That's what we're trying to achieve. Kibana does not replace _field_stats with a min/max aggregation but simply remove the optimization and let ES do the right thing regarding the time range filters.
Now what you're describing is not expected, can you share the query that takes several minutes "without the index subset" ?