Query cache

[jimmyjones2]

Since #1951 got fixed I thought I'd play around with the query cache, by enabling caching on the index and monitoring the ES stats. Doing a few (repeated) simple visualizations, I discovered the only way to get cache hits was to use absolute times. However this isn't really much of a use case as with absolute times you probably won't be refreshing anyway as the data would remain the same (unless the end date is in the future, I guess)

Looking at the ES docs, this is expected - when using relative times the start/end time is different with every query - maybe in this case query_cache=false should be sent, otherwise the cache is being polluted? (which another program could be making use of)

Is there anyway Kibana can utilise the cache? Trouble is aggs are done with a query over multiple indices, so the start and end time is used against every index. If the start/end were only sent to the first and last index, the rest could be cached, however this would require an ES change, unless there is something I'm missing? Massive shame, as I would have though this could give a nice performance boost. Any ideas?

[rashidkpc]

There's really no way to accomplish using the query cache for relative times at the moment. As you've noted elasticsearch maintains no concept of the first and last index, and doesn't know about the association between the filter and the index name.

[rashidkpc]

via @jimmyjones2 on #3793

The most common way I use Kibana is to query a lot of data, find something that looks interesting, filter on it, then maybe repeat, or remove the filter and try a different one. However every time I remove a filter to get back to the previous result set, a new query is run by Elasticsearch which can often be quite slow, but will return the same result as last time.

For this use case, only one user would ever hit a cached query due to their unique time period so using the Elasticsearch query cache (#2577) would just pollute it. How about Kibana caching the previous result set, so when a filter is removed, the previous results are instantly shown? I think it would be fine to not update the time period as when drilling you don't expect the data to be updated underneath you (#2911 would make this clear).

[fmeiser]

You could round the start and end to the lower and upper bound of the current interval. In this case elasticsearch could cache the query for at least the interval. There could be a problem with entries in the future, but most people do not have future events.

What do you think?

[jimmyjones2]

Should now be fixed in ES 5.0, last bits to be fixed in #6644