Re-evaluate warmers

[martijnvg]

Warmers are great, but have to be used with caution, because you may end up caching things that are never end up being used and picking the right search request is not always straight forward. Also running the query phase of a search request you may end up doing more work for warming then is required for caching purposes.

Nowadays we have:

• eager field data loading
• eager norms loading
• filter cache from 2.0 has been improved the only auto cache after a filter has been used a couple of times to make sure useless entries don't make it in the filter cache.

Warmers were added (around version 0.20) long before the above features/improvements were created. The only other benefit I can think of warmers give us, is that it warms up the inverted index, but this can also be done in a similar way Lucene's SimpleMergedSegmentWarmer is warming up (filling up the filesystem cache) the index. The SimpleMergedSegmentWarmer also access part of the term dictionary instead of running a query what would happen with warmers. I think just warming up the term directory is sufficient for warmers, also warming up the posting list may be too much as big chucks may end up never being accessed.

Maybe we should add a eager_loading option to field mappings, which can have the following options: fielddata, norms and index. (multiple values can be specified) This option will then supersede the other eager loading options.

[jpountz]

+1 It would be nice to remove query warmers and and track usage stats about queries in order to automatically warm up the right things.

[rmuir]

Honestly, i don't think we should have a separate eager_loading option.

One of these days, we are gonna flip the switch and remove this laziness in lucene for things like norms, the complexity drives me crazy every time I think about it.

So better now, than later, to start making things actually scale instead of trying to play tricks with lazy loading.

[clintongormley]

The problem of removing lazy loading is that lots of our users have old indices which are essentially archives, which fits well with lazy loading. Especially if we move to doc values by default, eager loading could use a lot of resources unnecessarily.

I'm in favour of removing the warmers API and using the eager/lazy options explicitly instead. Possibly we could add output to the explain API which provides the user info about which fields could benefit from eager loading.

As a second step, I love the idea of Elasticsearch managing eager loading automatically by tracking field usage.

[rmuir]

The problem of removing lazy loading is that lots of our users have old indices which are essentially archives, which fits well with lazy loading. Especially if we move to doc values by default, eager loading could use a lot of resources unnecessarily.

How so? Most docvalues don't use any heap memory. The laziness is mostly stupid and obselete... all warming does is ensure the first queries don't have to suffer from initialization of pointers and such. There is some small in-ram component, e.g. string fields load a small "index" to the terms, but thats just like the terms dictionary, which is already not lazy at all.

[clintongormley]

@rmuir made the good point that, eg if a user has lots of indices, most of which are seldom used (eg old logs) and so don't have norms, etc loaded eagerly, a single query could force loading of norms across all indices and use up all the memory. So relying on lazy loading can be a ticking time bomb.

[jpountz]

I hate lazy loading too (I really do), but I think the point of lazy loading is more about lots of seldomly used fields than indices. And this is unfortunately common due to the fact that our dynamic mappings generate analyzed string fields that have enabled norms by default.

[ppf2]

Discussed with @martijnvg previously on this. There is a use case to fire warmers to load up a custom/internal cache when new data is indexed. It can be useful to some users if we can make index warmers pluggable.