Frozen indices could increase resource usage on hot nodes?

[markharwood]

With our current approach to phasing searches (query then fetch) we assume that shards servicing a query will respond within similar timeframes. We keep Lucene reader contexts open until all shards have responded to the query phase so we can then fetch the final top doc IDs.

If there is a big gap in response times between the start of the query and the final set of top matches being returned from the query phase we are hanging on to resources for longer than we'd like. Frozen indices represent such a scenario where "hot" nodes undergoing constant indexing and search activity may now be asked to hang on to several reader contexts for longer while slower frozen indices hit by the same query lag in producing their responses.

One such scenario may be a phone company that keeps a "hot" window of 2 or 3 days' call data but has an investigation team who occasionally need to search for a phone number's calls going back a few years across frozen indices and are prepared to wait for the response.

In this case the long-running searches will impact the number of points-in-time that need to be kept open on the hot nodes. Of course the applications could use top-hits aggs instead of regular hits to make searches finish in the query phase. This would be a work-around but the concern raised here is that the default behaviour may be a little "trappy".

Thoughts? cc @jpountz

[elasticmachine]

Pinging @elastic/es-search

[s1monw]

@markharwood we don't keep the readers open across phases on frozen indices? Each phase if it's query or fetch open a new reader and trash it again after the phase finishes. I am not sure if I miss something but we are resource efficient here?

[jpountz]

@s1monw I think @markharwood 's point is about regular indices. Frozen indices just make it worse for regular indices because the time interval between the query phase and the fetch phase is likely longer if some frozen indices are queried.

[s1monw]

@s1monw I think @markharwood 's point is about regular indices. Frozen indices just make it worse for regular indices because the time interval between the query phase and the fetch phase is likely longer if some frozen indices are queried.

ah yeah I got confused. From my perspective it's not much different than a scroll but if there are real issues associated to this we could play some tricks and query the non-frozen indices last such that their context lifetime is shorter but I feel this is a bit jumping to conclusions. Not sure how prominent the issue is or if it's only theoretical?

[jpountz]

For now it is only theoretical. The context of this discussion is also increasing push for even colder storage than frozen indices, which would amplify this issue.

[s1monw]

I see, so what I was thinking about in the past was to allow to execute searches in a CCS like fashion where we can segment shards or based on indices and then execute one after another and merge the final results. We have all that in place infrastructure wise. This also ties into the long running queries which could solve this probem. /cc @javanna

[markharwood]

Is there value in considering a fetch phase which can fall back to retrieval based on elasticsearch _id (and version) rather than relying solely on Lucene doc IDs?
The majority of our use cases are write-once and for those cases we could have a fetch policy that was generally fast if the fetch phase happens soon after the query (reader context is still available, Lucene doc IDs still have currency) but has a fall-back to slower _id based lookups if the reader context has churned. This would allow nodes to churn readers based on a bounded resource-usage policy rather than an (unbounded?) dependency on slow-running shards.