Description
Today sorting by timestamp on a top-hits query that targets time-based indices doesn't take into account that the ranges of timestamp in each index don't overlap. The query phase computes the top N in each shard, independently of the results returned by shards that contain data before/after it. Considering that searches are now throttled by default and that we perform partial merges efficiently, it should be possible to record the bottom hit of the top hits after a partial merge and use it as a hint for any subsequent shard search. Each shard could then compare the bottom hit sort values with the range of values that it contains using the indexed BKD-tree and shortcut the query if the global bottom values are greater/smaller than the values contained in the shard.
In a sense that is the opposite of search_after.
There are multiple benefits if we apply this strategy:
-
Most/Least recent top hit queries on time-based indices would be considerably faster if they don't need to compute aggregations especially now that shards are pre-sorted by the primary sort field.
-
Shards that contain non-competitive document would not need to keep their context open since we'd early detect that the
fetchphase is not needed. This would also work if aggregations are needed since aggregations and top_hits can run independently. -
We could automatically set the
max_concurrent_shard_requestsandbatched_reduce_sizeto a low value if we detect that shards have sorted values that don't overlap. This would reduce the impact on the cluster while still providing much faster sorted queries. -
We could impose a default sort on timestamp for time-based indices in order to ensure that we don't run costly queries on this type of pattern by default.