Change the default batched_reduce_size of search requests

[jimczi]

Today we execute a partial reduce of search requests after we buffered at least 512 shard search results. The default, users can change this value with batched_reduce_size=N, seems quite high and can cause memory issue for queries that target a large amount of shards. We also want to use the partial reduce to speed up the search on subsequent search shard request (#51852) but users won't see the benefit unless they reduce the batched reduce size explicitly. Partial (and final) reduce are usually very fast so I am opening this issue to use a sane default that could save memories on coordinating node and speed up sorted queries on time-based indices (queries that can target a lot of shards). We have plenty of options so here's a non-exhaustive list:

1. Reduce the default to 5-10, that could slightly increase the overall latency but the benefit are non-negligible on time-based indices.
2. Reduce the default only for queries that can use the partial reduce to speed up subsequent shard search (sorted queries on time based indices).
3. Change the threshold to check the size of the buffered results rather than the absolute number of shards. We could also trigger partial reduce based on inactivity (if no more shard responded in the last N seconds/minutes).
4. Keep the default as it is :(

I am curious to hear your thoughts on these options.

[elasticmachine]

Pinging @elastic/es-analytics-geo (:Analytics/Aggregations)

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[polyfractal]

Wow yeah, 512 is high :)

I like the idea of triggering a reduce based on a timeout, but that seems potentially tricky to get right. And as you said, reductions should be pretty fast, so a lowish 5-50 default seems reasonable to me.

[jpountz]

I like the idea of just decreasing it to 5 or 10. It's very simple and most requests that hit a single index will keep doing a single reduction. There are some downsides too but the benefits outweigh downsides in my opinion.

[jimczi]

We've made several progress on this issue recently so apologies for not updating earlier.

While discussing the proposed change offline, we've decided to start with benchmarks to evaluate the effect of changing the default value.
We've started with the terms aggregation and opened #53216 to fix a performance bug.
In the meantime @nik9000 has changed the incremental reduce to lazily deserialize shard result aggregations on partial reduce.
This has two main advantages:

• We can now evaluate the memory cost of keeping shard aggregation results in memory more precisely. This is important if we want to allow partial reduce to be executed when a memory threshold is reached.
• We create the java representation of the aggregation tree only during the partial reduce. This makes these java objects short-lived which should help the overall garbage collection in the node.

In order to be able to effectively reduce the default value without impacting users, we think that more work is needed:

• Perform real merge-sort reduce on terms aggregations. This should reduce the cost of running incremental reduce.
• Provides micro-benchmark for date_histograms and histograms with different batched_reduce_size.

Once these tasks are completed we will resume the discussion with the additional informations in order to take a decision based on real measurements.