Description
We have currently an open issue in Kibana (elastic/kibana#32742) about setting pre_filter_shard_size to 1 for all requests in case you want to query frozen indexes. This caused a couple of questions I am copying over from the other issue:
As far as I understand the documentation, the pre-filtering phase does not have any "significant overhead". Why is this value 128 by default, and not by 1? If there is no significant overhead, but we know that especially when including frozen index, we'll gain a lot of performance benefit, what's the reason of not enabling that phase by default? Since we would set this to 1 in every request (since we don't know if it includes a frozen index or not), once you enabled querying frozen indexes in the advanced setting, I would really like to understand the drawbacks of setting this to 1.
Would it potentially make more sense, setting this by default to 1 on the Elasticsearch side once ignore_throttled=false is set. Or maybe ES would even be possible to determine if an frozen index will be hit by a query and then using that appropriate value? Or (much of the question on top): Could that potentially be 1 for all requests, what are the drawbacks there?