`action.search.shard_count.limit` should apply to the shard count post `can_match` phase

[jpountz]

action.search.shard_count.limit was initially introduced as a soft limit that prevented queries to hit too many shards. In 6.x we introduced the can_match phase that pre-filters non-matching shards and we decided to set this soft limit to unlimited by default.

Unfortunately this broke some users who were relying on this setting as a way to prevent their users from killing their clusters, e.g. by sending requests that don't have a time range.

I'd suggest to keep the default value of action.search.shard_count.limit to unlimited, but change the semantics of this setting to apply to the number of shards after the can_match request so that they could still protect their clusters against users who run queries without a time range.

Related discussion #35917

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[jpountz]

We discussed this issue today and our general feeling is that this setting should no longer be needed, but we don't want to move forward with the removal without making sure we're not missing anything. Specifically, we're thinking of two use-cases for this setting:

• Acting as a proxy against slow queries, by preventing users to run queries over too long time ranges.
• Avoiding wasted CPU cycles due to users triggering requests without a time range by mistake, which then never gets cancelled when the user walks away from their Kibana dashboard.

Are we missing something?

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)