Revise Default max concurrent search requests

[s1monw]

Spinoff from #31171

We today use 5 * numNodes in the cluster as the default number of concurrent search requests issues by a single high level request. This has several issues ie. in the CCS case. There is also valid arguments that this number is arbitrary. Lets discuss this and find a good default that also works for CCS

[elasticmachine]

Pinging @elastic/es-search-aggs

[s1monw]

In #31171 I originally used the avg. number of shards per index in the high level request which had issues as @jpountz stated:

I'm a bit unhappy that it breaks the rule that searching N indices with 1 shard performs the same as searching 1 index with N shards.

I think what we want to achieve here is that we gain a certain level of concurrency while at the same time don't overload the cluster with a search request hitting a large number of shards on a small number of nodes. The smallish clusters with large shard counts are the critical ones here IMO if a query matching all shards causing rejections because the queue of the search threadpool on a single node fills up quickly. So with a queue size or 1000 on the search threadpool I do think we have quite some room here to have a simple default like 5 * numberOfNodes.

I also looked into the CCS issue which is not an issue after all since we take the number of nodes that participate in the request into account already.

In-fact the CCS way is actually better because it only takes into account the nodes that participate in the request. I wonder if we should change the way we use this number and limit the number of concurrent shards requests per node instead. We can then do this dynamically in InitialSearchPhase and queue up shards if we hit a node too hard as a protection mechanism. That would make the default simpler I think and more effective? ie if we set the default to 5 I think we have a good default. WDYT?

[jpountz]

I think this metric makes more sense and is easier to reason about. +1

[ywelsch]

I wonder if we should change the way we use this number and limit the number of concurrent shards requests per node instead. We can then do this dynamically in InitialSearchPhase and queue up shards if we hit a node too hard as a protection mechanism. That would make the default simpler I think and more effective

+1 to making this a per-target node limit instead of a global one.

[jasontedor]

+1