Closed
Description
The can match phase fans out the original search request to all shards.
These individual requests have a cost:
- They serialize the same request multiple times.
- They need to resolve the index permissions multiple times.
Additionally, the broadcasting of these shard request is done on the network thread without any throttling, so search requests that target a large number of shards can disturb the cluster coordination, leading to nodes leaving the cluster.
Similar to #74648, we should move to a per-node request model for this phase. We can group shard request per node and perform the local operations for multiple shard/index at once. That would reduce the cost of coordination drastically and would leave room on data nodes to perform the operations on a separate thread pool.