[BUG] Search against index_pattern/alias with large number of shards results in OOM at co-ordinator node.

[kaushalmahi12]

Describe the bug

When searching an index pattern or alias with indices having large number of shards results in OOM for coordinator node if the shards.

When we make a search request spans large number of shards with each shard having significant result size, this triggers OOM. This happens because at coordinator node we buffer batched_reduce_size shard level query phase results in QueryPhaseResultConsumer class before we submit the merge for these buffered results on a separate thread.

The default value for this is quite high and CircuitBreaker doesn't break for these shard level results memory footprint.

Related component

No response

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

The query should be cancelled and return 4XX.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[sunqijun1]

Could you post the error log? Maybe your problem is related to this issue： #18963

[sandeshkr419]

[Search Triage] Thanks for reporting this, will await for the fix.

[kaushalmahi12]

Maybe your problem is related to this issue： #18963

This is not related to the mentioned issue, it happens because circuit breaker and task cancellation logic is not present in the shard level results consumption