Add circuit breaking logic for shard level results

[kaushalmahi12]

Description

This change addresses the OOM issue which is triggered when the coordinator node buffers the batched_reduce_size number of shard level results. In the current logic the circuit breaking logic is missing and can easily cause OOMs for memory intensive queries.
This change at a high level does the following

• Check the circuit breaker on each shard level result arrivals.
• Discard the new shard results post cancellation. Currently the request is only cancellable on partial or final reduce hence even if any resiliency mechanism cancels the task the co-ordinator node continue to process the shard level results until it receives batched_reduce_size results.
• Cancels the co-ordinator task and its child tasks as soon as the circuit breaker trips.

Current search request flow concerning this PR

Related Issues

Resolves #18999

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for b0ed156: SUCCESS

[codecov]

Codecov Report

❌ Patch coverage is 93.54839% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.85%. Comparing base (dc70bf6) to head (dcfe98c).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
...search/action/search/QueryPhaseResultConsumer.java	96.15%	1 Missing ⚠️
...pensearch/action/search/TransportSearchAction.java	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19066      +/-   ##
============================================
- Coverage     72.87%   72.85%   -0.02%     
+ Complexity    69380    69359      -21     
============================================
  Files          5647     5647              
  Lines        319084   319111      +27     
  Branches      46157    46159       +2     
============================================
- Hits         232528   232491      -37     
- Misses        67729    67753      +24     
- Partials      18827    18867      +40     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

❌ Gradle check result for 75e731d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for a73bbd3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for a73bbd3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 83eba80: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 927b6c8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for c96e172: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[jainankitk]

Test is still failing:

> Task :server:test

Tests with failures:
 - org.opensearch.action.search.QueryPhaseResultConsumerTests.testCircuitBreakerTriggersBeforeBatchedReduce

[github-actions]

✅ Gradle check result for dcfe98c: SUCCESS