[Bugfix] Fix incorrect document order when there's exception during batch ingest

[chishui]

Description

This is to fix a document incorrect order bug with batch ingestion when some docs run into exception.

bug is described here: #14336
the integration test here #14335 was also added.

Skip changelog and backport to 2.x

Related Issues

• [BUG] Batch ingestion appears to reorder documents in response with failures #14336
• Add failing tests for apparent batch reordering bug #14335

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for f54f43a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[chishui]

❌ Gradle check result for f54f43a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

> Task :server:test

Tests with failures:
 - org.opensearch.index.shard.IndexShardTests.testRefreshLevelRestoreShardFromRemoteStore

13137 tests completed, 1 failed, 32 skipped

open flaky issue #10785

[github-actions]

❌ Gradle check result for 02dde2a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[chishui]

❌ Gradle check result for 02dde2a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Tests with failures:

• org.opensearch.indices.IndicesRequestCacheIT.testQueryRewriteDates {p0={"opensearch.experimental.feature.pluggable.caching.enabled":"false"}}
• org.opensearch.indices.IndicesRequestCacheIT.classMethod

#14288

[github-actions]

❕ Gradle check result for 5ed3400: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.73%. Comparing base (b15cb0c) to head (5ed3400).
Report is 426 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14341      +/-   ##
============================================
+ Coverage     71.42%   71.73%   +0.31%     
- Complexity    59978    62086    +2108     
============================================
  Files          4985     5117     +132     
  Lines        282275   291712    +9437     
  Branches      40946    42168    +1222     
============================================
+ Hits         201603   209253    +7650     
- Misses        63999    65193    +1194     
- Partials      16673    17266     +593     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[andrross]

@chishui Can you send out a PR to address my comments about the tests?

[chishui]

@chishui Can you send out a PR to address my comments about the tests?

will do