Fix flaky integration test of ShardsLimitAllocationDeciderRemoteStoreEnabledIT

[pandeydivyansh1803]

Description

Fix flaky ShardsLimitAllocationDeciderRemoteStoreEnabledIT by avoiding race condition for remote backed clusters.

Problem:
Integration test for ShardsLimitAllocationDeciderRemoteStoreEnabledIT was failing intermittently due to an AlreadyClosedException in multiple threads. The exception occurred because the Directory was being closed while there were still ongoing segment upload operations to the remote store. Specifically, during the tearDown process, when the test was attempting to clean up indices, some threads were still trying to access the Directory to upload segments, but the Directory had already been closed. This race condition manifested as an AlreadyClosedException with the message "this Directory is closed" being thrown from the BaseDirectory.ensureOpen() method. The issue occurred in the RemoteDirectory.uploadBlob operation, which was attempting to read from a closed Directory while syncing segments to the remote store.

Solution:
The solution is implemented through a robust cleanup mechanism in the form of a cleanUp method. This method ensures proper synchronization of remote store operations before test teardown by executing a forced flush operation on all indices created during the test. The implementation uses a FlushRequest with force(true) to guarantee flush execution even without pending changes, and waitIfOngoing(true) to ensure completion of any concurrent flush operations. This synchronization step is crucial as it allows all segment uploads to complete before the Directory is closed, effectively preventing the AlreadyClosedException that was previously occurring due to race conditions. This change has proven stable across 2000+ test iterations.

Related Issues

Resolves #17931

Check List

• [✅] Functionality includes testing.
• [✅] API changes companion pull request created, if applicable.
• [✅] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for 0794e4c: SUCCESS

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.46%. Comparing base (560ac10) to head (18d8862).
Report is 57 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18236      +/-   ##
============================================
- Coverage     72.56%   72.46%   -0.11%     
+ Complexity    67261    67213      -48     
============================================
  Files          5476     5476              
  Lines        310478   310487       +9     
  Branches      45133    45133              
============================================
- Hits         225313   225005     -308     
- Misses        66840    67189     +349     
+ Partials      18325    18293      -32     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

❌ Gradle check result for 6de3633: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[andrross]

Thanks for fixing this @pandeydivyansh1803!

[github-actions]

❌ Gradle check result for 6de3633: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 18d8862: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.