[Backport 2.x] Fixing flaky org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testHighWatermarkNotExceeded test case

[opensearch-trigger-bot]

Backport 7694a27 from #4012

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/989/
• CommitID: f239176

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/994/
• CommitID: f239176

[dreamer-89]

@reta : Can you take a look, both 989, 994 jobs failed due to testHighWatermarkNotExceeded test.

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testHighWatermarkNotExceeded" -Dtests.seed=30B5259A6D582958 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sv-SE -Dtests.timezone=AST -Druntime.java=17

org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT > testHighWatermarkNotExceeded FAILED
    java.lang.AssertionError: Mismatching shard routings: []
    Expected: a collection with size <1>
         but: collection size was <0>
        at __randomizedtesting.SeedInfo.seed([30B5259A6D582958:D994C428ED9EE0B6]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.lambda$assertBusyWithDiskUsageRefresh$5(DiskThresholdDeciderIT.java:362)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1049)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.assertBusyWithDiskUsageRefresh(DiskThresholdDeciderIT.java:355)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testHighWatermarkNotExceeded(DiskThresholdDeciderIT.java:189)****

[reta]

@reta : Can you take a look, both 989, 994 jobs failed due to testHighWatermarkNotExceeded test.

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testHighWatermarkNotExceeded" -Dtests.seed=30B5259A6D582958 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sv-SE -Dtests.timezone=AST -Druntime.java=17

org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT > testHighWatermarkNotExceeded FAILED
    java.lang.AssertionError: Mismatching shard routings: []
    Expected: a collection with size <1>
         but: collection size was <0>
        at __randomizedtesting.SeedInfo.seed([30B5259A6D582958:D994C428ED9EE0B6]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.lambda$assertBusyWithDiskUsageRefresh$5(DiskThresholdDeciderIT.java:362)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1049)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.assertBusyWithDiskUsageRefresh(DiskThresholdDeciderIT.java:355)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testHighWatermarkNotExceeded(DiskThresholdDeciderIT.java:189)****

@dreamer-89 yes, the needed change [1] was merged after this PR was created, if next build fails, I will recreate backport manually ...

[1] #4013

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/995/
• CommitID: f239176

[tlfeng]

@reta Got it. 😂 The order to merge the PRs are critical.
While you don't have to create backport PR manually, you can checkout this branch backport/backport-4012-to-2.x locally and merge the upstream 2.x branch into this feature branch manually, and push to update the branch.

[reta]

@reta Got it. joy The order to merge the PRs are critical. While you don't have to create backport PR manually, you can checkout this branch backport/backport-4012-to-2.x locally and merge the upstream 2.x branch into this feature branch manually, and push to update the branch.

I sadly cannot push to origin :(

[tlfeng]

I sadly cannot push to origin :(

Ah, thanks for telling me.. Seems I am able to do that. Alright, then re-running the PR check will be a easier way. 😄

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/996/
• CommitID: f239176