[BUG] Flaky org.opensearch.monitor.fs.FsHealthServiceTests.testFailsHealthOnHungIOBeyondHealthyTimeout test

[ashking94]

Describe the bug
Flaky test - org.opensearch.monitor.fs.FsHealthServiceTests.testFailsHealthOnHungIOBeyondHealthyTimeout

To Reproduce
https://build.ci.opensearch.org/job/gradle-check/15383

Expected behavior
The test should always pass.

Plugins
NA

[tlfeng]

Looking at the test result:
Test Result (6 failures / +6)
org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting
org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting
org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting
org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting
org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica
org.opensearch.monitor.fs.FsHealthServiceTests.testFailsHealthOnHungIOBeyondHealthyTimeout

Tests with failures:
 - org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting
 - org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica

testCreateIndexWithMaxClusterShardSetting failed for 4 times due to retry mechanism, and the other 2 failed for 1 time.

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting" -Dtests.seed=5E8BE4AC6E063D80 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sr -Dtests.timezone=Africa/Bujumbura -Druntime.java=19

org.opensearch.cluster.shards.ClusterShardLimitIT > testCreateIndexWithMaxClusterShardSetting FAILED
    java.lang.AssertionError: MaxShardPerCluster 2 should be greater than or equal to MaxShardPerNode 1000
        at __randomizedtesting.SeedInfo.seed([5E8BE4AC6E063D80:146D1512C358FB38]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.cluster.shards.ClusterShardLimitIT.setMaxShardLimit(ClusterShardLimitIT.java:750)
        at org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting(ClusterShardLimitIT.java:259)

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica" -Dtests.seed=5E8BE4AC6E063D80 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-ES -Dtests.timezone=Asia/Yekaterinburg -Druntime.java=19

org.opensearch.indices.replication.SegmentReplicationIT > testScrollCreatedOnReplica FAILED
    java.lang.AssertionError: Files should be preserved
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica(SegmentReplicationIT.java:948)

org.opensearch.monitor.fs.FsHealthServiceTests > testFailsHealthOnHungIOBeyondHealthyTimeout FAILED
    java.lang.AssertionError: expected:<HEALTHY> but was:<UNHEALTHY>
        at __randomizedtesting.SeedInfo.seed([5E8BE4AC6E063D80:DCFFBAA51AFA4BC8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.opensearch.monitor.fs.FsHealthServiceTests.testFailsHealthOnHungIOBeyondHealthyTimeout(FsHealthServiceTests.java:237)

[andrross]

Fixed by #12366