[BUG] Test failure in FsHealthServiceTests.testFailsHealthOnHungIOBeyondHealthyTimeout

[nknize]

This reproducible failure tripped on PR #1243

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.monitor.fs.FsHealthServiceTests.testFailsHealthOnHungIOBeyondHealthyTimeout" -Dtests.seed=9E7CD8BEC4482945 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1" -Dtests.locale=ja-JP -Dtests.timezone=Etc/Zulu -Druntime.java=15

org.opensearch.monitor.fs.FsHealthServiceTests > testFailsHealthOnHungIOBeyondHealthyTimeout FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([9E7CD8BEC4482945:1C0886B7B0B45F0D]:0)
        at org.junit.Assert.fail(Assert.java:86)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.junit.Assert.assertTrue(Assert.java:52)
        at org.opensearch.monitor.fs.FsHealthServiceTests.testFailsHealthOnHungIOBeyondHealthyTimeout(FsHealthServiceTests.java:228)

[nknize]

/cc @Bukhtawar

[Bukhtawar]

Tried 100 iterations, unable to repro, this seems like a timing issue to me. I would try to re-adjust them a bit though

./gradlew ':server:test' --tests "org.opensearch.monitor.fs.FsHealthServiceTests.testFailsHealthOnHungIOBeyondHealthyTimeout" -Dtests.seed=9E7CD8BEC4482945 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1" -Dtests.locale=ja-JP -Dtests.timezone=Etc/Zulu -Druntime.java=15 -Dtests.iters=100

> Configure project :qa:os
Cannot add task 'destructiveDistroTest.docker' as a task with that name already exists.
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 6.6.1
  OS Info               : Mac OS X 10.15.7 (x86_64)
  JDK Version           : 15 (Oracle JDK)
  JAVA_HOME             : /Library/Java/JavaVirtualMachines/jdk-15.0.2.jdk/Contents/Home
  Random Testing Seed   : 9E7CD8BEC4482945
  In FIPS 140 mode      : false
=======================================

BUILD SUCCESSFUL in 6m 51s
29 actionable tasks: 1 executed, 28 up-to-date

[CEHENKLE]

resolving -- please reopen if you see it again, @nknize

[minalsha]

@nknize closing this issue for now. Please reopen the issue if you are able to repro again. Thanks