[BUG] HeapBasedRateTrackerTest are unreliable

[peternied]

What is the bug?
The tests in HeapBasedRateTrackerTest [1] expect behavior based on elapsed time vs System.currentTimeMillis() [2]. During test execution there is no guarantee that Thread.Sleep(...) times are anywhere near expected. This tests are unreliable as has been seen when attempting to onboard MacOS and Windows platforms.

How can one reproduce the bug?
Running the following command on a resource constrained machine many times until the tests fail
./gradlew test --tests org.opensearch.security.auth.limiting.HeapBasedRateTrackerTest

How should this be fixed?
The way time is calculated show be supplied by a TimeProvider that can be mocked during unit tests for reliability

[1] https://github.com/opensearch-project/security/blob/main/src/test/java/org/opensearch/security/auth/limiting/HeapBasedRateTrackerTest.java#L27
[2] https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/util/ratetracking/HeapBasedRateTracker.java#L88

[stephen-crawford]

@peternied for this issue did you only see errors during the Mac and Window launch? I am looking at it but have not gotten the test to fail yet.