[CI] FrozenEngineTests.testCircuitBreakerAccounting fails #36207
Description
- Branch: 6.x
- Build failure link: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=java8fips,nodes=virtual&&linux/97/consoleFull
- Reproduction line:
./gradlew :x-pack:plugin:core:test -Dtests.seed=C7C44F5FAA0DED6E -Dtests.class=org.elasticsearch.index.engine.FrozenEngineTests -Dtests.method="testCircuitBreakerAccounting" -Dtests.security.manager=true -Dtests.locale=sv -Dtests.timezone=US/Alaska -Dcompiler.java=11 -Druntime.java=8FIPS -Djavax.net.ssl.keyStorePassword=password -Djavax.net.ssl.trustStorePassword=password - Failure frequency: This test failed once in CI so far
Detailed trace
01:24:38 FAILURE 0.20s J0 | FrozenEngineTests.testCircuitBreakerAccounting <<< FAILURES!
01:24:38 > Throwable #1: java.lang.AssertionError: expected:<2051> but was:<1377>
01:24:38 > at __randomizedtesting.SeedInfo.seed([C7C44F5FAA0DED6E:44FF788AC7E5DD71]:0)
01:24:38 > at org.elasticsearch.index.engine.FrozenEngineTests.testCircuitBreakerAccounting(FrozenEngineTests.java:158)
01:24:38 > at java.lang.Thread.run(Thread.java:748)
I could provoke this error locally by running:
./gradlew :x-pack:plugin:core:test -Dtests.class=org.elasticsearch.index.engine.FrozenEngineTests -Dtests.method="testCircuitBreakerAccounting" -Dtests.security.manager=true -Dtests.locale=sv -Dtests.timezone=US/Alaska -Dcompiler.java=11 -Druntime.java=8 -Dtests.iters=1000
(i.e. without a seed and with a regular JDK 8). The test failed once in 1000 iterations with:
> Throwable #1: java.lang.AssertionError: expected:<2717> but was:<2043>
> at __randomizedtesting.SeedInfo.seed([ED10704C05D95875:C3C11539E587FE1B]:0)
> at org.elasticsearch.index.engine.FrozenEngineTests.testCircuitBreakerAccounting(FrozenEngineTests.java:158)
> at java.lang.Thread.run(Thread.java:748)
(for completeness, the reproduction line for that case was ./gradlew :x-pack:plugin:core:test -Dtests.seed=ED10704C05D95875 -Dtests.class=org.elasticsearch.index.engine.FrozenEngineTests -Dtests.method="testCircuitBreakerAccounting" -Dtests.security.manager=true -Dtests.locale=sv -Dtests.timezone=US/Alaska -Dcompiler.java=11 -Druntime.java=8).
I did run several more iterations and what's noticeable is that the expected value is always greater than the actual one so it seems that something is freeing up memory in the accounting circuit breaker while the test is running. @s1monw would you be able to take a look at this?