JvmErgonomicsTests.testExtractValidHeapSizeNoOptionPresent fails on 7.4

[dakrone]

It fails with:

org.elasticsearch.tools.launchers.JvmErgonomicsTests > testExtractValidHeapSizeNoOptionPresent FAILED
    java.lang.AssertionError: 
    Expected: a value greater than <0L>
         but: <0L> was equal to <0L>
        at __randomizedtesting.SeedInfo.seed([22C99F519C4644E0:535C63ED1A8DEF9E]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.tools.launchers.JvmErgonomicsTests.testExtractValidHeapSizeNoOptionPresent(JvmErgonomicsTests.java:59)

What's strange is that this did not print any reproduction line, so I wasn't able to try the exact test.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.4+multijob-windows-compatibility/os=windows-2019/64/consoleFull
https://gradle-enterprise.elastic.co/s/mm3dqwj56d4aa/console-log

[elasticmachine]

Pinging @elastic/es-core-infra

[dakrone]

Another failure:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.3+multijob-windows-compatibility/os=windows-2016/164/console
https://gradle-enterprise.elastic.co/s/biyd4wfkrhnns

[droberts195]

Every single Windows 2012r2 build on the 7.4 branch this month has failed with this same error: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.4+multijob-windows-compatibility/os=windows-2012-r2/

[droberts195]

Every single Windows 2012r2 build on the 7.4 branch

The fact that this hasn't been failing in 7.x doesn't mean it's not a problem in 7.x. The 7.x builds all fail because of the minio problem - see #42829 - and the test report shows that JvmErgonomicsTests has not run at all because of this.

There are no differences in either JvmErgonomics.java or JvmErgonomicsTests.java between 7.4 and 7.x so the chances are this is still a problem in 7.x too.

It's working in master though: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-windows-compatibility/os=windows-2012-r2/203/testReport/org.elasticsearch.tools.launchers/JvmErgonomicsTests/

[alpar-t]

As predicted, here's the failure on 7.x https://gradle-enterprise.elastic.co/s/k6qhteq27zplc

[williamrandolph]

It looks like this is a JDK8 issue. The default value for MaxHeapSize is physical RAM / 4. On large machines, these numbers can be larger than what fits into a 32-bit integer, but that's what JDK 8 tries to print, so max heap sizes will be truncated. Some common values get truncated to 0, unfortunately. You can test this out by supplying such values as arguments to the -Xmx parameter.

Using Powershell, the results come out like this:

PS C:\Users\william_brafford\elasticsearch> C:\Users\jenkins\.java\java8\bin\java.exe -XX:+PrintFlagsFinal -Xmx3g -version | select-string MaxHeapSize

    uintx MaxHeapSize                              := 3221225472                          {product}

That is the correct value, i.e., 3g => 3221225472 bytes. But:

PS C:\Users\william_brafford\elasticsearch> C:\Users\jenkins\.java\java8\bin\java.exe -XX:+PrintFlagsFinal -Xmx8g -version | select-string MaxHeapSize

    uintx MaxHeapSize                              := 0                                   {product}

That's not right! Or rather, (8 * 2^30) % (2^32) = 0. Any multiple of 4 GB will have this problem, since 4GB = 4 * 2^30 B = 2^32 B.

Unfortunately, our Windows tests are executing on instances with 96 GB RAM, meaning the 24GB max heap size is getting truncated to 0 in the output that we're parsing.

JDK 9 and beyond don't have this problem, and the master branch uses JDK 11 for the runtime, so that's why we aren't seeing this issue on master.

As for the fix — is there a way to make this test meaningful, given that there's a bug in the JDK 8 output? Also, is this a potential problem on Linux, or is it Windows-specific? I'll keep digging on this.

[williamrandolph]

The behavior turns out to be system-dependent — "uintx" holds the correct values on my Mac, but not on my Windows test machines.

The JDK bug report flagging this issue is here: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8074459

Since the problem behavior is in the JDK and not in our code, I think I will put out a PR that skips this test only on affected systems.

[williamrandolph]

On second thought, this bug might have some consequences for our ergonomic settings on affected machines. I'll figure out what the worst case would be and loop back for discussion.