ES 7.2 Fails to start due to OutOfMemory error

[fatmcgav]

Elasticsearch version (bin/elasticsearch --version):
7.2.0

Plugins installed:
Default MSI installation

JVM version (java -version):

java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

OS version (uname -a if on a Unix-like system):
Windows 2016

Description of the problem including expected versus actual behavior:
Installed Elasticsearch 7.2.0 using the Windows MSI package.
Selected 8GB of Heap space, with the machine having 16GB available.

Upon trying to start ES using .\bin\elasticsearch.exe, a Java OutOfMemory error is returned:

C:\Program Files\Elastic\Elasticsearch\7.2.0>.\bin\elasticsearch.exe
Exception in thread "main" java.lang.OutOfMemoryError: Direct buffer memory

        at java.nio.Bits.reserveMemory(Bits.java:694)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
        at sun.nio.ch.IOUtil.read(IOUtil.java:195)
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
        at com.fasterxml.jackson.dataformat.yaml.UTF8Reader.readBytes(UTF8Reader.java:327)
        at com.fasterxml.jackson.dataformat.yaml.UTF8Reader.loadMore(UTF8Reader.java:447)
        at com.fasterxml.jackson.dataformat.yaml.UTF8Reader.read(UTF8Reader.java:185)
        at com.fasterxml.jackson.dataformat.yaml.UTF8Reader.read(UTF8Reader.java:148)
        at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:184)
        at org.yaml.snakeyaml.reader.StreamReader.<init>(StreamReader.java:60)
        at com.fasterxml.jackson.dataformat.yaml.YAMLParser.<init>(YAMLParser.java:152)
        at com.fasterxml.jackson.dataformat.yaml.YAMLFactory._createParser(YAMLFactory.java:420)
        at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:321)
        at org.elasticsearch.common.xcontent.yaml.YamlXContent.createParser(YamlXContent.java:85)
        at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1087)
        at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1070)
        at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:83)
        at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:95)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

1. Install any Oracle Java 8 JDK version
2. Install Elasticsearch 7.2.0 using MSI installation
3. Select 8GB of Java memory
4. Attempt to start Elasticsearch using administrator command prompt:

C:\Program Files\Elastic\Elasticsearch\7.2.0>.\bin\elasticsearch.exe

Provide logs (if relevant):

Additional information:
I've confirmed that the issue appears to be limited to the Oracle Java 8 JDK, by setting JAVA_HOME to C:\Progra~1\Elastic\Elasticsearch\7.2.0\jdk and confirming that Elasticsearch starts cleanly and uses the bundled JDK version:

[2019-07-10T09:55:59,484][INFO ][o.e.n.Node               ] [FATMCGAV-WINDOW] node name [FATMCGAV-WINDOW], node ID [UbhLetpkQb6UUQp7WSqC3w], cluster name [elasticsearch]
[2019-07-10T09:55:59,484][INFO ][o.e.n.Node               ] [FATMCGAV-WINDOW] version[7.2.0], pid[4512], build[unknown/unknown/508c38a/2019-06-20T15:54:18.811730Z], OS[Windows Server 2016/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_144/25.144-b01]
[2019-07-10T09:55:59,500][INFO ][o.e.n.Node               ] [FATMCGAV-WINDOW] JVM home [C:\Program Files\Java\jdk1.8.0_144\jre]
[2019-07-10T09:55:59,500][INFO ][o.e.n.Node               ] [FATMCGAV-WINDOW] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=C:\Users\fatmcgav\AppData\Local\Temp\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=C:\ProgramData\Elastic\Elasticsearch\data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Xmx8192m, -Xms8192m, -XX:MaxDirectMemorySize=6g, -XX:+PrintFlagsFinal, -Dio.netty.allocator.type=unpooled, -Delasticsearch, -Des.path.home=C:\Program Files\Elastic\Elasticsearch\7.2.0, -Des.path.conf=C:\ProgramData\Elastic\Elasticsearch\config]

I've also confirmed that upgrading the Oracle Java 8 JDK to the latest release doesn't change the behaviour.

I believe this issue is related to the changes in #42006.

I tested the theory by setting the -XX:MaxDirectMemorySize flag in jvm.options, with some interesting results.
Setting the value to (Xmx / 2) ± 1 results in the JVM starting.
So if Xmx is set to 8192m, values of 4095m or 4097m worked, but 4096m didn't.

Also, setting the value to match Xmx, which mirrors the 7.1 behaviour fails with the same exception.

[elasticmachine]

Pinging @elastic/es-core-infra

[CyrilLacroix]

@fatmcgav We had the same issue, thanks for the fix!

[alexhass]

Same issue with 7.3

[alexhass]

Found something... I had an older JRE8 installed. I had JAVA_HOME deleted, but Elastic Search still found the old 8.x Java and used it. It has not used the JDK in the Elastic Search folder as documented. No idea why it is using a different Java version if there is no JAVA_HOME defined.

With the embedded JDK12 version set as JAVA_HOME it starts up, but not with an JDK8.

[williamrandolph]

I believe this issue is a near-duplicate of #47384 / #48365. What's new to me in this ticket is that setting -XX:MaxDirectMemorySize=4g is causing a problem. I'll see if there's anything I can do to fix that.

In the meantime, using JDK 9 or newer should be a reliable workaround.

Apologies for the delayed response.

[voflo]

Hi! We have the same issue using ES 7.4 and oracle java 8. Any prognosis on a fix for this?

[williamrandolph]

@voflo Yes — I've found a fix ( #48657 ) and I'm in the process of backporting it for 7.x bugfix branches. In the meantime, you should be able to avoid the issue most effectively by using an --XX:MaxDirectMemorySize that's just below a multiple of 4g. For example, if 4g is the value you want, that's equal to 4096m, so --XX:MaxDirectMemorySize=4095m ought to be effective as a workaround.

[williamrandolph]

I believe this should have been closed when #48657 got merged.