Hardcoded Heap settings for docker environments

[dennybaa]

elasticsearch/distribution/src/config/jvm.options

Lines 22 to 23 in 2b4aaab

	-Xms${heap.min}
	-Xmx${heap.max}

Is there any possibility not to hardcode these deprecated settings, they disable -XX:+UseContainerSupport which is enabled by default.

[jasontedor]

Why do you think that these settings are deprecated? It's also not clear to me why you think that setting the heap size disables container support. I see the contrary:

JDK 13:

16:59:42 [jason@totoro:~] $ /opt/java/jdk-13/bin/java -Xms2g -Xmx2g -XX:+PrintFlagsFinal -version | grep -i container
     bool PreferContainerQuotaForCPUCount          = true                                      {product} {default}
     bool UseContainerSupport                      = true                                      {product} {default}
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

JDK 14:

17:01:43 [jason@totoro:~] $ /opt/java/jdk-14/bin/java -Xms2g -Xmx2g -XX:+PrintFlagsFinal -version | grep -i container
     bool PreferContainerQuotaForCPUCount          = true                                      {product} {default}
     bool UseContainerSupport                      = true                                      {product} {default}
openjdk version "14-ea" 2020-03-17
OpenJDK Runtime Environment (build 14-ea+20-879)
OpenJDK 64-Bit Server VM (build 14-ea+20-879, mixed mode, sharing)
17:01:57 [jason@totoro:~] $ 

We do expect our users that are not running in Docker containers or don't want to rely on container support to explicitly set the heap setting. The defaults that we ship with are intended for a good out -of-box experience, but not for any production use-cases. For Docker, we expect most users to mount over a jvm.options file and configure the heap size although we understand that some instead configure memory limits. The difficulty that we have with memory limits is that it's too indirect, the JVM needs memory beyond the heap size for its own uses, so it would be sharing that memory limit with the indirectly configured heap.

I'd like to better understand the difficulty that you're experiencing with this though, can you explain, especially in light of what appears to me that heap size is not deprecated, and setting it doesn't disable container support.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Packaging)

[dennybaa]

Hi @jasontedor,

Well they are indeed not really deprecated, but on containerized environments are not required since UseContainerSupport is enabled by default (https://bugs.openjdk.java.net/browse/JDK-8196595). The fact is that these options override heap size auto-configuration based on cgroup limits. Surely JVM needs memory beyond its heap size, which is successfully treated with -XX:MaxRAMPercentage (by default 25%). So basically on a containerized environment (especially when cgroup limits are controlled), these two parameters are sufficient to control heap properly and moreover they have a default value. So there's no need to control heap by setting xmx, xms, maxram etc - it just works.

I could not agree more with all the points you gave above.

For Docker, we expect most users to mount over a jvm.options ...

Yes, right, but it would be nice to have the jvm.options configurable in the container, for example mountable, use defaults or nothing at all. I've been desperately trying to find elasticsearch oss docker container images sources, but I haven't succeed. This issue exactly relates to elasticsearch oss docker container image and I would like to rise this issue there. Since I haven't found anything I popped in here (

[jasontedor]

Thanks for the quick response and feedback @dennybaa.

For clarity, this is the right repository for the official Elasticsearch Docker images, the build is contained (pun!) in this repository.

Yes, right, but it would be nice to have the jvm.options configurable in the container, for example mountable, use defaults or nothing at all.

Given this, I still think that we are making the right tradeoffs. One aspect that we try to maintain is that our users have the same default experience no matter how they deploy Elasticsearch, whether it be via an RPM package, or our Docker images. Otherwise, users have the option to mount over a JVM options file, and if you so desire, remove the heap setting to base it on the memory limits of the container.

[jasontedor]

Relates #42660. Note that point 2. there is no longer valid since #41962.

[jasontedor]

Relates #42754, where our docs continue to recommend that you set the heap size, even if you've set a memory limit. We understand the appeal of -XX:+UseContainerSupport but we don't think it gives the right control over the heap size and direct memory size.

[dennybaa]

One aspect that we try to maintain is that our users have the same default experience no matter how they deploy Elasticsearch, our users have the same default experience no matter how they deploy Elasticsearch

Yup, right but this not configurable yet in the container, but it should be. JVM is shipped with the container as well as jvm.options file and currently forces the container to pick up these defaults.

@jasontedor So you say #41962, that if we mount an empty jvm.options file, parsing won't fail and no defaults will be substituted, right?

[dennybaa]

Well you might think to make it configurable and not rely on #42006, but rely on it if xmx xmx are set, otherwise rely on that what the newer features provide), IMO

[jasontedor]

JVM is shipped with the container as well as jvm.options file and currently forces the container to pick up these defaults.

Right, but this is deliberate as I mentioned so that users have the same out-of-box experience no matter which distribution they use.

So you say #41962, that if we mount an empty jvm.options file, parsing won't fail and no defaults will be substituted, right?

Parsing won't fail, but you can't use an empty JVM options file currently, as there are important settings that we set on behalf or users that are needed for Elasticsearch to run properly. If those settings aren't there, Elasticsearch won't run properly. Note that we are in the process of moving many of these into internal configuration (#48252), but that's not how it is today.

[dennybaa]

Note that we are in the process of moving many of these into internal configuration (#48252), but that's not how it is today.

Sounds good, once it happens I believe that what I'm talking about can be made configurable. I.e. by default can behave like xmx, xms are defaulted, but can be switched off to trigger the new possibility of heap "auto configuration". Well, thank you, ES is truly a great project 👍

[jasontedor]

Thank you for the kind words. Given the discussion here, I am going to close the issue. Let me know if that’s an error.