Groovy scripting - possible memory leak ES 2.3.3

[bogdanovich]

Elasticsearch version: 2.3.3

JVM version: java version "1.8.0_40"
Java(TM) SE Runtime Environment (build 1.8.0_40-b25)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)

OS version: centos 6.3 (x86_64) :: linux 2.6.32-279.el6.x86_64

Description of the problem including expected versus actual behavior:

We upgraded our cluster from 1.4.4 to 2.3.3 and now experiencing an issue where nodes become unresponsive and get heavily garbage collected. A Node restart helps for about 24 hours until 24Gb of RAM get fully utilized again.

Provide logs (if relevant):

jmap -histo:live on the node with 91.1% heap used shows that 4.5Gb is taken by org.codehaus.groovy.runtime.metaclass.MetaMethodIndex

num     #instances         #bytes  class name
----------------------------------------------
  1:      86241328     4829514368  org.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry
  2:      66209193     1922638888  [Ljava.lang.Object;
  3:      34089692     1636305216  java.lang.invoke.MethodHandleImpl$CountingWrapper
  4:      62450344     1498808256  org.codehaus.groovy.util.FastArray
  5:      33948832     1086362624  java.util.HashMap$Node
  6:        223427     1063930664  [B
  7:       6938969      999212432  [Lorg.codehaus.groovy.util.ComplexKeyHashMap$Entry;
  8:      27089896      866876672  java.lang.invoke.BoundMethodHandle$Species_LL
  9:      17686899      707475960  java.lang.invoke.BoundMethodHandle$Species_L3
 10:       5818726      612509464  [C
 11:         23980      604964856  [I
 12:      16356260      523400320  org.codehaus.groovy.util.SingleKeyHashMap$Entry
 13:        495640      515466624  [Lorg.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry;
 14:       7202607      460966848  java.lang.invoke.BoundMethodHandle$Species_L4IL4
 15:       2020760      383997512  [Ljava.util.HashMap$Node;
 16:       7202607      345725136  java.lang.invoke.BoundMethodHandle$Species_L5
 17:       3003762      264331056  java.lang.reflect.Method
 18:       7884148      252292736  java.lang.invoke.BoundMethodHandle$Species_L
 19:       7434592      237906944  groovy.lang.MetaBeanProperty
 20:       4553471      218566608  java.util.HashMap
 21:       2508224      200657920  java.lang.reflect.Constructor

[dakrone]

I believe this may be related to #18572 , which is resolved in 2.4.0 by #18975

[rmuir]

Its more than that though. Don't compile a new unique script with each request!!! Use params !!!!

[rmuir]

users that do this, will continue to hit problems, including jvm crapping out with stuff like https://bugs.openjdk.java.net/browse/JDK-8023191

seriously, the root cause is that the scripting api encourages users to send a new unique script with each request. one symptom was that es had a leak of those tons of generated classes. another symptom, will be that the jdk has a code cache leak, not fixed until java 9. Other symptoms will be slow performance, due to compiler thread pressure and other problems.

But we should address the root cause: there needs to be documentation, limits, something to prevent the anti-pattern.

[jasontedor]

Relates #8632

[bogdanovich]

Would CMSClassUnloadingEnabled flag temporarily fix the problem?

[rmuir]

no.

[rmuir]

The way to prevent the problem, again, is not to send a new unique script with each request. Instead, let the script take parameters, such as my_var, whatever parts are changing for each request, and change those parameters via params. See https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html, there are examples doing this.

That means, your script will only be compiled once, and things will be much faster.

[bogdanovich]

@rmuir Why then it was working fine for months in ES 1.4.4 ?

PS. Thanks for clarifying the issue. We indeed use unique scripts every request and will fix that.

[rmuir]

Who knows, perhaps ES 1.4.4 didnt leak? Perhaps it leaked slower, because it used groovy without invokedynamic. Not particularly interesting to me.

CMSClassUnloading is not relevant, you are using java 8, so its already on anyway by default. But CMS never stands a chance to collect the class, because of #18572

But forget all about that too: because its going to cause all kinds of other problems for your jvm (see my links and comments above), besides being horribly slow.

That's why i'd rather us not pass around a bunch of aspirin trying to treat symptoms, but address the root cause, which is that its too easy to compile a brand new slightly-different script with each request. Obviously the api invites this (thats what happens when an API's efficiency relies on caching!), but that is hard to fix, so for now at least we should try to add some kind of system limit + documentation encouraging people to do it the fast way (using params).

[clintongormley]

The docs for scripting have already been improved (see the Prefer parameters block on https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html)

We report the number of compilations and cache evictions in the node stats (GET _nodes/stats/script).

The only other thing I can think of doing is to add a safeguard which limits the number of compilations per minute to e.g. 10

Does this sound reasonable?