Elasticsearch 5.2.2, Memory keeps on increasing steadily untill ES gets killed by System OOM Killer

[jay-dihenkar]

Elasticsearch version: ES Version 5.2.2

Plugins installed: [ only defaults ]

JVM version (java -version): Oracle JDK 1.8.0_112-b15

OS version (uname -a if on a Unix-like system): centos 6.8 ( 2.6.32-642.6.2.el6 )

Description of the problem including expected versus actual behavior:

Our Environment is: 3 node ES Cluster with 3 data nodes. We have upgraded ES from 2.3.3 to 5.2.2.

The data nodes are allocated 31gb heap ( as recommended by ES Community ). Node1 mostly serves the search requests while Node 2/ Node 3 are used for Bulk insertions.

We have seen a constant surge in the Memory usage of ES Node 1 ( node mostly used for search queries ), it starts up with 32g res memory and then res memory goes to 40g...45g...50g.. 56g...60g...62g and KILLED! ( by kernel's OOM killer ). This happens over the duration of 24-30hrs. The only thing we can do at this point is restart ES and same repeats over.

I have already gone through https://discuss.elastic.co/t/out-of-memory-invoked-oom-killer/807951 but this doesn't apply here as we are running on a Physical server with centos 6.8 ( 2.6.32-642.6.2.el6 ) with 64GB RAM and 24 core processors.

From this article, https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html , we can make out that ( guess ) that the memory above 32g+ is related to Lucene caching. Can someone please throw more light on what's happening here?

I have gone through github issues which already says some memory leak issue was already fixed before es5.2.2.

It'd be great if someone can help understand this behavior and possible solution to this.

PS: This was not happening with ES 2.3.3 on contrary.

Some more details:

No FULL GCs at all, following is the output of jstat just before ES got killed by system OOM killer

Timestamp         S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT   
       367764.3   0.00 100.00  19.66  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
       367769.3   0.00 100.00  24.77  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
       367774.3   0.00 100.00  30.94  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
       367779.3   0.00 100.00  34.65  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
       367784.3   0.00 100.00  40.53  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
       367789.3   0.00 100.00  43.95  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
       367794.3   0.00 100.00  48.67  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
       367799.3   0.00 100.00  57.06  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
       367804.3   0.00 100.00  64.82  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
       367809.3   0.00 100.00  70.31  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
Timestamp         S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT   
       367814.3   0.00 100.00  76.48  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
       367819.3   0.00 100.00  78.75  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
       367824.3   0.00 100.00  85.45  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
       367829.3   0.00 100.00  89.59  77.69  92.53  84.63   3967  227.608     0    0.000  227.608
       367834.3   0.00 100.00   1.36  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367839.3   0.00 100.00   9.36  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367844.3   0.00 100.00  11.64  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367849.3   0.00 100.00  18.63  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367854.3   0.00 100.00  23.14  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367859.3   0.00 100.00  29.02  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
Timestamp         S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT   
       367864.3   0.00 100.00  33.43  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367869.3   0.00 100.00  39.25  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367874.3   0.00 100.00  45.80  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367879.3   0.00 100.00  50.99  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367884.3   0.00 100.00  57.25  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367889.3   0.00 100.00  63.12  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367894.3   0.00 100.00  68.36  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367899.3   0.00 100.00  76.95  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367902.6   0.00 100.00  82.29  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367902.6   0.00 100.00  82.29  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
Timestamp         S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT   
       367902.6   0.00 100.00  82.29  77.44  92.53  84.63   3968  227.667     0    0.000  227.667
       367902.6   0.00 100.00  82.29  77.44  92.53  84.63   3968  227.667     0    0.000  227.667

[abeyad]

I have already gone through https://discuss.elastic.co/t/out-of-memory-invoked-oom-killer/807951

This is an invalid page. Did you mean https://discuss.elastic.co/t/out-of-memory-invoked-oom-killer/80795 (without the 1 at the end)? That question refers to a 2.4 cluster.

Its impossible to know what the problem is without a heap dump.

You might try disabling the netty pooled allocator, by starting ES with

-Dio.netty.allocator.type=unpooled

See #22452 for a history on this topic.

You may also be encountering heap spikes on certain aggregations, see #24215

But without a heap dump, its impossible to tell exactly.

[clintongormley]

I would also suggest upgrading to the latest version as a number of potential causes have already been fixed.

[jasontedor]

This is being killed by the OOM killer, not a Java heap OOM so I'm not sure about disabling the pooled allocator or heaping because of aggregations. It could be that there is a native leak of which there have been a few bugs there recently. However, there is also a known native leak in the JDK that is fixed in 8u131 (e.g., https://bugs.openjdk.java.net/browse/JDK-8178124). So indeed, I think you should upgrade both Elasticsearch and your JDK to the latest and only if this continues to reproduce after should a closer inspection be made.

[jasontedor]

Also, in case it's not obvious, you should do these upgrades one at a time so that if the issue is resolved by one of these two steps you know which one it is.

[jay-dihenkar]

@jasontedor: Upgrades are expensive in terms of time/resources. I'd prefer JDK upgrade first.
Can you provide some assistance to dig into it further (key factors to check) before we decide and execute upgrades?

[abeyad]

Can you provide some assistance to dig into it further (key factors to check) before we decide and execute upgrades?

What do you mean? If you upgrade the JDK and that doesn't solve the issue, then look to upgrading the ES version, because as @clintongormley and @jasontedor mentioned, several of these causes have been taken care of in the latest version.

If you have a heap dump, it should give an indication beforehand of whether the problem relates to those we've already seen or its something totally different. Regardless, it seems trying to upgrade the one query node is simpler than trying to analyze a heap dump.

[jasontedor]

No additional feedback, closing.

[jay-dihenkar]

Hi Jason,

We upgraded JDK last week to 8u131 but do not see any improvements yet. Will let you know further if we have any improvements or we may look forward to ES Upgrade.

Thanks

[jay-dihenkar]

@jasontedor : Is this in any way related to issue being faced? - https://www.elastic.co/blog/tracking-down-native-memory-leaks-in-elasticsearch

[jasontedor]

@jay-dihenkar The leaks discovered there were fixed in 5.2.1.

[jay-dihenkar]

I have been randomly experimenting... Looks like if req circuit breaker limit is greater than parent circuit breaker and parent circuit breaker gets tripped multiple times causes memory to leak. ( tested on es 5.2.x and 5.3.x ).

Also adding to this... looks like heap size and stack size are also related, on an ES having 300gb data with 16gb heap and 1m stack size takes much more memory (28gb res mem) than the same setup with 12gb heap and 320kb (20 gb res mem). Both work well.

PS: midnight experiments so I didn't document results. Feel free to ignore if this looks too vague.