application hanging on OOME

[eshkel]

Elasticsearch version:
elasticsearch-5.1.1
Plugins installed: []
no
JVM version:
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
OS version:
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04

Description of the problem including expected versus actual behavior:
Single node application hang up after about 40 minutes running (about 3 000 000 documents added ). The usage is something like inserting about 2000 documents per second, each documents portion inserted as a bulk operation. That behavior was observed for ES with 28 000 000 records running with 2g heap size. After that all records was removed and heap size was set to 3g - the result was same. Run again, records was removed and heap size set to 4g - result was same. Seems the server can hold more documents (previously it was 28 000 000) but it can't handle constant loading.

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open hist-dev 8dThttInTbKw4LZH6UnJ0g 5 1 3013503 0 298.3mb 298.3mb

Steps to reproduce:

1. start server
2. run documents feeder with 2000 documents portion each second.
3. wait about hour
4. application hang up

Provide logs (if relevant):
[2017-04-05T19:17:56,969][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3384] overhead, spent [1.2s] collecting in the last [1.6s]
[2017-04-05T19:17:59,511][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3386] overhead, spent [1.2s] collecting in the last [1.5s]
[2017-04-05T19:18:02,289][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3388] overhead, spent [944ms] collecting in the last [1.7s]
[2017-04-05T19:18:04,666][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3390] overhead, spent [1.2s] collecting in the last [1.3s]
[2017-04-05T19:18:06,494][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3391] overhead, spent [894ms] collecting in the last [1.8s]
[2017-04-05T19:18:10,131][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3394] overhead, spent [1.5s] collecting in the last [1.6s]
[2017-04-05T19:18:11,132][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3395] overhead, spent [431ms] collecting in the last [1s]
[2017-04-05T19:18:13,474][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3396] overhead, spent [1.8s] collecting in the last [2.3s]
[2017-04-05T19:18:14,475][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3397] overhead, spent [291ms] collecting in the last [1s]
[2017-04-05T19:18:16,754][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3398] overhead, spent [2s] collecting in the last [2.2s]
[2017-04-05T19:18:17,755][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3399] overhead, spent [295ms] collecting in the last [1s]
[2017-04-05T19:18:20,078][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3400] overhead, spent [2.1s] collecting in the last [2.3s]
[2017-04-05T19:18:21,080][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3401] overhead, spent [299ms] collecting in the last [1s]
[2017-04-05T19:18:23,432][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3402] overhead, spent [2s] collecting in the last [2.3s]
[2017-04-05T19:18:24,433][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3403] overhead, spent [299ms] collecting in the last [1s]
[2017-04-05T19:18:26,526][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3404] overhead, spent [1.9s] collecting in the last [2s]
[2017-04-05T19:18:27,527][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3405] overhead, spent [307ms] collecting in the last [1s]
[2017-04-05T19:18:29,457][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3406] overhead, spent [1.5s] collecting in the last [1.9s]
[2017-04-05T19:18:30,458][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3407] overhead, spent [537ms] collecting in the last [1s]
[2017-04-05T19:18:32,739][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3408] overhead, spent [1.8s] collecting in the last [2.2s]
[2017-04-05T19:18:33,744][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3409] overhead, spent [555ms] collecting in the last [1s]
[2017-04-05T19:18:35,878][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3410] overhead, spent [1.6s] collecting in the last [2.1s]
[2017-04-05T19:18:36,879][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3411] overhead, spent [312ms] collecting in the last [1s]
[2017-04-05T19:18:39,119][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3412] overhead, spent [2s] collecting in the last [2.2s]
[2017-04-05T19:18:40,120][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3413] overhead, spent [317ms] collecting in the last [1s]
[2017-04-05T19:18:42,487][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3414] overhead, spent [2s] collecting in the last [2.3s]
[2017-04-05T19:18:43,488][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3415] overhead, spent [318ms] collecting in the last [1s]
[2017-04-05T19:18:45,677][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3416] overhead, spent [1.9s] collecting in the last [2.1s]
[2017-04-05T19:18:46,678][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3417] overhead, spent [318ms] collecting in the last [1s]
[2017-04-05T19:18:49,083][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3418] overhead, spent [2s] collecting in the last [2.4s]
[2017-04-05T19:18:50,084][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3419] overhead, spent [322ms] collecting in the last [1s]
[2017-04-05T19:18:52,044][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3420] overhead, spent [1.7s] collecting in the last [1.9s]
[2017-04-05T19:18:53,045][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3421] overhead, spent [319ms] collecting in the last [1s]
[2017-04-05T19:18:55,088][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3422] overhead, spent [1.7s] collecting in the last [2s]
[2017-04-05T19:18:56,088][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3423] overhead, spent [324ms] collecting in the last [1s]
[2017-04-05T19:18:58,432][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3424] overhead, spent [1.9s] collecting in the last [2.3s]
[2017-04-05T19:18:59,433][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3425] overhead, spent [326ms] collecting in the last [1s]
[2017-04-05T19:19:01,308][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3426] overhead, spent [1.7s] collecting in the last [1.6s]
[2017-04-05T19:19:02,309][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3427] overhead, spent [331ms] collecting in the last [1.2s]
[2017-04-05T19:19:03,875][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3428] overhead, spent [1.4s] collecting in the last [1.5s]
[2017-04-05T19:19:04,876][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3429] overhead, spent [583ms] collecting in the last [1s]
[2017-04-05T19:19:06,815][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3430] overhead, spent [1.4s] collecting in the last [1.9s]
[2017-04-05T19:19:07,816][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3431] overhead, spent [589ms] collecting in the last [1s]
[2017-04-05T19:19:10,020][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3432] overhead, spent [1.5s] collecting in the last [1.9s]
[2017-04-05T19:19:11,052][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3433] overhead, spent [593ms] collecting in the last [1.2s]
[2017-04-05T19:19:12,858][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3434] overhead, spent [1.4s] collecting in the last [1.8s]
[2017-04-05T19:19:13,859][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3435] overhead, spent [601ms] collecting in the last [1s]
[2017-04-05T19:19:16,207][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3436] overhead, spent [1.7s] collecting in the last [2.3s]
[2017-04-05T19:19:17,208][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3437] overhead, spent [337ms] collecting in the last [1s]
[2017-04-05T19:19:18,883][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3438] overhead, spent [1.3s] collecting in the last [1.6s]
[2017-04-05T19:19:19,884][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3439] overhead, spent [341ms] collecting in the last [1s]
[2017-04-05T19:19:21,430][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3440] overhead, spent [1.3s] collecting in the last [1.5s]
[2017-04-05T19:19:22,431][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3441] overhead, spent [341ms] collecting in the last [1s]
[2017-04-05T19:19:24,094][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3442] overhead, spent [1.3s] collecting in the last [1.6s]
[2017-04-05T19:19:25,095][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3443] overhead, spent [342ms] collecting in the last [1s]
[2017-04-05T19:19:26,856][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3444] overhead, spent [1.3s] collecting in the last [1.7s]
[2017-04-05T19:19:27,856][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3445] overhead, spent [349ms] collecting in the last [1s]
[2017-04-05T19:19:29,244][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3446] overhead, spent [1.2s] collecting in the last [1.3s]
[2017-04-05T19:19:30,245][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3447] overhead, spent [613ms] collecting in the last [1s]
[2017-04-05T19:19:32,509][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3448] overhead, spent [1.7s] collecting in the last [1.9s]
[2017-04-05T19:19:33,510][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3449] overhead, spent [353ms] collecting in the last [1.2s]
[2017-04-05T19:19:35,454][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3450] overhead, spent [1.7s] collecting in the last [1.9s]
[2017-04-05T19:19:36,455][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3451] overhead, spent [354ms] collecting in the last [1s]
[2017-04-05T19:19:37,768][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3452] overhead, spent [1.1s] collecting in the last [1.3s]
[2017-04-05T19:19:38,769][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3453] overhead, spent [624ms] collecting in the last [1s]
[2017-04-05T19:19:40,210][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3454] overhead, spent [977ms] collecting in the last [1.4s]
[2017-04-05T19:19:41,211][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3455] overhead, spent [624ms] collecting in the last [1s]
[2017-04-05T19:19:43,076][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3456] overhead, spent [1.2s] collecting in the last [1.8s]
[2017-04-05T19:19:44,077][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3457] overhead, spent [359ms] collecting in the last [1s]
[2017-04-05T19:19:45,731][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3458] overhead, spent [1.2s] collecting in the last [1.6s]
[2017-04-05T19:19:46,732][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3459] overhead, spent [359ms] collecting in the last [1s]
[2017-04-05T19:19:48,066][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3460] overhead, spent [1.1s] collecting in the last [1.3s]
[2017-04-05T19:19:49,067][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3461] overhead, spent [636ms] collecting in the last [1s]
[2017-04-05T19:19:51,159][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3462] overhead, spent [1.4s] collecting in the last [2s]
[2017-04-05T19:19:52,160][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3463] overhead, spent [364ms] collecting in the last [1s]
[2017-04-05T19:19:54,020][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3464] overhead, spent [1.4s] collecting in the last [1.8s]
[2017-04-05T19:19:55,021][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3465] overhead, spent [366ms] collecting in the last [1s]
[2017-04-05T19:19:56,672][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3466] overhead, spent [1.4s] collecting in the last [1.6s]
[2017-04-05T19:19:57,673][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3467] overhead, spent [370ms] collecting in the last [1s]
[2017-04-05T19:19:59,116][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3468] overhead, spent [1.2s] collecting in the last [1.4s]
[2017-04-05T19:20:00,117][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3469] overhead, spent [373ms] collecting in the last [1s]
[2017-04-05T19:20:02,114][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3470] overhead, spent [1.7s] collecting in the last [1.9s]
[2017-04-05T19:20:03,115][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3471] overhead, spent [661ms] collecting in the last [1s]
[2017-04-05T19:20:04,511][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3472] overhead, spent [1s] collecting in the last [1.3s]
[2017-04-05T19:20:05,512][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3473] overhead, spent [675ms] collecting in the last [1s]
[2017-04-05T19:20:06,985][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3474] overhead, spent [1s] collecting in the last [1.4s]
[2017-04-05T19:20:07,986][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3475] overhead, spent [679ms] collecting in the last [1s]
[2017-04-05T19:20:09,817][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3476] overhead, spent [1.2s] collecting in the last [1.8s]
[2017-04-05T19:20:10,818][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3477] overhead, spent [381ms] collecting in the last [1s]
[2017-04-05T19:20:12,097][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3478] overhead, spent [1s] collecting in the last [1.2s]
[2017-04-05T19:20:13,098][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3479] overhead, spent [679ms] collecting in the last [1s]
[2017-04-05T19:20:15,768][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3480] overhead, spent [1.8s] collecting in the last [2.6s]
[2017-04-05T19:20:16,771][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3481] overhead, spent [339ms] collecting in the last [1s]
[2017-04-05T19:20:18,296][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3482] overhead, spent [824ms] collecting in the last [1.5s]
[2017-04-05T19:20:19,296][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3483] overhead, spent [344ms] collecting in the last [1s]
[2017-04-05T19:20:21,297][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3485] overhead, spent [346ms] collecting in the last [1s]
[2017-04-05T19:20:26,416][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3490] overhead, spent [342ms] collecting in the last [1s]
[2017-04-05T19:20:28,759][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3491] overhead, spent [1.3s] collecting in the last [2.3s]
[2017-04-05T19:20:51,764][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3513] overhead, spent [1.2s] collecting in the last [1.9s]
[2017-04-05T19:21:18,820][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3539] overhead, spent [1.2s] collecting in the last [2s]
[2017-04-05T19:21:43,770][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3563] overhead, spent [1.2s] collecting in the last [1.9s]
[2017-04-05T19:22:09,378][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3588] overhead, spent [1.3s] collecting in the last [1.6s]
[2017-04-05T19:22:35,356][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3613] overhead, spent [1.4s] collecting in the last [1.9s]
[2017-04-05T19:22:54,596][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3631] overhead, spent [1.4s] collecting in the last [2.2s]
[2017-04-05T19:22:55,987][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3632] overhead, spent [567ms] collecting in the last [1.3s]
[2017-04-05T19:23:14,254][ERROR][o.e.t.n.Netty4Utils ] fatal error on the network layer
at org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:141)
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:76)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:296)
at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:861)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:375)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:66)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
at org.elasticsearch.http.netty4.cors.Netty4CorsHandler.channelRead(Netty4CorsHandler.java:76)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:536)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:490)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
at java.lang.Thread.run(Thread.java:745)
[2017-04-05T19:23:14,259][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3633] overhead, spent [2.6s] collecting in the last [2.7s]
[2017-04-05T19:23:17,460][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3635] overhead, spent [1.3s] collecting in the last [1.4s]
[2017-04-05T19:23:22,723][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3636] overhead, spent [5.2s] collecting in the last [2.6s]
[2017-04-05T19:23:24,402][INFO ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3637] overhead, spent [1.6s] collecting in the last [4.2s]
[2017-04-05T19:23:25,653][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3638] overhead, spent [1.2s] collecting in the last [1.2s]
[2017-04-05T19:23:28,216][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3639] overhead, spent [1.4s] collecting in the last [1.4s]
[2017-04-05T19:23:29,296][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3640] overhead, spent [2.1s] collecting in the last [2.1s]
[2017-04-05T19:23:30,331][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3641] overhead, spent [1s] collecting in the last [1s]
[2017-04-05T19:23:33,203][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3642] overhead, spent [2.8s] collecting in the last [2.8s]
[2017-04-05T19:23:34,324][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3643] overhead, spent [1.1s] collecting in the last [1.1s]
[2017-04-05T19:23:35,599][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3644] overhead, spent [1.2s] collecting in the last [1.2s]
[2017-04-05T19:23:14,406][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [xp-1] fatal error in thread [Thread-1122], exiting
java.lang.OutOfMemoryError: Java heap space
at io.netty.buffer.UnpooledHeapByteBuf.(UnpooledHeapByteBuf.java:46) ~[?:?]
at io.netty.buffer.UnpooledByteBufAllocator.newHeapBuffer(UnpooledByteBufAllocator.java:59) ~[?:?]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:160) ~[?:?]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:151) ~[?:?]
at io.netty.buffer.Unpooled.buffer(Unpooled.java:116) ~[?:?]
at io.netty.buffer.Unpooled.copiedBuffer(Unpooled.java:435) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:64) ~[?:?]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) ~[?:?]
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:66) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) ~[?:?]
at org.elasticsearch.http.netty4.cors.Netty4CorsHandler.channelRead(Netty4CorsHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) ~[?:?]
[2017-04-05T19:23:36,934][WARN ][o.e.m.j.JvmGcMonitorService] [xp-1] [gc][3645] overhead, spent [1.3s] collecting in the last [1.3s]

[jasontedor]

You should upgrade, you're on a version of Elasticsearch that does not disable the Netty recycler which is prone to the problem that you're experiencing here. See #22452.

If that does not fix your problem, you need to look at a heap dump (it should be produced when Elasticsearch OOMs) and see what is occupying all the heap.

Beyond this, Elastic provides a forum for asking general questions and instead prefers to use GitHub only for verified bug reports and feature requests.

As such, I hope you don't mind that I close this.

[rohanil]

@eshkel Could you solve the problem by suggestions mentioned by @jasontedor ? I am facing the same problem in ES 5.6.