OutOfMemoryException: Java Heap Space while indexing

[leilengma]

Elasticsearch version:5.0.0

Plugins installed: []

JVM version: 1.8.0_111

OS version: Windows 7

Description of the problem including expected versus actual behavior:

I am using elasticsearch to store objects which are structured as follows:

    properties: {
        date: { type: "date", format:"epoch_second"},
        id: { type: "long" },
        documents: {
            type: "nested",
            properties: {
                id: { type: "long" },
                state: {type: "keyword"},
                class: { type: "text" },
                pages: {
                    type:"nested",
                    properties: {
                        imageId: { type: "long" },
                        is_attachment: { type: "boolean" },
                        thumbnailUri: {type: "text"},
                        imageUri: {type:"text"}, 
                        fields: {
                            type:"nested",
                            properties: {
                                id: { type: "long" },
                                content: { type: "text"}
                            }
                        }
                    }
                }
            }
        }
    }

I am running a single node which has only one index with 5 shards and 0 replicas.

Settings of my node:
bootstrap.memory_lock: true
indices.memory.index_buffer_size: 512mb
indices.fielddata.cache.size: 512mb

jvm options:
-Xms2700m
-Xmx2700m

Settings of that single index:

     "index": {
        "refresh_interval": "-1",
        "number_of_shards": "5",
        "provided_name": "last_test",
        "creation_date": "1481036664335",
        "number_of_replicas": "0",
        "uuid": "UwXRs9Z1Sj2kh7Y0f6AfOg",
        "version": {
           "created": "5000099"
        }
     }

I am using elasticsearchpy client and the bulk api to index 200 documents in one request.

The problem is that while elasticsearch is indexing the documents, the jvm heap usage keeps increasing slowly until an OutOfMemory exception: Java Heap Space occurs.

I also took a look into the heap dump created by elasticsearch when that exception occurs:

and memory usage plot:

Since I am not familiar with both elasticsearch and jvm heap management, I cannot find out why this happens. Have I done somthing wrong? Or it is just due to insufficient heap size?

Thanks for any support and opinions.

Steps to reproduce:

1. Indexing documents for a while

Provide logs (if relevant):

[2016-12-28T09:04:57,470][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][34] overhead, spent [417ms] collecting in the last [1.2s]
[2016-12-28T09:04:59,509][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][36] overhead, spent [435ms] collecting in the last [1s]
[2016-12-28T09:05:01,667][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][38] overhead, spent [574ms] collecting in the last [1.1s]
[2016-12-28T09:05:03,684][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][40] overhead, spent [277ms] collecting in the last [1s]
[2016-12-28T09:05:05,686][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][42] overhead, spent [320ms] collecting in the last [1s]
[2016-12-28T09:05:07,687][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][44] overhead, spent [361ms] collecting in the last [1s]
[2016-12-28T09:05:09,689][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][46] overhead, spent [378ms] collecting in the last [1s]
[2016-12-28T09:05:11,692][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][48] overhead, spent [285ms] collecting in the last [1s]
[2016-12-28T09:05:14,989][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][51] overhead, spent [373ms] collecting in the last [1.2s]
[2016-12-28T09:05:15,991][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][52] overhead, spent [381ms] collecting in the last [1s]
[2016-12-28T09:05:17,993][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][54] overhead, spent [538ms] collecting in the last [1s]
[2016-12-28T09:05:20,096][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][56] overhead, spent [601ms] collecting in the last [1s]
[2016-12-28T09:05:22,393][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][58] overhead, spent [478ms] collecting in the last [1.2s]
[2016-12-28T09:05:24,747][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][60] overhead, spent [476ms] collecting in the last [1.3s]
[2016-12-28T09:05:25,759][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][61] overhead, spent [319ms] collecting in the last [1s]
[2016-12-28T09:05:27,769][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][63] overhead, spent [660ms] collecting in the last [1s]
[2016-12-28T09:05:29,771][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][65] overhead, spent [801ms] collecting in the last [1s]
[2016-12-28T09:05:30,772][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][66] overhead, spent [469ms] collecting in the last [1s]
[2016-12-28T09:05:32,773][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][68] overhead, spent [352ms] collecting in the last [1s]
[2016-12-28T09:05:34,836][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][70] overhead, spent [637ms] collecting in the last [1s]
[2016-12-28T09:05:36,876][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][72] overhead, spent [612ms] collecting in the last [1s]
[2016-12-28T09:05:37,877][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][73] overhead, spent [502ms] collecting in the last [1s]
[2016-12-28T09:05:39,878][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][75] overhead, spent [855ms] collecting in the last [1s]
[2016-12-28T09:05:40,901][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][76] overhead, spent [365ms] collecting in the last [1s]
[2016-12-28T09:05:44,258][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][79] overhead, spent [560ms] collecting in the last [1.1s]
[2016-12-28T09:05:45,263][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][80] overhead, spent [766ms] collecting in the last [1s]
[2016-12-28T09:05:47,321][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][82] overhead, spent [552ms] collecting in the last [1s]
[2016-12-28T09:05:48,412][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][83] overhead, spent [1s] collecting in the last [1.1s]
[2016-12-28T09:05:49,712][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][84] overhead, spent [640ms] collecting in the last [1.3s]
[2016-12-28T09:05:51,918][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][86] overhead, spent [855ms] collecting in the last [1.1s]
[2016-12-28T09:05:53,052][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][87] overhead, spent [1s] collecting in the last [1.1s]
[2016-12-28T09:05:54,077][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][88] overhead, spent [866ms] collecting in the last [1s]
[2016-12-28T09:05:55,080][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][89] overhead, spent [331ms] collecting in the last [1s]
[2016-12-28T09:05:57,158][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][91] overhead, spent [846ms] collecting in the last [1s]
[2016-12-28T09:05:58,659][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][92] overhead, spent [958ms] collecting in the last [1.4s]
[2016-12-28T09:05:59,697][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][93] overhead, spent [356ms] collecting in the last [1s]
[2016-12-28T09:06:00,920][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][94] overhead, spent [529ms] collecting in the last [1.2s]
[2016-12-28T09:06:01,922][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][95] overhead, spent [800ms] collecting in the last [1s]
[2016-12-28T09:06:03,325][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][96] overhead, spent [653ms] collecting in the last [1.2s]
[2016-12-28T09:06:05,473][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][98] overhead, spent [979ms] collecting in the last [1.1s]
[2016-12-28T09:06:06,553][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][99] overhead, spent [1s] collecting in the last [1s]
[2016-12-28T09:06:07,675][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][100] overhead, spent [791ms] collecting in the last [1s]
[2016-12-28T09:06:08,910][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][101] overhead, spent [643ms] collecting in the last [1.3s]
[2016-12-28T09:06:10,953][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][103] overhead, spent [867ms] collecting in the last [1s]
[2016-12-28T09:06:12,011][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][104] overhead, spent [1s] collecting in the last [1s]
[2016-12-28T09:06:13,335][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][105] overhead, spent [1.1s] collecting in the last [1s]
[2016-12-28T09:06:14,628][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][106] overhead, spent [870ms] collecting in the last [1.6s]
[2016-12-28T09:06:16,021][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][107] overhead, spent [1.1s] collecting in the last [1.4s]
[2016-12-28T09:06:17,438][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][108] overhead, spent [463ms] collecting in the last [1.1s]
[2016-12-28T09:06:18,603][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][109] overhead, spent [1.1s] collecting in the last [1.4s]
[2016-12-28T09:06:19,645][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][110] overhead, spent [1s] collecting in the last [1s]
[2016-12-28T09:06:21,153][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][111] overhead, spent [1.2s] collecting in the last [1.4s]
[2016-12-28T09:06:22,586][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][112] overhead, spent [1.2s] collecting in the last [1.1s]
[2016-12-28T09:06:49,673][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][114] overhead, spent [1.6s] collecting in the last [1.4s]
[2016-12-28T09:06:50,221][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] fatal error in thread [elasticsearch[node-1][bulk][T#6]], exiting
java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.(FreqProxTermsWriterPerField.java:207) ~[lucene-core-6.2.0.jar:6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:39:36]
at org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.newInstance(FreqProxTermsWriterPerField.java:230) ~[lucene-core-6.2.0.jar:6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:39:36]
at org.apache.lucene.index.ParallelPostingsArray.grow(ParallelPostingsArray.java:46) ~[lucene-core-6.2.0.jar:6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:39:36]
at org.apache.lucene.index.TermsHashPerField$PostingsBytesStartArray.grow(TermsHashPerField.java:250) ~[lucene-core-6.2.0.jar:6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:39:36]
at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:271) ~[lucene-core-6.2.0.jar:6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:39:36]
at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:149) ~[lucene-core-6.2.0.jar:6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:39:36]
at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:762) ~[lucene-core-6.2.0.jar:6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:39:36]
at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:417) ~[lucene-core-6.2.0.jar:6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:39:36]
at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:373) ~[lucene-core-6.2.0.jar:6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:39:36]
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:272) ~[lucene-core-6.2.0.jar:6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:39:36]
at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:433) ~[lucene-core-6.2.0.jar:6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:39:36]
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1375) ~[lucene-core-6.2.0.jar:6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:39:36]
at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1351) ~[lucene-core-6.2.0.jar:6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:39:36]
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:533) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:517) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:408) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:552) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:542) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:191) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:351) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.bulk.TransportShardBulkAction.index(TransportShardBulkAction.java:158) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.bulk.TransportShardBulkAction.handleItem(TransportShardBulkAction.java:137) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:123) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:74) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:78) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:50) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:902) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:872) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:319) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:254) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:838) ~[elasticsearch-5.0.0.jar:5.0.0]

[jpountz]

Please share the output of curl -XGET 'http://localhost:9200/_segments'.

[leilengma]

@jpountz, thanks for the reply.

Here is the output of curl -XGET 'http://localhost:9200/_segments':
{
"_shards": {
"total": 7,
"successful": 6,
"failed": 0
},
"indices": {
".kibana": {
"shards": {
"0": [
{
"routing": {
"state": "STARTED",
"primary": true,
"node": "WPLfUuTtTOOCBGyAAUgUPQ"
},
"num_committed_segments": 3,
"num_search_segments": 3,
"segments": {
"_s": {
"generation": 28,
"num_docs": 8,
"deleted_docs": 2,
"size_in_bytes": 17124,
"memory_in_bytes": 4356,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_u": {
"generation": 30,
"num_docs": 1,
"deleted_docs": 0,
"size_in_bytes": 6167,
"memory_in_bytes": 2552,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_v": {
"generation": 31,
"num_docs": 1,
"deleted_docs": 0,
"size_in_bytes": 5470,
"memory_in_bytes": 2411,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
}
}
}
]
}
},
"last_test": {
"shards": {
"0": [
{
"routing": {
"state": "STARTED",
"primary": true,
"node": "WPLfUuTtTOOCBGyAAUgUPQ"
},
"num_committed_segments": 36,
"num_search_segments": 36,
"segments": {
"_4c7": {
"generation": 5623,
"num_docs": 66112352,
"deleted_docs": 0,
"size_in_bytes": 4717336730,
"memory_in_bytes": 27129066,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_838": {
"generation": 10484,
"num_docs": 65939594,
"deleted_docs": 0,
"size_in_bytes": 4847003734,
"memory_in_bytes": 30023426,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_ce5": {
"generation": 16061,
"num_docs": 3253997,
"deleted_docs": 111133,
"size_in_bytes": 275843701,
"memory_in_bytes": 2519895,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_cef": {
"generation": 16071,
"num_docs": 2866403,
"deleted_docs": 0,
"size_in_bytes": 233759076,
"memory_in_bytes": 2204734,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_dt1": {
"generation": 17893,
"num_docs": 79026,
"deleted_docs": 0,
"size_in_bytes": 6647083,
"memory_in_bytes": 71075,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e3h": {
"generation": 18269,
"num_docs": 535919,
"deleted_docs": 0,
"size_in_bytes": 44362089,
"memory_in_bytes": 353447,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e3k": {
"generation": 18272,
"num_docs": 520417,
"deleted_docs": 0,
"size_in_bytes": 43048874,
"memory_in_bytes": 343827,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e3w": {
"generation": 18284,
"num_docs": 70813,
"deleted_docs": 0,
"size_in_bytes": 5596851,
"memory_in_bytes": 49575,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e41": {
"generation": 18289,
"num_docs": 102418,
"deleted_docs": 0,
"size_in_bytes": 31964616,
"memory_in_bytes": 88504,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e49": {
"generation": 18297,
"num_docs": 66047581,
"deleted_docs": 0,
"size_in_bytes": 4943250733,
"memory_in_bytes": 30431819,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_e4g": {
"generation": 18304,
"num_docs": 8724,
"deleted_docs": 0,
"size_in_bytes": 2774240,
"memory_in_bytes": 10960,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e4n": {
"generation": 18311,
"num_docs": 2569400,
"deleted_docs": 0,
"size_in_bytes": 634697146,
"memory_in_bytes": 1928224,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e4r": {
"generation": 18315,
"num_docs": 87878,
"deleted_docs": 0,
"size_in_bytes": 27690284,
"memory_in_bytes": 79418,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e4x": {
"generation": 18321,
"num_docs": 82408,
"deleted_docs": 0,
"size_in_bytes": 25953380,
"memory_in_bytes": 75470,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e4y": {
"generation": 18322,
"num_docs": 14338,
"deleted_docs": 0,
"size_in_bytes": 4652651,
"memory_in_bytes": 15958,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e51": {
"generation": 18325,
"num_docs": 9045051,
"deleted_docs": 0,
"size_in_bytes": 1169816928,
"memory_in_bytes": 5019921,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e53": {
"generation": 18327,
"num_docs": 48420,
"deleted_docs": 0,
"size_in_bytes": 15367729,
"memory_in_bytes": 43256,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e5d": {
"generation": 18337,
"num_docs": 98513,
"deleted_docs": 0,
"size_in_bytes": 30886406,
"memory_in_bytes": 84915,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e5e": {
"generation": 18338,
"num_docs": 4129878,
"deleted_docs": 0,
"size_in_bytes": 1024783466,
"memory_in_bytes": 2755551,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e5h": {
"generation": 18341,
"num_docs": 87534,
"deleted_docs": 0,
"size_in_bytes": 27594471,
"memory_in_bytes": 79081,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e5l": {
"generation": 18345,
"num_docs": 72787,
"deleted_docs": 0,
"size_in_bytes": 23027917,
"memory_in_bytes": 63567,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e5q": {
"generation": 18350,
"num_docs": 2455996,
"deleted_docs": 0,
"size_in_bytes": 613982442,
"memory_in_bytes": 1821794,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e5r": {
"generation": 18351,
"num_docs": 223254,
"deleted_docs": 0,
"size_in_bytes": 70260162,
"memory_in_bytes": 146945,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e5x": {
"generation": 18357,
"num_docs": 32936,
"deleted_docs": 0,
"size_in_bytes": 10612296,
"memory_in_bytes": 35271,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e5y": {
"generation": 18358,
"num_docs": 166526,
"deleted_docs": 0,
"size_in_bytes": 52366266,
"memory_in_bytes": 121898,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e5z": {
"generation": 18359,
"num_docs": 3470569,
"deleted_docs": 0,
"size_in_bytes": 1078374572,
"memory_in_bytes": 2351133,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e60": {
"generation": 18360,
"num_docs": 229967,
"deleted_docs": 0,
"size_in_bytes": 71872186,
"memory_in_bytes": 150171,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e61": {
"generation": 18361,
"num_docs": 331701,
"deleted_docs": 0,
"size_in_bytes": 103650663,
"memory_in_bytes": 196386,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e62": {
"generation": 18362,
"num_docs": 93505,
"deleted_docs": 0,
"size_in_bytes": 29596112,
"memory_in_bytes": 83752,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e63": {
"generation": 18363,
"num_docs": 5551,
"deleted_docs": 0,
"size_in_bytes": 1823262,
"memory_in_bytes": 8736,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e64": {
"generation": 18364,
"num_docs": 376457,
"deleted_docs": 0,
"size_in_bytes": 116557040,
"memory_in_bytes": 236723,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e65": {
"generation": 18365,
"num_docs": 380676,
"deleted_docs": 0,
"size_in_bytes": 117302649,
"memory_in_bytes": 233663,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e66": {
"generation": 18366,
"num_docs": 387163,
"deleted_docs": 0,
"size_in_bytes": 120056114,
"memory_in_bytes": 244484,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e67": {
"generation": 18367,
"num_docs": 371731,
"deleted_docs": 0,
"size_in_bytes": 114856693,
"memory_in_bytes": 228470,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e68": {
"generation": 18368,
"num_docs": 18558,
"deleted_docs": 0,
"size_in_bytes": 5965636,
"memory_in_bytes": 19497,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_e69": {
"generation": 18369,
"num_docs": 395200,
"deleted_docs": 0,
"size_in_bytes": 122435340,
"memory_in_bytes": 251988,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
}
}
}
],
"1": [
{
"routing": {
"state": "STARTED",
"primary": true,
"node": "WPLfUuTtTOOCBGyAAUgUPQ"
},
"num_committed_segments": 30,
"num_search_segments": 30,
"segments": {
"_3o1": {
"generation": 4753,
"num_docs": 68045552,
"deleted_docs": 0,
"size_in_bytes": 4846715621,
"memory_in_bytes": 27510033,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_83o": {
"generation": 10500,
"num_docs": 67119099,
"deleted_docs": 0,
"size_in_bytes": 4931250663,
"memory_in_bytes": 30765091,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_biz": {
"generation": 14939,
"num_docs": 10669877,
"deleted_docs": 0,
"size_in_bytes": 861375445,
"memory_in_bytes": 5724209,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_cw8": {
"generation": 16712,
"num_docs": 91529,
"deleted_docs": 0,
"size_in_bytes": 7682230,
"memory_in_bytes": 78242,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d3x": {
"generation": 16989,
"num_docs": 66498397,
"deleted_docs": 0,
"size_in_bytes": 5004745393,
"memory_in_bytes": 31445265,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_d4b": {
"generation": 17003,
"num_docs": 1896694,
"deleted_docs": 0,
"size_in_bytes": 589046467,
"memory_in_bytes": 1279423,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d4g": {
"generation": 17008,
"num_docs": 13169,
"deleted_docs": 0,
"size_in_bytes": 4162989,
"memory_in_bytes": 14520,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d4h": {
"generation": 17009,
"num_docs": 14542,
"deleted_docs": 0,
"size_in_bytes": 4601219,
"memory_in_bytes": 15453,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d4p": {
"generation": 17017,
"num_docs": 25150,
"deleted_docs": 0,
"size_in_bytes": 8043209,
"memory_in_bytes": 24567,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d4t": {
"generation": 17021,
"num_docs": 5146107,
"deleted_docs": 0,
"size_in_bytes": 1059782446,
"memory_in_bytes": 3244788,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d53": {
"generation": 17031,
"num_docs": 114114,
"deleted_docs": 0,
"size_in_bytes": 36031304,
"memory_in_bytes": 94590,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5b": {
"generation": 17039,
"num_docs": 2082921,
"deleted_docs": 0,
"size_in_bytes": 647058876,
"memory_in_bytes": 1454140,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5c": {
"generation": 17040,
"num_docs": 224966,
"deleted_docs": 0,
"size_in_bytes": 70470538,
"memory_in_bytes": 148386,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5d": {
"generation": 17041,
"num_docs": 184571,
"deleted_docs": 0,
"size_in_bytes": 57729866,
"memory_in_bytes": 129842,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5f": {
"generation": 17043,
"num_docs": 318605,
"deleted_docs": 0,
"size_in_bytes": 99673550,
"memory_in_bytes": 189914,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5i": {
"generation": 17046,
"num_docs": 321130,
"deleted_docs": 0,
"size_in_bytes": 100046866,
"memory_in_bytes": 191608,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5j": {
"generation": 17047,
"num_docs": 447376,
"deleted_docs": 0,
"size_in_bytes": 140485760,
"memory_in_bytes": 280963,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5n": {
"generation": 17051,
"num_docs": 389953,
"deleted_docs": 0,
"size_in_bytes": 121911306,
"memory_in_bytes": 239753,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5o": {
"generation": 17052,
"num_docs": 3617071,
"deleted_docs": 0,
"size_in_bytes": 1125390046,
"memory_in_bytes": 2425759,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5q": {
"generation": 17054,
"num_docs": 146907,
"deleted_docs": 0,
"size_in_bytes": 46298188,
"memory_in_bytes": 113578,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5r": {
"generation": 17055,
"num_docs": 273481,
"deleted_docs": 0,
"size_in_bytes": 85675867,
"memory_in_bytes": 168779,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5s": {
"generation": 17056,
"num_docs": 346974,
"deleted_docs": 0,
"size_in_bytes": 109113499,
"memory_in_bytes": 206575,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5u": {
"generation": 17058,
"num_docs": 6459,
"deleted_docs": 0,
"size_in_bytes": 2067493,
"memory_in_bytes": 9041,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5v": {
"generation": 17059,
"num_docs": 460377,
"deleted_docs": 0,
"size_in_bytes": 143217480,
"memory_in_bytes": 291530,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5w": {
"generation": 17060,
"num_docs": 379308,
"deleted_docs": 0,
"size_in_bytes": 117213242,
"memory_in_bytes": 230128,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5x": {
"generation": 17061,
"num_docs": 380813,
"deleted_docs": 0,
"size_in_bytes": 118202912,
"memory_in_bytes": 239714,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5y": {
"generation": 17062,
"num_docs": 309077,
"deleted_docs": 0,
"size_in_bytes": 95731858,
"memory_in_bytes": 180808,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d5z": {
"generation": 17063,
"num_docs": 1046079,
"deleted_docs": 0,
"size_in_bytes": 219128810,
"memory_in_bytes": 692506,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d60": {
"generation": 17064,
"num_docs": 222746,
"deleted_docs": 0,
"size_in_bytes": 69046134,
"memory_in_bytes": 141001,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_d61": {
"generation": 17065,
"num_docs": 174434,
"deleted_docs": 0,
"size_in_bytes": 54236933,
"memory_in_bytes": 120003,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
}
}
}
],
"2": [
{
"routing": {
"state": "STARTED",
"primary": true,
"node": "WPLfUuTtTOOCBGyAAUgUPQ"
},
"num_committed_segments": 29,
"num_search_segments": 29,
"segments": {
"_32e": {
"generation": 3974,
"num_docs": 67959232,
"deleted_docs": 0,
"size_in_bytes": 4842172295,
"memory_in_bytes": 27496825,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_7f3": {
"generation": 9615,
"num_docs": 67295853,
"deleted_docs": 0,
"size_in_bytes": 4922552439,
"memory_in_bytes": 30159252,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_al2": {
"generation": 13718,
"num_docs": 3431743,
"deleted_docs": 0,
"size_in_bytes": 279995891,
"memory_in_bytes": 2561468,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bvg": {
"generation": 15388,
"num_docs": 66292505,
"deleted_docs": 0,
"size_in_bytes": 4966026551,
"memory_in_bytes": 30988287,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_bvn": {
"generation": 15395,
"num_docs": 37141,
"deleted_docs": 0,
"size_in_bytes": 11783818,
"memory_in_bytes": 39272,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bvo": {
"generation": 15396,
"num_docs": 3418652,
"deleted_docs": 0,
"size_in_bytes": 589815164,
"memory_in_bytes": 2470765,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bw1": {
"generation": 15409,
"num_docs": 2324992,
"deleted_docs": 0,
"size_in_bytes": 605097779,
"memory_in_bytes": 1700994,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bw2": {
"generation": 15410,
"num_docs": 12720,
"deleted_docs": 0,
"size_in_bytes": 4093683,
"memory_in_bytes": 14581,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bw9": {
"generation": 15417,
"num_docs": 13388,
"deleted_docs": 0,
"size_in_bytes": 4329124,
"memory_in_bytes": 14974,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bwc": {
"generation": 15420,
"num_docs": 8431050,
"deleted_docs": 0,
"size_in_bytes": 1213272955,
"memory_in_bytes": 4791327,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bwp": {
"generation": 15433,
"num_docs": 2753774,
"deleted_docs": 0,
"size_in_bytes": 855409401,
"memory_in_bytes": 1909645,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bwr": {
"generation": 15435,
"num_docs": 192553,
"deleted_docs": 0,
"size_in_bytes": 60334826,
"memory_in_bytes": 133260,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bwu": {
"generation": 15438,
"num_docs": 226375,
"deleted_docs": 0,
"size_in_bytes": 70717617,
"memory_in_bytes": 147890,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bwy": {
"generation": 15442,
"num_docs": 272235,
"deleted_docs": 0,
"size_in_bytes": 85322520,
"memory_in_bytes": 167530,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bx1": {
"generation": 15445,
"num_docs": 1513356,
"deleted_docs": 0,
"size_in_bytes": 264262043,
"memory_in_bytes": 1063982,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bx3": {
"generation": 15447,
"num_docs": 19683,
"deleted_docs": 0,
"size_in_bytes": 6255722,
"memory_in_bytes": 20426,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bx4": {
"generation": 15448,
"num_docs": 373797,
"deleted_docs": 0,
"size_in_bytes": 117480943,
"memory_in_bytes": 226009,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bx5": {
"generation": 15449,
"num_docs": 384200,
"deleted_docs": 0,
"size_in_bytes": 120445965,
"memory_in_bytes": 238059,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bx6": {
"generation": 15450,
"num_docs": 376871,
"deleted_docs": 0,
"size_in_bytes": 117597558,
"memory_in_bytes": 226304,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bx9": {
"generation": 15453,
"num_docs": 74344,
"deleted_docs": 0,
"size_in_bytes": 23645070,
"memory_in_bytes": 65537,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bxb": {
"generation": 15455,
"num_docs": 162864,
"deleted_docs": 0,
"size_in_bytes": 20232738,
"memory_in_bytes": 121024,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bxc": {
"generation": 15456,
"num_docs": 189963,
"deleted_docs": 0,
"size_in_bytes": 59794801,
"memory_in_bytes": 133366,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bxd": {
"generation": 15457,
"num_docs": 324748,
"deleted_docs": 0,
"size_in_bytes": 101081136,
"memory_in_bytes": 191511,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bxf": {
"generation": 15459,
"num_docs": 307572,
"deleted_docs": 0,
"size_in_bytes": 95762366,
"memory_in_bytes": 179140,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bxg": {
"generation": 15460,
"num_docs": 5867,
"deleted_docs": 0,
"size_in_bytes": 1858692,
"memory_in_bytes": 8464,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bxk": {
"generation": 15464,
"num_docs": 86834,
"deleted_docs": 0,
"size_in_bytes": 26882902,
"memory_in_bytes": 74845,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bxl": {
"generation": 15465,
"num_docs": 3533499,
"deleted_docs": 0,
"size_in_bytes": 1094646823,
"memory_in_bytes": 2399533,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bxm": {
"generation": 15466,
"num_docs": 267150,
"deleted_docs": 0,
"size_in_bytes": 82807741,
"memory_in_bytes": 157797,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_bxn": {
"generation": 15467,
"num_docs": 126408,
"deleted_docs": 0,
"size_in_bytes": 39580906,
"memory_in_bytes": 95077,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
}
}
}
],
"3": [
{
"routing": {
"state": "STARTED",
"primary": true,
"node": "WPLfUuTtTOOCBGyAAUgUPQ"
},
"num_committed_segments": 36,
"num_search_segments": 36,
"segments": {
"_30p": {
"generation": 3913,
"num_docs": 69022652,
"deleted_docs": 0,
"size_in_bytes": 4911122624,
"memory_in_bytes": 27725423,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_6xp": {
"generation": 8989,
"num_docs": 67269775,
"deleted_docs": 0,
"size_in_bytes": 4917615812,
"memory_in_bytes": 30033032,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_amu": {
"generation": 13782,
"num_docs": 91315,
"deleted_docs": 0,
"size_in_bytes": 7668102,
"memory_in_bytes": 77447,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_arj": {
"generation": 13951,
"num_docs": 37200,
"deleted_docs": 0,
"size_in_bytes": 3143751,
"memory_in_bytes": 31931,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_axf": {
"generation": 14163,
"num_docs": 85654,
"deleted_docs": 0,
"size_in_bytes": 7195246,
"memory_in_bytes": 75161,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b3m": {
"generation": 14386,
"num_docs": 45097,
"deleted_docs": 0,
"size_in_bytes": 3814476,
"memory_in_bytes": 38292,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b3u": {
"generation": 14394,
"num_docs": 5939,
"deleted_docs": 0,
"size_in_bytes": 1895733,
"memory_in_bytes": 8657,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b3z": {
"generation": 14399,
"num_docs": 67020565,
"deleted_docs": 0,
"size_in_bytes": 4969774699,
"memory_in_bytes": 31145243,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_b46": {
"generation": 14406,
"num_docs": 3161753,
"deleted_docs": 0,
"size_in_bytes": 546605764,
"memory_in_bytes": 2336367,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b47": {
"generation": 14407,
"num_docs": 8580,
"deleted_docs": 0,
"size_in_bytes": 2701094,
"memory_in_bytes": 10763,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b4i": {
"generation": 14418,
"num_docs": 2029807,
"deleted_docs": 0,
"size_in_bytes": 631356323,
"memory_in_bytes": 1411405,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b4p": {
"generation": 14425,
"num_docs": 4743,
"deleted_docs": 0,
"size_in_bytes": 1491271,
"memory_in_bytes": 7757,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b4t": {
"generation": 14429,
"num_docs": 7238,
"deleted_docs": 0,
"size_in_bytes": 2283498,
"memory_in_bytes": 9650,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b4u": {
"generation": 14430,
"num_docs": 5430476,
"deleted_docs": 0,
"size_in_bytes": 1078900594,
"memory_in_bytes": 3424256,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b54": {
"generation": 14440,
"num_docs": 118794,
"deleted_docs": 0,
"size_in_bytes": 37314594,
"memory_in_bytes": 95064,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b56": {
"generation": 14442,
"num_docs": 205840,
"deleted_docs": 0,
"size_in_bytes": 64661928,
"memory_in_bytes": 140073,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b58": {
"generation": 14444,
"num_docs": 89714,
"deleted_docs": 0,
"size_in_bytes": 28508835,
"memory_in_bytes": 80741,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b5a": {
"generation": 14446,
"num_docs": 219564,
"deleted_docs": 0,
"size_in_bytes": 68838917,
"memory_in_bytes": 145857,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b5c": {
"generation": 14448,
"num_docs": 146573,
"deleted_docs": 0,
"size_in_bytes": 45864126,
"memory_in_bytes": 113351,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b5d": {
"generation": 14449,
"num_docs": 70149,
"deleted_docs": 0,
"size_in_bytes": 22193285,
"memory_in_bytes": 62692,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b5i": {
"generation": 14454,
"num_docs": 91460,
"deleted_docs": 0,
"size_in_bytes": 28683041,
"memory_in_bytes": 82208,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b5j": {
"generation": 14455,
"num_docs": 3430680,
"deleted_docs": 0,
"size_in_bytes": 1065082272,
"memory_in_bytes": 2332652,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b5k": {
"generation": 14456,
"num_docs": 275681,
"deleted_docs": 0,
"size_in_bytes": 86077284,
"memory_in_bytes": 170013,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b5p": {
"generation": 14461,
"num_docs": 208519,
"deleted_docs": 0,
"size_in_bytes": 65444858,
"memory_in_bytes": 140920,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b5q": {
"generation": 14462,
"num_docs": 26207,
"deleted_docs": 0,
"size_in_bytes": 8369801,
"memory_in_bytes": 25806,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b5r": {
"generation": 14463,
"num_docs": 205913,
"deleted_docs": 0,
"size_in_bytes": 64121402,
"memory_in_bytes": 137913,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b5w": {
"generation": 14468,
"num_docs": 279280,
"deleted_docs": 0,
"size_in_bytes": 87270525,
"memory_in_bytes": 171181,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b5x": {
"generation": 14469,
"num_docs": 9119621,
"deleted_docs": 0,
"size_in_bytes": 1332683690,
"memory_in_bytes": 5017016,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b5y": {
"generation": 14470,
"num_docs": 5829,
"deleted_docs": 0,
"size_in_bytes": 1814784,
"memory_in_bytes": 8664,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b5z": {
"generation": 14471,
"num_docs": 228485,
"deleted_docs": 0,
"size_in_bytes": 71145657,
"memory_in_bytes": 142343,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b60": {
"generation": 14472,
"num_docs": 342916,
"deleted_docs": 0,
"size_in_bytes": 105979128,
"memory_in_bytes": 204802,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b61": {
"generation": 14473,
"num_docs": 368481,
"deleted_docs": 0,
"size_in_bytes": 114213896,
"memory_in_bytes": 223131,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b62": {
"generation": 14474,
"num_docs": 363859,
"deleted_docs": 0,
"size_in_bytes": 112573920,
"memory_in_bytes": 219411,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b63": {
"generation": 14475,
"num_docs": 259428,
"deleted_docs": 0,
"size_in_bytes": 80051108,
"memory_in_bytes": 153825,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b64": {
"generation": 14476,
"num_docs": 385307,
"deleted_docs": 0,
"size_in_bytes": 119904371,
"memory_in_bytes": 244613,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_b65": {
"generation": 14477,
"num_docs": 13107,
"deleted_docs": 0,
"size_in_bytes": 4140673,
"memory_in_bytes": 14820,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
}
}
}
],
"4": [
{
"routing": {
"state": "STARTED",
"primary": true,
"node": "WPLfUuTtTOOCBGyAAUgUPQ"
},
"num_committed_segments": 37,
"num_search_segments": 37,
"segments": {
"_2in": {
"generation": 3263,
"num_docs": 68175655,
"deleted_docs": 0,
"size_in_bytes": 4854470960,
"memory_in_bytes": 27514238,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_61w": {
"generation": 7844,
"num_docs": 67146680,
"deleted_docs": 0,
"size_in_bytes": 4929018725,
"memory_in_bytes": 30693346,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_9qm": {
"generation": 12622,
"num_docs": 1994978,
"deleted_docs": 0,
"size_in_bytes": 161824370,
"memory_in_bytes": 1569131,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_9w6": {
"generation": 12822,
"num_docs": 113097,
"deleted_docs": 0,
"size_in_bytes": 9483483,
"memory_in_bytes": 89492,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_a0v": {
"generation": 12991,
"num_docs": 43469,
"deleted_docs": 0,
"size_in_bytes": 3682409,
"memory_in_bytes": 36628,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_a1h": {
"generation": 13013,
"num_docs": 1971690,
"deleted_docs": 0,
"size_in_bytes": 159922895,
"memory_in_bytes": 1546091,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_abk": {
"generation": 13376,
"num_docs": 24853,
"deleted_docs": 0,
"size_in_bytes": 7925503,
"memory_in_bytes": 24283,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_abv": {
"generation": 13387,
"num_docs": 66981229,
"deleted_docs": 0,
"size_in_bytes": 4952775166,
"memory_in_bytes": 30583481,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": false
},
"_abw": {
"generation": 13388,
"num_docs": 2085986,
"deleted_docs": 0,
"size_in_bytes": 170652310,
"memory_in_bytes": 1655551,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_ac2": {
"generation": 13394,
"num_docs": 11569,
"deleted_docs": 0,
"size_in_bytes": 3744138,
"memory_in_bytes": 13619,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_ac7": {
"generation": 13399,
"num_docs": 2890952,
"deleted_docs": 0,
"size_in_bytes": 591417542,
"memory_in_bytes": 2151761,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_ace": {
"generation": 13406,
"num_docs": 5697,
"deleted_docs": 0,
"size_in_bytes": 1876954,
"memory_in_bytes": 8622,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_acf": {
"generation": 13407,
"num_docs": 21535,
"deleted_docs": 0,
"size_in_bytes": 6894589,
"memory_in_bytes": 21310,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_acm": {
"generation": 13414,
"num_docs": 12348,
"deleted_docs": 0,
"size_in_bytes": 3930733,
"memory_in_bytes": 14092,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_aco": {
"generation": 13416,
"num_docs": 5099933,
"deleted_docs": 0,
"size_in_bytes": 993995159,
"memory_in_bytes": 3259076,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_acr": {
"generation": 13419,
"num_docs": 74635,
"deleted_docs": 0,
"size_in_bytes": 23617022,
"memory_in_bytes": 65378,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_acs": {
"generation": 13420,
"num_docs": 59230,
"deleted_docs": 0,
"size_in_bytes": 18941516,
"memory_in_bytes": 56117,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_acy": {
"generation": 13426,
"num_docs": 2108598,
"deleted_docs": 0,
"size_in_bytes": 539022905,
"memory_in_bytes": 1507225,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_ad1": {
"generation": 13429,
"num_docs": 108907,
"deleted_docs": 0,
"size_in_bytes": 34251170,
"memory_in_bytes": 91562,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_ad4": {
"generation": 13432,
"num_docs": 81007,
"deleted_docs": 0,
"size_in_bytes": 25380001,
"memory_in_bytes": 73489,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_ad8": {
"generation": 13436,
"num_docs": 25796,
"deleted_docs": 0,
"size_in_bytes": 8290919,
"memory_in_bytes": 25336,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_ad9": {
"generation": 13437,
"num_docs": 76741,
"deleted_docs": 0,
"size_in_bytes": 24330237,
"memory_in_bytes": 66105,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_adc": {
"generation": 13440,
"num_docs": 5198679,
"deleted_docs": 0,
"size_in_bytes": 1227536202,
"memory_in_bytes": 3239470,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_ade": {
"generation": 13442,
"num_docs": 162829,
"deleted_docs": 0,
"size_in_bytes": 50904268,
"memory_in_bytes": 121628,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_adk": {
"generation": 13448,
"num_docs": 166833,
"deleted_docs": 0,
"size_in_bytes": 52079808,
"memory_in_bytes": 120800,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_adl": {
"generation": 13449,
"num_docs": 20507,
"deleted_docs": 0,
"size_in_bytes": 6534291,
"memory_in_bytes": 20888,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_adm": {
"generation": 13450,
"num_docs": 3151203,
"deleted_docs": 0,
"size_in_bytes": 978444540,
"memory_in_bytes": 2142560,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_adn": {
"generation": 13451,
"num_docs": 166225,
"deleted_docs": 0,
"size_in_bytes": 51964738,
"memory_in_bytes": 121755,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_ado": {
"generation": 13452,
"num_docs": 375640,
"deleted_docs": 0,
"size_in_bytes": 117766237,
"memory_in_bytes": 228471,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_adp": {
"generation": 13453,
"num_docs": 269399,
"deleted_docs": 0,
"size_in_bytes": 83912884,
"memory_in_bytes": 167055,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_adq": {
"generation": 13454,
"num_docs": 270292,
"deleted_docs": 0,
"size_in_bytes": 84167266,
"memory_in_bytes": 158950,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_adr": {
"generation": 13455,
"num_docs": 349176,
"deleted_docs": 0,
"size_in_bytes": 108136153,
"memory_in_bytes": 209516,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_ads": {
"generation": 13456,
"num_docs": 366497,
"deleted_docs": 0,
"size_in_bytes": 113349303,
"memory_in_bytes": 221296,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_adt": {
"generation": 13457,
"num_docs": 373118,
"deleted_docs": 0,
"size_in_bytes": 115642151,
"memory_in_bytes": 230812,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_adu": {
"generation": 13458,
"num_docs": 151652,
"deleted_docs": 0,
"size_in_bytes": 47099880,
"memory_in_bytes": 112347,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_adv": {
"generation": 13459,
"num_docs": 328670,
"deleted_docs": 0,
"size_in_bytes": 101633877,
"memory_in_bytes": 194337,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
},
"_adw": {
"generation": 13460,
"num_docs": 67057,
"deleted_docs": 0,
"size_in_bytes": 21105613,
"memory_in_bytes": 60473,
"committed": true,
"search": true,
"version": "6.2.0",
"compound": true
}
}
}
]
}
}
}
}

[jpountz]

You have about 1.2 billion Lucene documents (which includes the top-level documents plus those that are created implicitly due to the use of the nested type in the mapping) on your node and 3 levels of nesting, so I would expect about 3 * 1.2B / 8 = 429MB of memory used by the bitset filter cache (which keeps track of the hierarchy of nesting). This is consistent with what heap analysis reports.

I think you should look into either increasing the memory you give to your node (but do not go beyond half the memory of your host) or fix your mappings to use nesting less.

[leilengma]

Thanks for the answer.

Just one more question. Is it normal to have so many instances of io.netty.util.Recycler$WeakOrderQueue$Link in heap?

25%-35% of the heap is filled with these instances each time when OOM occurs.

[jpountz]

That part is less clear to me since it is allocated by Netty, a dependency that we use for networking. Pinging @jasontedor in case that rings a bell to him.

[leilengma]

Thanks for the feedback.

I searched a little bit in Netty community with respect to this issue.

According to issues 3166, 4147 and 5902, it is possible for recyclers in netty to grow and then causes OOM exceptions.

Some one mentioned in 5902 mentioned that the netty recycler could be disabled in version 4.0.31.Final by adding the following settings in jvm options:
-Dio.netty.recycler.maxCapacity.default=0

I tried that setting, but the same OOM still occurred. The difference is that the heap is now filled with

io.netty.util.Recycler$WeakOrderQueue$Link and io.netty.util.Recycler$Stack instances.

Just in order to comfirm whether that netty option mentioned above would make any difference, I would like to ask which exact version of netty 4.0 is applied in Elasticsearch 5.0.0?

[jpountz]

It is 4.1.5.Final https://github.com/elastic/elasticsearch/blob/v5.0.0/modules/transport-netty4/build.gradle#L36

Likely related to #22406.

[jasontedor]

Sorry, I didn't see this one and I missed your ping @jpountz since I was out on vacation last week but I just saw it referred to #22406.

Is it normal to have so many instances of io.netty.util.Recycler$WeakOrderQueue$Link in heap?

No, I think something is broken in Netty here. I think that objects are not getting recycled quickly enough, and I think there are cases where it can grow larger than it should. You can try disabling the recycler with -Dio.netty.recycler.maxCapacityPerThread=0, the setting -Dio.netty.recycler.maxCapacity.default=0 does not do anything.

[leilengma]

@jasontedor Thanks for the feedback.
I tested my node this morning for 2 hours after disabling the recycler and everything works fine. The performance of the node did not drop and was even better.
So disabling the recyler solved the problem for me.
Thanks again.