Data node with spinning disk crashes (OutOfMemoryError)

[redlus]

Hi everyone.

We're experiencing a critical production issue when trying to utilize cold-nodes architecture (moving indices from SSD disk nodes to HDD disk nodes). The following the is the full report:

Elasticsearch version (bin/elasticsearch --version):
5.2.2

Plugins installed: []
ingest-attachment
ingest-geoip
mapper-murmur3
mapper-size
repository-azure
repository-gcs
repository-s3
x-pack

JVM version (java -version):
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux 4.4.0-47-generic #68-Ubuntu SMP 2016 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Our cluster has fast, local SSD disks nodes for indexing new data, and slower spinning SSD disks for older data. esdatacold nodes specification:
8 cores
32GB of ram (16GB allocated for es heap)
10TB of disks (5 spinning disks, 2TB each formatted using GPT and ext4)
~700 shards / 3TB of data used on each node

Older indices are backed-up using a snapshot to cold storage, and then we change their settings:

{
    "index.routing.allocation.include._name": "esdatacold*",
    "index.routing.allocation.total_shards_per_node": "-1"
}

Additionally, to minimize the amount of work each esdatacold handling consecutively, we've reduced cluster.routing.allocation.node_concurrent_recoveries to 1 (moving only 1 shard per node into the esdatacold nodes).
However, the elasticsearch process in these nodes crashes with an OutOfMemory exception. Looking at the stacktrace, it seems that all threads are waiting for a lock to be released. This is the full stacktrace of the process: https://pastebin.com/6q8GtSeg

I can understand if an overloaded node (e.g. high number of shards per GB heap or something similar) performs poorly, but a crash has severe performance effects on the whole cluster, and this should definitely be protected against.

Is there anything we can do to prevent this crash?
Is there any memory configuration we're missing?

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

None yet. This is a production-scale issue, so it is not easy to reproduce.

I'd be happy to provide additional details, whatever is needed.

Thanks!

[albertzaharovits]

@elastic/es-distributed

[tlrx]

This looks like another Netty's recycler issue, but it should be disabled in 5.2.2 thanks to #22452.

@redlus Can you please provide the JVM options for the node esdatacold2-prod? They are printed out in the logs at startup time (something like [INFO ][o.e.n.Node] [node-0] JVM arguments [...] ) or you can list them using curl -XGET 'http://localhost:9200/_nodes/node-0/jvm?pretty'

Also, a heap dump would help to identify the objects that fill the memory.

elasticsearch[esdatacold2-prod][[transport_server_worker.default]][T#14] [DAEMON] RUNNABLE tid: 60
java.lang.OutOfMemoryError.<init>() OutOfMemoryError.java:48
io.netty.buffer.PoolArena$HeapArena.newUnpooledChunk(int) PoolArena.java:661

[jasontedor]

@tlrx The JVM arguments are not logged at startup nor available in the API in 5.2.2, that is only since 5.4.1 (#24450, #24451). Here, to have the JVM arguments, the user needs to shell into an instance running Elasticsearch and execute jps -l -m -v and share the output.

[tlrx]

Thanks @jasontedor. I thought it was older but I admit I didn't double checked.

[farin99]

We will upload the full memory dump (17 GB, 6GB compressed), until then here is an export form yourKit of the objects by class. Hope it will help.
Objects-by-class.csv.zip

[redlus]

@tlrx here are the JVM options, along with GC/memory pools if relevant:

{
  "jvm": {
    "pid": 77144,
    "version": "1.8.0_151",
    "vm_name": "OpenJDK 64-Bit Server VM",
    "vm_version": "25.151-b12",
    "vm_vendor": "Oracle Corporation",
    "start_time_in_millis": 1519136454276,
    "mem": {
      "heap_init_in_bytes": 16106127360,
      "heap_max_in_bytes": 16036397056,
      "non_heap_init_in_bytes": 2555904,
      "non_heap_max_in_bytes": 0,
      "direct_max_in_bytes": 16036397056
    },
    "gc_collectors": [
      "ParNew",
      "ConcurrentMarkSweep"
    ],
    "memory_pools": [
      "Code Cache",
      "Metaspace",
      "Compressed Class Space",
      "Par Eden Space",
      "Par Survivor Space",
      "CMS Old Gen"
    ],
    "using_compressed_ordinary_object_pointers": "true"
  }
}

The heap dump is pretty large, but I'll upload it soon.

[tlrx]

We will upload the full memory dump (17 GB, 6GB compressed), until then here is an export form yourKit of the objects by class. Hope it will help.

@farin99 @redlus A full memory dump might contain sensitive data so I'm not sure you want to make it public. But an overview of the biggest objects can help.

[tlrx]

Thanks @redlus. Can you apply Jason's comment and provide the output please?

[redlus]

@tlrx for sure..:
77144 org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -XX:MaxGCPauseMillis=800 -Des.path.home=/usr/share/elasticsearch

[redlus]

@tlrx also, does the .csv file @farin99 uploaded contain the object requested?
e.g.

[jasontedor]

@redlus You have to run this as the same user that runs the Elasticsearch process and on a machine where one of the instances in your cluster is running.

[farin99]

Here is an overview of the biggest objects
Objects-with-biggest-retained-size.csv.zip