Elasticsearch 2.2 fails to start with jna tmp dir configured (core dump)

[fxh]

Elasticsearch version 2.2

JVM version: 1.7.0_75

OS version: CentOS release 6.7

Description of the problem including expected versus actual behavior:
Starting elasticsearch fails to start if the following is added to the /etc/sysconfig/elasticsearch file:

# Additional Java OPTS
ES_JAVA_OPTS="-Djna.tmpdir=/usr/share/elasticsearch/tmp -Djava.io.tmpdir=/usr/share/elasticsearch/tmp"

With this line commented out, ES starts but JNA is disabled as the default /tmp/ directory is mounted with noexec. If /tmp/ is mounted without noexec all works as expected.
/usr/share/elasticsearch/tmp exists, belongs to the right user/group and has permissions 755.

Steps to reproduce:

[root@storagenode01 ~]# service elasticsearch start
Starting elasticsearch: /usr/share/elasticsearch/bin/elasticsearch: line 155: 28878 Aborted                 (core dumped) exec "$JAVA" $JAVA_OPTS $ES_JAVA_OPTS -Des.path.home="$ES_HOME" -cp "$ES_CLASSPATH" org.elasticsearch.bootstrap.Elasticsearch start "$@" 0>&-
                                                           [FAILED]

Provide logs (if relevant):

[root@storagenode01 ~]# cat /tmp/hs_err_pid28216.log
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f424226f40a, pid=28216, tid=139922878629632
#
# JRE version: Java(TM) SE Runtime Environment (7.0_75-b13) (build 1.7.0_75-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.75-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [jna4948368637624641726.tmp+0x1240a]  ffi_prep_closure_loc+0x1a
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x00007f424c00a000):  JavaThread "main" [_thread_in_native, id=28218, stack(0x00007f42556a0000,0x00007f42557a1000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000000000000

Registers:
RAX=0x00007f424226f8c2, RBX=0x00007f425579ed48, RCX=0x00007f424c44a7b0, RDX=0x00007f4242264590
RSP=0x00007f425579eae0, RBP=0x00007f425579eae0, RSI=0x00007f424c44a7d0, RDI=0x0000000000000000
R8 =0x00007f425007ae43, R9 =0x0000000000000002, R10=0x00007f425579e870, R11=0x00007f424226f3f0
R12=0x0000000000000000, R13=0x0000000000000008, R14=0x00007f424c44a7b0, R15=0x0000000000000004
RIP=0x00007f424226f40a, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000006
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f425579eae0)
0x00007f425579eae0:   00007f425579ebb0 00007f424226a4dd
0x00007f425579eaf0:   0000000000000005 00007f425579ed38
0x00007f425579eb00:   00007f425579ed50 00007f424c00a1e8
0x00007f425579eb10:   0000000100000000 00007f424c44a940
0x00007f425579eb20:   00007f424c450f70 00007f424c44a7d0
0x00007f425579eb30:   000000015579eca0 00007f424c44a980
0x00007f425579eb40:   00007f424c44a960 00007f424c44a790
0x00007f425579eb50:   0000000000000001 000000035579ed48
0x00007f425579eb60:   00007f425579ed40 00007f425579ed38
0x00007f425579eb70:   00007f4200000000 00007f425007ae43
0x00007f425579eb80:   00000000fb599510 00000000fb599510
0x00007f425579eb90:   0000000000000000 00000000fb599510
0x00007f425579eba0:   00007f425579ed50 00007f424c00a000
0x00007f425579ebb0:   00007f425579eca0 00007f425006fd98
0x00007f425579ebc0:   00007f425579ed30 00007f425579ed28
0x00007f425579ebd0:   0000000000000002 00007f4242271eb0
0x00007f425579ebe0:   00007f4242271eb0 00007f425579ecf8
0x00007f425579ebf0:   00007f4254c43f70 0000000000000000
0x00007f425579ec00:   0000000000000000 00007f425579ecd0
0x00007f425579ec10:   0000000000000000 00007f425579ecc0
0x00007f425579ec20:   00007f425579eda8 00007f425579ecc0
0x00007f425579ec30:   00007f424c00a998 00007f4243a48f90
0x00007f425579ec40:   0000000900000010 0000000c00000024
0x00007f425579ec50:   00007f424c00a000 00007f42000000b6
0x00007f425579ec60:   00007f425579ec60 00000000fb599510
0x00007f425579ec70:   00007f425579ed50 00000000fb59f350
0x00007f425579ec80:   0000000000000000 00000000fb599510
0x00007f425579ec90:   0000000000000000 00007f425579ecc0
0x00007f425579eca0:   00007f425579eda8 00007f4250063175
0x00007f425579ecb0:   00000000bbc87fb8 00007f425006bcd7
0x00007f425579ecc0:   00000000bae0d660 0000000000000000
0x00007f425579ecd0:   00000000bbd64530 0000000000000000

Instructions: (pc=0x00007f424226f40a)
0x00007f424226f3ea:   66 90 66 66 66 90 8b 06 55 41 b9 02 00 00 00 48
0x00007f424226f3fa:   89 e5 ff c8 83 f8 01 77 44 48 8b 05 c6 49 10 00
0x00007f424226f40a:   66 c7 07 49 bb 4c 89 47 0c 66 c7 47 0a 49 ba 48
0x00007f424226f41a:   89 47 02 8b 46 1c 48 89 77 18 48 89 57 20 48 89

Register to memory mapping:

RAX=0x00007f424226f8c2: ffi_closure_unix64+0 in /usr/share/elasticsearch/tmp/jna4948368637624641726.tmp at 0x00007f424225d000
RBX=0x00007f425579ed48 is pointing into the stack for thread: 0x00007f424c00a000
RCX=0x00007f424c44a7b0 is an unknown value
RDX=0x00007f4242264590: <offset 0x7590> in /usr/share/elasticsearch/tmp/jna4948368637624641726.tmp at 0x00007f424225d000
RSP=0x00007f425579eae0 is pointing into the stack for thread: 0x00007f424c00a000
RBP=0x00007f425579eae0 is pointing into the stack for thread: 0x00007f424c00a000
RSI=0x00007f424c44a7d0 is an unknown value
RDI=0x0000000000000000 is an unknown value
R8 =0x00007f425007ae43 is at code_begin+99 in an Interpreter codelet
invokestatic  184 invokestatic  [0x00007f425007ade0, 0x00007f425007af20]  320 bytes
R9 =0x0000000000000002 is an unknown value
R10=0x00007f425579e870 is pointing into the stack for thread: 0x00007f424c00a000
R11=0x00007f424226f3f0: ffi_prep_closure_loc+0 in /usr/share/elasticsearch/tmp/jna4948368637624641726.tmp at 0x00007f424225d000
R12=0x0000000000000000 is an unknown value
R13=0x0000000000000008 is an unknown value
R14=0x00007f424c44a7b0 is an unknown value
R15=0x0000000000000004 is an unknown value


Stack: [0x00007f42556a0000,0x00007f42557a1000],  sp=0x00007f425579eae0,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [jna4948368637624641726.tmp+0x1240a]  ffi_prep_closure_loc+0x1a
C  [jna4948368637624641726.tmp+0xd4dd]  Java_com_sun_jna_Native_registerMethod+0x45d
j  com.sun.jna.Native.registerMethod(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/String;[I[J[JIJJLjava/lang/Class;JIZ[Lcom/sun/jna/ToNativeConverter;Lcom/sun/jna/FromNativeConverter;Ljava/lang/String;)J+0
j  com.sun.jna.Native.register(Ljava/lang/Class;Lcom/sun/jna/NativeLibrary;)V+979
j  com.sun.jna.Native.register(Ljava/lang/Class;Ljava/lang/String;)V+28
j  com.sun.jna.Native.register(Ljava/lang/String;)V+7
j  org.elasticsearch.bootstrap.JNACLibrary.<clinit>()V+45
v  ~StubRoutines::call_stub
V  [libjvm.so+0x600045]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x365
V  [libjvm.so+0x5feaa8]  JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x28
V  [libjvm.so+0x5c2f1a]  instanceKlass::call_class_initializer(Thread*)+0xca
V  [libjvm.so+0x5c3174]  instanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x234
V  [libjvm.so+0x5c35ca]  instanceKlass::initialize(Thread*)+0x6a
V  [libjvm.so+0x74454e]  LinkResolver::resolve_static_call(CallInfo&, KlassHandle&, Symbol*, Symbol*, KlassHandle, bool, bool, Thread*)+0x11e
V  [libjvm.so+0x7446b6]  LinkResolver::resolve_invokestatic(CallInfo&, constantPoolHandle, int, Thread*)+0xe6
V  [libjvm.so+0x5f846f]  InterpreterRuntime::resolve_invoke(JavaThread*, Bytecodes::Code)+0x17f
j  org.elasticsearch.bootstrap.JNANatives.definitelyRunningAsRoot()Z+8
j  org.elasticsearch.bootstrap.Natives.definitelyRunningAsRoot()Z+22
j  org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Ljava/nio/file/Path;ZZZ)V+8
j  org.elasticsearch.bootstrap.Bootstrap.setup(ZLorg/elasticsearch/common/settings/Settings;Lorg/elasticsearch/env/Environment;)V+43
j  org.elasticsearch.bootstrap.Bootstrap.init([Ljava/lang/String;)V+238
j  org.elasticsearch.bootstrap.Elasticsearch.main([Ljava/lang/String;)V+1
v  ~StubRoutines::call_stub
V  [libjvm.so+0x600045]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x365
V  [libjvm.so+0x5feaa8]  JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x28
V  [libjvm.so+0x638a59]  jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)+0x219
V  [libjvm.so+0x641672]  jni_CallStaticVoidMethod+0x162
C  [libjli.so+0x36d9]  JavaMain+0x7e9

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  com.sun.jna.Native.registerMethod(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/String;[I[J[JIJJLjava/lang/Class;JIZ[Lcom/sun/jna/ToNativeConverter;Lcom/sun/jna/FromNativeConverter;Ljava/lang/String;)J+0
j  com.sun.jna.Native.register(Ljava/lang/Class;Lcom/sun/jna/NativeLibrary;)V+979
j  com.sun.jna.Native.register(Ljava/lang/Class;Ljava/lang/String;)V+28
j  com.sun.jna.Native.register(Ljava/lang/String;)V+7
j  org.elasticsearch.bootstrap.JNACLibrary.<clinit>()V+45
v  ~StubRoutines::call_stub
j  org.elasticsearch.bootstrap.JNANatives.definitelyRunningAsRoot()Z+8
j  org.elasticsearch.bootstrap.Natives.definitelyRunningAsRoot()Z+22
j  org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Ljava/nio/file/Path;ZZZ)V+8
j  org.elasticsearch.bootstrap.Bootstrap.setup(ZLorg/elasticsearch/common/settings/Settings;Lorg/elasticsearch/env/Environment;)V+43
j  org.elasticsearch.bootstrap.Bootstrap.init([Ljava/lang/String;)V+238
j  org.elasticsearch.bootstrap.Elasticsearch.main([Ljava/lang/String;)V+1
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00007f424c11e000 JavaThread "Service Thread" daemon [_thread_blocked, id=28231, stack(0x00007f4242f6c000,0x00007f424306d000)]
  0x00007f424c11b000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=28230, stack(0x00007f424306d000,0x00007f424316e000)]
  0x00007f424c119000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=28229, stack(0x00007f424316e000,0x00007f424326f000)]
  0x00007f424c116800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=28228, stack(0x00007f424326f000,0x00007f4243370000)]
  0x00007f424c114800 JavaThread "Surrogate Locker Thread (Concurrent GC)" daemon [_thread_blocked, id=28227, stack(0x00007f4243370000,0x00007f4243471000)]
  0x00007f424c0eb800 JavaThread "Finalizer" daemon [_thread_blocked, id=28226, stack(0x00007f4243471000,0x00007f4243572000)]
  0x00007f424c0e9800 JavaThread "Reference Handler" daemon [_thread_blocked, id=28225, stack(0x00007f4243572000,0x00007f4243673000)]
=>0x00007f424c00a000 JavaThread "main" [_thread_in_native, id=28218, stack(0x00007f42556a0000,0x00007f42557a1000)]

Other Threads:
  0x00007f424c0e5000 VMThread [stack: 0x00007f4243673000,0x00007f4243774000] [id=28224]
  0x00007f424c129000 WatcherThread [stack: 0x00007f4242e6b000,0x00007f4242f6c000] [id=28232]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 par new generation   total 78656K, used 16826K [0x00000000bae00000, 0x00000000c0350000, 0x00000000cfac0000)
  eden space 69952K,  24% used [0x00000000bae00000, 0x00000000bbe6e950, 0x00000000bf250000)
  from space 8704K,   0% used [0x00000000bf250000, 0x00000000bf250000, 0x00000000bfad0000)
  to   space 8704K,   0% used [0x00000000bfad0000, 0x00000000bfad0000, 0x00000000c0350000)
 concurrent mark-sweep generation total 174784K, used 0K [0x00000000cfac0000, 0x00000000da570000, 0x00000000fae00000)
 concurrent-mark-sweep perm gen total 21248K, used 8193K [0x00000000fae00000, 0x00000000fc2c0000, 0x0000000100000000)

Card table byte_map: [0x00007f424bad3000,0x00007f424bcfd000] byte_map_base: 0x00007f424b4fc000

Polling page: 0x00007f42557ac000

Code Cache  [0x00007f425005d000, 0x00007f42502cd000, 0x00007f425305d000)
 total_blobs=312 nmethods=36 adapters=230 free_code_cache=48617Kb largest_free_block=49754112

Compilation events (10 events):
Event: 0.281 Thread 0x00007f424c119000 nmethod 32 0x00007f42500cd2d0 code [0x00007f42500cd420, 0x00007f42500cd518]
Event: 0.282 Thread 0x00007f424c119000   33   !         sun.misc.URLClassPath$JarLoader::getResource (91 bytes)
Event: 0.304 Thread 0x00007f424c119000 nmethod 33 0x00007f42500de510 code [0x00007f42500de8c0, 0x00007f42500e0128]
Event: 0.304 Thread 0x00007f424c119000   34             java.util.Arrays::copyOf (19 bytes)
Event: 0.305 Thread 0x00007f424c119000 nmethod 34 0x00007f42500de0d0 code [0x00007f42500de220, 0x00007f42500de3f8]
Event: 0.306 Thread 0x00007f424c11b000 nmethod 31 0x00007f42500e5650 code [0x00007f42500e5a20, 0x00007f42500e7288]
Event: 0.313 Thread 0x00007f424c119000   35             java.util.Arrays::copyOfRange (63 bytes)
Event: 0.317 Thread 0x00007f424c119000 nmethod 35 0x00007f42500d6d90 code [0x00007f42500d6ee0, 0x00007f42500d70f8]
Event: 0.327 Thread 0x00007f424c11b000   36             java.lang.String::<init> (67 bytes)
Event: 0.329 Thread 0x00007f424c11b000 nmethod 36 0x00007f42500d6810 code [0x00007f42500d6960, 0x00007f42500d6bd8]

GC Heap History (0 events):
No events

Deoptimization events (1 events):
Event: 0.189 Thread 0x00007f424c00a000 Uncommon trap: reason=null_check action=make_not_entrant pc=0x00007f42500c0a60 method=java.lang.String.equals(Ljava/lang/Object;)Z @ 8

Internal exceptions (10 events):
Event: 0.330 Thread 0x00007f424c00a000 Threw 0x00000000bbd50128 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u75/1940/hotspot/src/share/vm/prims/jvm.cpp:1281
Event: 0.332 Thread 0x00007f424c00a000 Threw 0x00000000bbd64960 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u75/1940/hotspot/src/share/vm/prims/jvm.cpp:1281
Event: 0.332 Thread 0x00007f424c00a000 Threw 0x00000000bbd6bde8 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u75/1940/hotspot/src/share/vm/prims/jvm.cpp:1281
Event: 0.332 Thread 0x00007f424c00a000 Threw 0x00000000bbd6f078 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u75/1940/hotspot/src/share/vm/prims/jvm.cpp:1281
Event: 0.333 Thread 0x00007f424c00a000 Threw 0x00000000bbd71ef8 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u75/1940/hotspot/src/share/vm/prims/jvm.cpp:1281
Event: 0.333 Thread 0x00007f424c00a000 Threw 0x00000000bbd75608 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u75/1940/hotspot/src/share/vm/prims/jvm.cpp:1281
Event: 0.333 Thread 0x00007f424c00a000 Threw 0x00000000bbd87288 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u75/1940/hotspot/src/share/vm/prims/jvm.cpp:1281
Event: 0.334 Thread 0x00007f424c00a000 Threw 0x00000000bbd8a530 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u75/1940/hotspot/src/share/vm/prims/jvm.cpp:1281
Event: 0.334 Thread 0x00007f424c00a000 Threw 0x00000000bbd8d850 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u75/1940/hotspot/src/share/vm/prims/jvm.cpp:1281
Event: 0.334 Thread 0x00007f424c00a000 Threw 0x00000000bbd90b70 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u75/1940/hotspot/src/share/vm/prims/jvm.cpp:1281

Events (10 events):
Event: 0.333 loading class 0x00007f424c43de50
Event: 0.333 loading class 0x00007f424c43de50 done
Event: 0.333 loading class 0x00007f424c44a060
Event: 0.333 loading class 0x00007f424c44a060 done
Event: 0.334 loading class 0x00007f424c44a0a0
Event: 0.334 loading class 0x00007f424c44a0a0 done
Event: 0.334 loading class 0x00007f424c44a200
Event: 0.334 loading class 0x00007f424c44a200 done
Event: 0.334 loading class 0x00007f424c44a240
Event: 0.334 loading class 0x00007f424c44a240 done


Dynamic libraries:
00400000-00401000 r-xp 00000000 fd:00 19763                              /usr/java/jdk1.7.0_75/bin/java
00600000-00601000 rw-p 00000000 fd:00 19763                              /usr/java/jdk1.7.0_75/bin/java
00f22000-00f43000 rw-p 00000000 00:00 0                                  [heap]
bae00000-c0350000 rw-p 00000000 00:00 0
c0350000-cfac0000 rw-p 00000000 00:00 0
cfac0000-da570000 rw-p 00000000 00:00 0
da570000-fae00000 rw-p 00000000 00:00 0
fae00000-fc2c0000 rw-p 00000000 00:00 0
fc2c0000-100000000 rw-p 00000000 00:00 0
7f4200000000-7f4200021000 rw-p 00000000 00:00 0
7f4200021000-7f4204000000 ---p 00000000 00:00 0
7f4208000000-7f42082fd000 rw-p 00000000 00:00 0
7f42082fd000-7f420c000000 ---p 00000000 00:00 0
7f420c000000-7f420c021000 rw-p 00000000 00:00 0
7f420c021000-7f4210000000 ---p 00000000 00:00 0
7f4210000000-7f4210021000 rw-p 00000000 00:00 0
7f4210021000-7f4214000000 ---p 00000000 00:00 0
7f4214000000-7f421442f000 rw-p 00000000 00:00 0
7f421442f000-7f4218000000 ---p 00000000 00:00 0
7f4218000000-7f4218021000 rw-p 00000000 00:00 0
7f4218021000-7f421c000000 ---p 00000000 00:00 0
7f421e16f000-7f4224000000 r--p 00000000 fd:00 11888                      /usr/lib/locale/locale-archive
7f4224000000-7f4224021000 rw-p 00000000 00:00 0
7f4224021000-7f4228000000 ---p 00000000 00:00 0
7f4228000000-7f4228021000 rw-p 00000000 00:00 0
7f4228021000-7f422c000000 ---p 00000000 00:00 0
7f422c000000-7f422c021000 rw-p 00000000 00:00 0
7f422c021000-7f4230000000 ---p 00000000 00:00 0
7f4230000000-7f4230021000 rw-p 00000000 00:00 0
7f4230021000-7f4234000000 ---p 00000000 00:00 0
7f4234000000-7f4234021000 rw-p 00000000 00:00 0
7f4234021000-7f4238000000 ---p 00000000 00:00 0
7f4238000000-7f4238021000 rw-p 00000000 00:00 0
7f4238021000-7f423c000000 ---p 00000000 00:00 0
7f423c000000-7f423c021000 rw-p 00000000 00:00 0
7f423c021000-7f4240000000 ---p 00000000 00:00 0
7f4242015000-7f424225d000 rw-p 00000000 00:00 0
7f424225d000-7f4242274000 r-xp 00000000 fd:00 3301                       /usr/share/elasticsearch/tmp/jna4948368637624641726.tmp (deleted)
7f4242274000-7f4242373000 ---p 00017000 fd:00 3301                       /usr/share/elasticsearch/tmp/jna4948368637624641726.tmp (deleted)
7f4242373000-7f4242375000 rw-p 00016000 fd:00 3301                       /usr/share/elasticsearch/tmp/jna4948368637624641726.tmp (deleted)
7f4242375000-7f424237d000 r-xp 00000000 fd:00 19924                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libmanagement.so
7f424237d000-7f424257c000 ---p 00008000 fd:00 19924                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libmanagement.so
7f424257c000-7f424257d000 rw-p 00007000 fd:00 19924                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libmanagement.so
7f424257d000-7f424258d000 r-xp 00000000 fd:00 19927                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libnio.so
7f424258d000-7f424278d000 ---p 00010000 fd:00 19927                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libnio.so
7f424278d000-7f424278e000 rw-p 00010000 fd:00 19927                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libnio.so
7f424278e000-7f4242a8e000 rw-p 00000000 00:00 0
7f4242a8e000-7f4242aa4000 r-xp 00000000 fd:00 19926                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libnet.so
7f4242aa4000-7f4242ca4000 ---p 00016000 fd:00 19926                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libnet.so
7f4242ca4000-7f4242ca5000 rw-p 00016000 fd:00 19926                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libnet.so
7f4242ca5000-7f4242cad000 r--s 00062000 fd:00 22811                      /usr/share/elasticsearch/lib/lucene-backward-codecs-5.4.1.jar
7f4242cad000-7f4242cb1000 r--s 0002c000 fd:00 22821                      /usr/share/elasticsearch/lib/lucene-spatial-5.4.1.jar
7f4242cb1000-7f4242cb2000 r--s 00001000 fd:00 22825                      /usr/share/elasticsearch/lib/securesm-1.0.jar
7f4242cb2000-7f4242cdd000 r--s 001fc000 fd:00 22502                      /usr/share/elasticsearch/lib/guava-18.0.jar
7f4242cdd000-7f4242ce1000 r--s 0002d000 fd:00 22822                      /usr/share/elasticsearch/lib/lucene-spatial3d-5.4.1.jar
7f4242ce1000-7f4242ceb000 r--s 00059000 fd:00 22819                      /usr/share/elasticsearch/lib/lucene-queryparser-5.4.1.jar
7f4242ceb000-7f4242cf1000 r--s 00037000 fd:00 22818                      /usr/share/elasticsearch/lib/lucene-queries-5.4.1.jar
7f4242cf1000-7f4242cf2000 r--s 0000c000 fd:00 22483                      /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar
7f4242cf2000-7f4242cf4000 r--s 00007000 fd:00 22816                      /usr/share/elasticsearch/lib/lucene-memory-5.4.1.jar
7f4242cf4000-7f4242cf5000 r--s 0000b000 fd:00 22511                      /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.2.jar
7f4242cf5000-7f4242cf9000 r--s 0001a000 fd:00 22815                      /usr/share/elasticsearch/lib/lucene-join-5.4.1.jar
7f4242cf9000-7f4242cfc000 r--s 00022000 fd:00 22827                      /usr/share/elasticsearch/lib/spatial4j-0.5.jar
7f4242cfc000-7f4242d13000 r--s 00100000 fd:00 22503                      /usr/share/elasticsearch/lib/hppc-0.7.1.jar
7f4242d13000-7f4242d17000 r--s 0003c000 fd:00 22506                      /usr/share/elasticsearch/lib/jackson-core-2.6.2.jar
7f4242d17000-7f4242d1c000 r--s 00038000 fd:00 22823                      /usr/share/elasticsearch/lib/lucene-suggest-5.4.1.jar
7f4242d1c000-7f4242d20000 r--s 00018000 fd:00 22486                      /usr/share/elasticsearch/lib/compiler-0.8.13.jar
7f4242d20000-7f4242d26000 r--s 0003c000 fd:00 22826                      /usr/share/elasticsearch/lib/snakeyaml-1.15.jar
7f4242d26000-7f4242d2a000 r--s 000dc000 fd:00 22804                      /usr/share/elasticsearch/lib/jna-4.1.0.jar
7f4242d2a000-7f4242d33000 r--s 0006f000 fd:00 22809                      /usr/share/elasticsearch/lib/log4j-1.2.17.jar
7f4242d33000-7f4242d43000 r--s 00088000 fd:00 22806                      /usr/share/elasticsearch/lib/joda-time-2.8.2.jar
7f4242d43000-7f4242d47000 r--s 00027000 fd:00 22817                      /usr/share/elasticsearch/lib/lucene-misc-5.4.1.jar
7f4242d47000-7f4242d49000 r--s 00008000 fd:00 22805                      /usr/share/elasticsearch/lib/joda-convert-1.2.jar
7f4242d49000-7f4242d5a000 r--s 000b2000 fd:00 22808                      /usr/share/elasticsearch/lib/jts-1.13.jar
7f4242d5a000-7f4242d73000 r--s 0012c000 fd:00 22824                      /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar
7f4242d73000-7f4242d7c000 r--s 00046000 fd:00 22513                      /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.2.jar
7f4242d7c000-7f4242d7e000 r--s 00011000 fd:00 22512                      /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.2.jar
7f4242d7e000-7f4242d81000 r--s 00018000 fd:00 22813                      /usr/share/elasticsearch/lib/lucene-grouping-5.4.1.jar
7f4242d81000-7f4242da9000 r--s 00218000 fd:00 22812                      /usr/share/elasticsearch/lib/lucene-core-5.4.1.jar
7f4242da9000-7f4242dbb000 r--s 00170000 fd:00 22810                      /usr/share/elasticsearch/lib/lucene-analyzers-common-5.4.1.jar
7f4242dbb000-7f4242e6b000 r--s 0083d000 fd:00 22500                      /usr/share/elasticsearch/lib/elasticsearch-2.2.0.jar
7f4242e6b000-7f4242e6c000 ---p 00000000 00:00 0
7f4242e6c000-7f4242f6c000 rw-p 00000000 00:00 0
7f4242f6c000-7f4242f6f000 ---p 00000000 00:00 0
7f4242f6f000-7f424306d000 rw-p 00000000 00:00 0
7f424306d000-7f4243070000 ---p 00000000 00:00 0
7f4243070000-7f424316e000 rw-p 00000000 00:00 0
7f424316e000-7f4243171000 ---p 00000000 00:00 0
7f4243171000-7f424326f000 rw-p 00000000 00:00 0
7f424326f000-7f4243272000 ---p 00000000 00:00 0
7f4243272000-7f4243370000 rw-p 00000000 00:00 0
7f4243370000-7f4243373000 ---p 00000000 00:00 0
7f4243373000-7f4243471000 rw-p 00000000 00:00 0
7f4243471000-7f4243474000 ---p 00000000 00:00 0
7f4243474000-7f4243572000 rw-p 00000000 00:00 0
7f4243572000-7f4243575000 ---p 00000000 00:00 0
7f4243575000-7f4243673000 rw-p 00000000 00:00 0
7f4243673000-7f4243674000 ---p 00000000 00:00 0
7f4243674000-7f42437bf000 rw-p 00000000 00:00 0
7f42437bf000-7f424397e000 r--s 039eb000 fd:00 22381                      /usr/java/jdk1.7.0_75/jre/lib/rt.jar
7f424397e000-7f4243afb000 rw-p 00000000 00:00 0
7f4243afb000-7f4243afc000 ---p 00000000 00:00 0
7f4243afc000-7f4243dfe000 rw-p 00000000 00:00 0
7f4243dfe000-7f4243dff000 ---p 00000000 00:00 0
7f4243dff000-7f4244000000 rw-p 00000000 00:00 0
7f4244000000-7f4244021000 rw-p 00000000 00:00 0
7f4244021000-7f4248000000 ---p 00000000 00:00 0
7f4248005000-7f4248009000 r--s 00020000 fd:00 22814                      /usr/share/elasticsearch/lib/lucene-highlighter-5.4.1.jar
7f4248009000-7f424800e000 r--s 0003e000 fd:00 22820                      /usr/share/elasticsearch/lib/lucene-sandbox-5.4.1.jar
7f424800e000-7f4248010000 r--s 0000e000 fd:00 22807                      /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar
7f4248010000-7f4248012000 r--s 0000b000 fd:00 22828                      /usr/share/elasticsearch/lib/t-digest-3.0.jar
7f4248012000-7f4248014000 r--s 00012000 fd:00 22493                      /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar
7f4248014000-7f424801c000 r--s 00066000 fd:00 22482                      /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar
7f424801c000-7f424b5cb000 rw-p 00000000 00:00 0
7f424b5cb000-7f424b6cf000 rw-p 00000000 00:00 0
7f424b6cf000-7f424bafe000 rw-p 00000000 00:00 0
7f424bafe000-7f424bb79000 rw-p 00000000 00:00 0
7f424bb79000-7f424bbcf000 rw-p 00000000 00:00 0
7f424bbcf000-7f424bcd3000 rw-p 00000000 00:00 0
7f424bcd3000-7f424bcde000 rw-p 00000000 00:00 0
7f424bcde000-7f424bcfc000 rw-p 00000000 00:00 0
7f424bcfc000-7f424bcfd000 rw-p 00000000 00:00 0
7f424bcfd000-7f424bcfe000 ---p 00000000 00:00 0
7f424bcfe000-7f424bdfe000 rw-p 00000000 00:00 0
7f424bdfe000-7f424bdff000 ---p 00000000 00:00 0
7f424bdff000-7f424beff000 rw-p 00000000 00:00 0
7f424beff000-7f424bf00000 ---p 00000000 00:00 0
7f424bf00000-7f424c000000 rw-p 00000000 00:00 0
7f424c000000-7f424c45c000 rw-p 00000000 00:00 0
7f424c45c000-7f4250000000 ---p 00000000 00:00 0
7f4250002000-7f425003e000 rw-p 00000000 00:00 0
7f425003e000-7f425005d000 rw-p 00000000 00:00 0
7f425005d000-7f42502cd000 rwxp 00000000 00:00 0
7f42502cd000-7f425305d000 rw-p 00000000 00:00 0
7f425305d000-7f4253077000 r-xp 00000000 fd:00 19938                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libzip.so
7f4253077000-7f4253277000 ---p 0001a000 fd:00 19938                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libzip.so
7f4253277000-7f4253278000 rw-p 0001a000 fd:00 19938                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libzip.so
7f4253278000-7f4253284000 r-xp 00000000 fd:00 14665                      /lib64/libnss_files-2.12.so
7f4253284000-7f4253484000 ---p 0000c000 fd:00 14665                      /lib64/libnss_files-2.12.so
7f4253484000-7f4253485000 r--p 0000c000 fd:00 14665                      /lib64/libnss_files-2.12.so
7f4253485000-7f4253486000 rw-p 0000d000 fd:00 14665                      /lib64/libnss_files-2.12.so
7f4253486000-7f42534af000 r-xp 00000000 fd:00 19908                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libjava.so
7f42534af000-7f42536af000 ---p 00029000 fd:00 19908                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libjava.so
7f42536af000-7f42536b1000 rw-p 00029000 fd:00 19908                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libjava.so
7f42536b1000-7f42536be000 r-xp 00000000 fd:00 19937                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libverify.so
7f42536be000-7f42538bd000 ---p 0000d000 fd:00 19937                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libverify.so
7f42538bd000-7f42538bf000 rw-p 0000c000 fd:00 19937                      /usr/java/jdk1.7.0_75/jre/lib/amd64/libverify.so
7f42538bf000-7f42538c6000 r-xp 00000000 fd:00 22663                      /lib64/librt-2.12.so
7f42538c6000-7f4253ac5000 ---p 00007000 fd:00 22663                      /lib64/librt-2.12.so
7f4253ac5000-7f4253ac6000 r--p 00006000 fd:00 22663                      /lib64/librt-2.12.so
7f4253ac6000-7f4253ac7000 rw-p 00007000 fd:00 22663                      /lib64/librt-2.12.so
7f4253ac7000-7f4253b4a000 r-xp 00000000 fd:00 14232                      /lib64/libm-2.12.so
7f4253b4a000-7f4253d49000 ---p 00083000 fd:00 14232                      /lib64/libm-2.12.so
7f4253d49000-7f4253d4a000 r--p 00082000 fd:00 14232                      /lib64/libm-2.12.so
7f4253d4a000-7f4253d4b000 rw-p 00083000 fd:00 14232                      /lib64/libm-2.12.so
7f4253d4b000-7f42548be000 r-xp 00000000 fd:00 19942                      /usr/java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
7f42548be000-7f4254abe000 ---p 00b73000 fd:00 19942                      /usr/java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
7f4254abe000-7f4254b82000 rw-p 00b73000 fd:00 19942                      /usr/java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
7f4254b82000-7f4254bc3000 rw-p 00000000 00:00 0
7f4254bc3000-7f4254d4d000 r-xp 00000000 fd:00 3286                       /lib64/libc-2.12.so
7f4254d4d000-7f4254f4d000 ---p 0018a000 fd:00 3286                       /lib64/libc-2.12.so
7f4254f4d000-7f4254f51000 r--p 0018a000 fd:00 3286                       /lib64/libc-2.12.so
7f4254f51000-7f4254f52000 rw-p 0018e000 fd:00 3286                       /lib64/libc-2.12.so
7f4254f52000-7f4254f57000 rw-p 00000000 00:00 0
7f4254f57000-7f4254f59000 r-xp 00000000 fd:00 13342                      /lib64/libdl-2.12.so
7f4254f59000-7f4255159000 ---p 00002000 fd:00 13342                      /lib64/libdl-2.12.so
7f4255159000-7f425515a000 r--p 00002000 fd:00 13342                      /lib64/libdl-2.12.so
7f425515a000-7f425515b000 rw-p 00003000 fd:00 13342                      /lib64/libdl-2.12.so
7f425515b000-7f4255171000 r-xp 00000000 fd:00 21163                      /usr/java/jdk1.7.0_75/lib/amd64/jli/libjli.so
7f4255171000-7f4255371000 ---p 00016000 fd:00 21163                      /usr/java/jdk1.7.0_75/lib/amd64/jli/libjli.so
7f4255371000-7f4255372000 rw-p 00016000 fd:00 21163                      /usr/java/jdk1.7.0_75/lib/amd64/jli/libjli.so
7f4255372000-7f4255389000 r-xp 00000000 fd:00 3310                       /lib64/libpthread-2.12.so
7f4255389000-7f4255589000 ---p 00017000 fd:00 3310                       /lib64/libpthread-2.12.so
7f4255589000-7f425558a000 r--p 00017000 fd:00 3310                       /lib64/libpthread-2.12.so
7f425558a000-7f425558b000 rw-p 00018000 fd:00 3310                       /lib64/libpthread-2.12.so
7f425558b000-7f425558f000 rw-p 00000000 00:00 0
7f425558f000-7f42555af000 r-xp 00000000 fd:00 3277                       /lib64/ld-2.12.so
7f42555b0000-7f42555e2000 rw-p 00000000 00:00 0
7f42555e2000-7f4255698000 rw-p 00000000 00:00 0
7f4255698000-7f42556a0000 rw-s 00000000 fd:01 37                         /tmp/hsperfdata_elastic/28216
7f42556a0000-7f42556a3000 ---p 00000000 00:00 0
7f42556a3000-7f42557a5000 rw-p 00000000 00:00 0
7f42557a5000-7f42557a9000 r--s 0008b000 fd:00 20725                      /usr/java/jdk1.7.0_75/jre/lib/jsse.jar
7f42557a9000-7f42557ab000 r--s 00019000 fd:00 22406                      /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar
7f42557ab000-7f42557ac000 rw-p 00000000 00:00 0
7f42557ac000-7f42557ad000 r--p 00000000 00:00 0
7f42557ad000-7f42557ae000 rw-p 00000000 00:00 0
7f42557ae000-7f42557af000 r--p 0001f000 fd:00 3277                       /lib64/ld-2.12.so
7f42557af000-7f42557b0000 rw-p 00020000 fd:00 3277                       /lib64/ld-2.12.so
7f42557b0000-7f42557b1000 rw-p 00000000 00:00 0
7fff3d014000-7fff3d029000 rw-p 00000000 00:00 0                          [stack]
7fff3d031000-7fff3d032000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
jvm_args: -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Djna.tmpdir=/usr/share/elasticsearch/tmp -Djava.io.tmpdir=/usr/share/elasticsearch/tmp -Des.path.home=/usr/share/elasticsearch
java_command: org.elasticsearch.bootstrap.Elasticsearch start -p /var/run/elasticsearch/elasticsearch.pid -d -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=/sbin:/usr/sbin:/bin:/usr/bin
SHELL=/bin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0x9a21f0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x9a21f0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x81ba40], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x81ba40], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x81ba40], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x81ba40], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x81d240], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x81e360], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0x81e360], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x81e360], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004


---------------  S Y S T E M  ---------------

OS:CentOS release 6.7 (Final)

uname:Linux 2.6.32-573.7.1.el6.x86_64 #1 SMP Tue Sep 22 22:00:00 UTC 2015 x86_64
libc:glibc 2.12 NPTL 2.12
rlimit: STACK 10240k, CORE 0k, NPROC 1024, NOFILE 65535, AS infinity
load average:0.12 0.10 0.08

/proc/meminfo:
MemTotal:        3924284 kB
MemFree:         2475800 kB
Buffers:           95816 kB
Cached:          1058380 kB
SwapCached:         1656 kB
Active:           724236 kB
Inactive:         496232 kB
Active(anon):      52848 kB
Inactive(anon):    13496 kB
Active(file):     671388 kB
Inactive(file):   482736 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       1048572 kB
SwapFree:        1015456 kB
Dirty:               272 kB
Writeback:             0 kB
AnonPages:         40536 kB
Mapped:            21068 kB
Shmem:                16 kB
Slab:             177536 kB
SReclaimable:     112616 kB
SUnreclaim:        64920 kB
KernelStack:        2880 kB
PageTables:         3948 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3010712 kB
Committed_AS:     502800 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      150516 kB
VmallocChunk:   34359576444 kB
HardwareCorrupted:     0 kB
AnonHugePages:      6144 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8192 kB
DirectMap2M:     2088960 kB
DirectMap1G:     2097152 kB


CPU:total 4 (1 cores per cpu, 1 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, erms, tsc, tscinvbit

/proc/cpuinfo:
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
stepping    : 2
microcode   : 45
cpu MHz     : 3491.914
cache size  : 15360 KB
physical id : 0
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 15
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb pln pts dts fsgsbase smep
bogomips    : 6983.82
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
stepping    : 2
microcode   : 45
cpu MHz     : 3491.914
cache size  : 15360 KB
physical id : 2
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 15
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb pln pts dts fsgsbase smep
bogomips    : 6983.82
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
stepping    : 2
microcode   : 45
cpu MHz     : 3491.914
cache size  : 15360 KB
physical id : 4
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 4
initial apicid  : 4
fpu     : yes
fpu_exception   : yes
cpuid level : 15
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb pln pts dts fsgsbase smep
bogomips    : 6983.82
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 3
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
stepping    : 2
microcode   : 45
cpu MHz     : 3491.914
cache size  : 15360 KB
physical id : 6
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 6
initial apicid  : 6
fpu     : yes
fpu_exception   : yes
cpuid level : 15
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb pln pts dts fsgsbase smep
bogomips    : 6983.82
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 3924284k(2475800k free), swap 1048572k(1015456k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (24.75-b04) for linux-amd64 JRE (1.7.0_75-b13), built on Dec 18 2014 16:45:18 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

time: Wed May 11 10:18:58 2016
elapsed time: 0 seconds

[bubo77]

i have the same issue
#18406

[clintongormley]

This looks like a JNA problem. See http://mail-archives.apache.org/mod_mbox/cassandra-user/201405.mbox/%3Cpan.2014.05.27.15.14.54@googlemail.com%3E

Try just setting the jna tmpdir to a tmp folder without noexec.

[bubo77]

@clintongormley if you read this ticket and mine again you can see that we tried to point the ES config to a new directory with appropriate rights and it still doesn't work.

ls -alZ /var/lib/elasticsearch/
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0 .
drwxr-xr-x. root root unconfined_u:object_r:usr_t:s0 ..
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0 graylog2
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0 tmp

[clintongormley]

Related to #14372

[jasontedor]

I've tried to reproduce this on Centos 6.7 and Fedora 23. It does not reproduce for me.

I notice that both error reports show death trying to register the C library. This the first step before we can even make any native calls. Thus, we are immediately dying here and it's not that we are interacting poorly with other native calls (e.g., mlockall or seccomp).

Is there additional configuration at play here that might be relevant? What else can you share about the system? What kernel do you have? I tested on 2.6.32-573.12.1.el6.x86_64 (Centos 6.7) and 4.2.3-300.fc23.x86_64 (Fedora 23). What's the output of ldd --version? What happens if you add -Djna.nosys=false to the ES_JAVA_OPTS?

[fxh]

Kernel: 2.6.32-573.7.1.el6.x86_64
CentOS release 6.7 (Final)

$ ldd --version
ldd (GNU libc) 2.12
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

Adding -Djna.nosys=false did not help:

Starting elasticsearch: /usr/share/elasticsearch/bin/elasticsearch: line 155: 31386 Aborted                 (core dumped) exec "$JAVA" $JAVA_OPTS $ES_JAVA_OPTS -Des.path.home="$ES_HOME" -cp "$ES_CLASSPATH" org.elasticsearch.bootstrap.Elasticsearch start "$@" 0>&-

[bubo77]

CentOS Linux release 7.2.1511 (Core)
Kernel: 3.10.0-327.10.1.el7.x86_64

ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper

My tmp dir is on an LVM disk but it doesn't matter if the tmp dir is a symlink or not.

ls -alZ /export/apps/elasticsearch/
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0       .
drwxr-xr-x. root          root          unconfined_u:object_r:usr_t:s0   ..
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0       graylog2
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0       tmp

[jasontedor]

@fxh @bubo77 Thanks. Nothing alarming there. Can you add these system properties to ES_JAVA_OPTS (leave off the -Djna.nosys=false):

-Djna.debug_load.jna=true -Djna.debug_load=true -Djna.tmpdir=/path/to/tmp -Djava.io.tmpdir=/path/to/tmp

and start Elasticsearch (only the first two should be new). This should produce output like:

Looking in classpath from sun.misc.Launcher$AppClassLoader@5c647e05 for /com/sun/jna/linux-x86-64/libjnidispatch.so
Found library resource at jar:file:/home/jason/elasticsearch/elasticsearch-5.0.0-SNAPSHOT/lib/jna-4.1.0.jar!/com/sun/jna/linux-x86-64/libjnidispatch.so
Trying /home/jason/elasticsearch/elasticsearch-5.0.0-SNAPSHOT/tmp/jna6458947537802950020.tmp
Found jnidispatch at /home/jason/elasticsearch/elasticsearch-5.0.0-SNAPSHOT/tmp/jna6458947537802950020.tmp

Can you share that and any additional JNA debugging output?

[jasontedor]

Also, can you share your SELinux configuration via sestatus and getsebool -a? If you're willing to, can you completely disable SELinux and try as well (preferably through /etc/selinux/config and a reboot)? If that starts up successfully, that will at least confirm my current hunch that it's SELinux and we just need to find the right configuration to let JNA do it's thing.

[bubo77]

sestatus:

tmp]# sestatus -v
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

Process contexts:
Current context:                unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Init context:                   system_u:system_r:init_t:s0
/usr/sbin/sshd                  system_u:system_r:sshd_t:s0-s0:c0.c1023

File contexts:
Controlling terminal:           unconfined_u:object_r:user_devpts_t:s0
/etc/passwd                     system_u:object_r:passwd_file_t:s0
/etc/shadow                     system_u:object_r:shadow_t:s0
/bin/bash                       system_u:object_r:shell_exec_t:s0
/bin/login                      system_u:object_r:login_exec_t:s0
/bin/sh                         system_u:object_r:bin_t:s0 -> system_u:object_r:shell_exec_t:s0
/sbin/agetty                    system_u:object_r:getty_exec_t:s0
/sbin/init                      system_u:object_r:bin_t:s0 -> system_u:object_r:init_exec_t:s0
/usr/sbin/sshd                  system_u:object_r:sshd_exec_t:s0

getsebool

abrt_anon_write --> off
abrt_handle_event --> off
abrt_upload_watch_anon_write --> on
antivirus_can_scan_system --> on
antivirus_use_jit --> off
auditadm_exec_content --> on
authlogin_nsswitch_use_ldap --> off
authlogin_radius --> off
authlogin_yubikey --> off
awstats_purge_apache_log_files --> off
boinc_execmem --> on
cdrecord_read_content --> off
cluster_can_network_connect --> off
cluster_manage_all_files --> off
cluster_use_execmem --> off
cobbler_anon_write --> off
cobbler_can_network_connect --> off
cobbler_use_cifs --> off
cobbler_use_nfs --> off
collectd_tcp_network_connect --> off
condor_tcp_network_connect --> off
conman_can_network --> off
cron_can_relabel --> off
cron_system_cronjob_use_shares --> off
cron_userdomain_transition --> on
cups_execmem --> off
cvs_read_shadow --> off
daemons_dump_core --> off
daemons_enable_cluster_mode --> off
daemons_use_tcp_wrapper --> off
daemons_use_tty --> off
dbadm_exec_content --> on
dbadm_manage_user_files --> off
dbadm_read_user_files --> off
deny_execmem --> off
deny_ptrace --> off
dhcpc_exec_iptables --> off
dhcpd_use_ldap --> off
domain_fd_use --> on
domain_kernel_load_modules --> off
entropyd_use_audio --> on
exim_can_connect_db --> off
exim_manage_user_files --> off
exim_read_user_files --> off
fcron_crond --> off
fenced_can_network_connect --> off
fenced_can_ssh --> off
fips_mode --> on
ftp_home_dir --> off
ftpd_anon_write --> off
ftpd_connect_all_unreserved --> off
ftpd_connect_db --> off
ftpd_full_access --> off
ftpd_use_cifs --> off
ftpd_use_fusefs --> off
ftpd_use_nfs --> off
ftpd_use_passive_mode --> off
git_cgi_enable_homedirs --> off
git_cgi_use_cifs --> off
git_cgi_use_nfs --> off
git_session_bind_all_unreserved_ports --> off
git_session_users --> off
git_system_enable_homedirs --> off
git_system_use_cifs --> off
git_system_use_nfs --> off
gitosis_can_sendmail --> off
glance_api_can_network --> off
glance_use_execmem --> off
glance_use_fusefs --> off
global_ssp --> off
gluster_anon_write --> off
gluster_export_all_ro --> off
gluster_export_all_rw --> on
gpg_web_anon_write --> off
gssd_read_tmp --> on
guest_exec_content --> on
haproxy_connect_any --> off
httpd_anon_write --> off
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_connect_ftp --> off
httpd_can_connect_ldap --> off
httpd_can_connect_mythtv --> off
httpd_can_connect_zabbix --> off
httpd_can_network_connect --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_memcache --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off
httpd_dbus_avahi --> off
httpd_dbus_sssd --> off
httpd_dontaudit_search_dirs --> off
httpd_enable_cgi --> on
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_graceful_shutdown --> on
httpd_manage_ipa --> off
httpd_mod_auth_ntlm_winbind --> off
httpd_mod_auth_pam --> off
httpd_read_user_content --> off
httpd_run_ipa --> off
httpd_run_preupgrade --> off
httpd_run_stickshift --> off
httpd_serve_cobbler_files --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_sys_script_anon_write --> off
httpd_tmp_exec --> off
httpd_tty_comm --> off
httpd_unified --> off
httpd_use_cifs --> off
httpd_use_fusefs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off
httpd_use_openstack --> off
httpd_use_sasl --> off
httpd_verify_dns --> off
icecast_use_any_tcp_ports --> off
irc_use_any_tcp_ports --> off
irssi_use_full_network --> off
kdumpgui_run_bootloader --> off
kerberos_enabled --> on
ksmtuned_use_cifs --> off
ksmtuned_use_nfs --> off
logadm_exec_content --> on
logging_syslogd_can_sendmail --> off
logging_syslogd_run_nagios_plugins --> off
logging_syslogd_use_tty --> on
login_console_enabled --> on
logrotate_use_nfs --> off
logwatch_can_network_connect_mail --> off
lsmd_plugin_connect_any --> off
mailman_use_fusefs --> off
mcelog_client --> off
mcelog_exec_scripts --> on
mcelog_foreground --> off
mcelog_server --> off
minidlna_read_generic_user_content --> off
mmap_low_allowed --> off
mock_enable_homedirs --> off
mount_anyfile --> on
mozilla_plugin_bind_unreserved_ports --> off
mozilla_plugin_can_network_connect --> off
mozilla_plugin_use_bluejeans --> off
mozilla_plugin_use_gps --> off
mozilla_plugin_use_spice --> off
mozilla_read_content --> off
mpd_enable_homedirs --> off
mpd_use_cifs --> off
mpd_use_nfs --> off
mplayer_execstack --> off
mysql_connect_any --> off
nagios_run_pnp4nagios --> off
nagios_run_sudo --> off
named_tcp_bind_http_port --> off
named_write_master_zones --> off
neutron_can_network --> off
nfs_export_all_ro --> on
nfs_export_all_rw --> on
nfsd_anon_write --> off
nis_enabled --> off
nscd_use_shm --> on
openshift_use_nfs --> off
openvpn_can_network_connect --> on
openvpn_enable_homedirs --> on
openvpn_run_unconfined --> off
pcp_bind_all_unreserved_ports --> off
pcp_read_generic_logs --> off
piranha_lvs_can_network_connect --> off
polipo_connect_all_unreserved --> off
polipo_session_bind_all_unreserved_ports --> off
polipo_session_users --> off
polipo_use_cifs --> off
polipo_use_nfs --> off
polyinstantiation_enabled --> off
postfix_local_write_mail_spool --> on
postgresql_can_rsync --> off
postgresql_selinux_transmit_client_label --> off
postgresql_selinux_unconfined_dbadm --> on
postgresql_selinux_users_ddl --> on
pppd_can_insmod --> off
pppd_for_user --> off
privoxy_connect_any --> on
prosody_bind_http_port --> off
puppetagent_manage_all_files --> off
puppetmaster_use_db --> off
racoon_read_shadow --> off
rsync_anon_write --> off
rsync_client --> off
rsync_export_all_ro --> off
rsync_full_access --> off
samba_create_home_dirs --> off
samba_domain_controller --> off
samba_enable_home_dirs --> off
samba_export_all_ro --> off
samba_export_all_rw --> off
samba_load_libgfapi --> off
samba_portmapper --> off
samba_run_unconfined --> off
samba_share_fusefs --> off
samba_share_nfs --> off
sanlock_use_fusefs --> off
sanlock_use_nfs --> off
sanlock_use_samba --> off
saslauthd_read_shadow --> off
secadm_exec_content --> on
secure_mode --> off
secure_mode_insmod --> off
secure_mode_policyload --> off
selinuxuser_direct_dri_enabled --> on
selinuxuser_execheap --> off
selinuxuser_execmod --> on
selinuxuser_execstack --> on
selinuxuser_mysql_connect_enabled --> off
selinuxuser_ping --> on
selinuxuser_postgresql_connect_enabled --> off
selinuxuser_rw_noexattrfile --> on
selinuxuser_share_music --> off
selinuxuser_tcp_server --> off
selinuxuser_udp_server --> off
selinuxuser_use_ssh_chroot --> off
sftpd_anon_write --> off
sftpd_enable_homedirs --> off
sftpd_full_access --> off
sftpd_write_ssh_home --> off
sge_domain_can_network_connect --> off
sge_use_nfs --> off
smartmon_3ware --> off
smbd_anon_write --> off
spamassassin_can_network --> off
spamd_enable_home_dirs --> on
squid_connect_any --> on
squid_use_tproxy --> off
ssh_chroot_rw_homedirs --> off
ssh_keysign --> off
ssh_sysadm_login --> off
staff_exec_content --> on
staff_use_svirt --> off
swift_can_network --> off
sysadm_exec_content --> on
telepathy_connect_all_ports --> off
telepathy_tcp_connect_generic_network_ports --> on
tftp_anon_write --> off
tftp_home_dir --> off
tmpreaper_use_nfs --> off
tmpreaper_use_samba --> off
tor_bind_all_unreserved_ports --> off
tor_can_network_relay --> off
unconfined_chrome_sandbox_transition --> on
unconfined_login --> on
unconfined_mozilla_plugin_transition --> on
unprivuser_use_svirt --> off
use_ecryptfs_home_dirs --> off
use_fusefs_home_dirs --> off
use_lpd_server --> off
use_nfs_home_dirs --> off
use_samba_home_dirs --> off
user_exec_content --> on
varnishd_connect_any --> off
virt_read_qemu_ga_data --> off
virt_rw_qemu_ga_data --> off
virt_sandbox_use_all_caps --> on
virt_sandbox_use_audit --> on
virt_sandbox_use_mknod --> off
virt_sandbox_use_netlink --> off
virt_sandbox_use_nfs --> off
virt_sandbox_use_samba --> off
virt_sandbox_use_sys_admin --> off
virt_transition_userdomain --> off
virt_use_comm --> off
virt_use_execmem --> off
virt_use_fusefs --> off
virt_use_nfs --> off
virt_use_rawip --> off
virt_use_samba --> off
virt_use_sanlock --> off
virt_use_usb --> on
virt_use_xserver --> off
webadm_manage_user_files --> off
webadm_read_user_files --> off
wine_mmap_zero_ignore --> off
xdm_bind_vnc_tcp_port --> off
xdm_exec_bootloader --> off
xdm_sysadm_login --> off
xdm_write_home --> off
xen_use_nfs --> off
xend_run_blktap --> on
xend_run_qemu --> on
xguest_connect_network --> on
xguest_exec_content --> on
xguest_mount_media --> on
xguest_use_bluetooth --> on
xserver_clients_write_xshm --> off
xserver_execmem --> off
xserver_object_manager --> off
zabbix_can_network --> off
zarafa_setrlimit --> off
zebra_write_config --> off
zoneminder_anon_write --> off
zoneminder_run_sudo --> off

systemctl dump:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f8c619d340a, pid=19921, tid=140242008348416
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [jna46830484485062808.tmp+0x1240a]  ffi_prep_closure_loc+0x1a
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x00007f8c9c00a000):  JavaThread "main" [_thread_in_native, id=19939, stack(0x00007f8ca3071000,0x00007f8ca3172000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000

Registers:
RAX=0x00007f8c619d38c2, RBX=0x00007f8ca316fb28, RCX=0x00007f8c9c5667f0, RDX=0x00007f8c619c8590
RSP=0x00007f8ca316f8c0, RBP=0x00007f8ca316f8c0, RSI=0x00007f8c9c566810, RDI=0x0000000000000000
R8 =0x00007f8c9c00ab60, R9 =0x0000000000000002, R10=0x00007f8ca316f650, R11=0x00007f8c619d33f0
R12=0x0000000000000000, R13=0x0000000000000008, R14=0x00007f8c9c5667f0, R15=0x0000000000000004
RIP=0x00007f8c619d340a, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000006
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f8ca316f8c0)
0x00007f8ca316f8c0:   00007f8ca316f990 00007f8c619ce4dd
0x00007f8ca316f8d0:   00007f8ca316f950 00007f8ca316fb18
0x00007f8ca316f8e0:   00007f8ca316fb30 00007f8c9c00a1f8
0x00007f8ca316f8f0:   0000000100000000 00007f8c9c56b660
0x00007f8ca316f900:   00007f8c9c56b7a0 00007f8c9c566810
0x00007f8ca316f910:   00000001a316fa80 00007f8c9c566910
0x00007f8ca316f920:   00007f8c9c56a5b0 00007f8c9c56a590
0x00007f8ca316f930:   0000000000000001 00000003a316fb28
0x00007f8ca316f940:   00007f8ca316fb20 00007f8ca316fb18
0x00007f8ca316f950:   00007f8c00000000 00007f8c9c00ab60
0x00007f8ca316f960:   00007f8c7dcbcb78 00007f8c7dcbcb78
0x00007f8ca316f970:   0000000000000000 00007f8c7dcbcb78
0x00007f8ca316f980:   00007f8ca316fb30 00007f8c9c00a000
0x00007f8ca316f990:   00007f8ca316fa80 00007f8c8d0156d4
0x00007f8ca316f9a0:   00007f8ca316fb10 00007f8ca316fb08
0x00007f8ca316f9b0:   0000000000000002 00007f8c619d5eb0
0x00007f8ca316f9c0:   00007f8c619d5eb0 00007f8ca316fad8
0x00007f8ca316f9d0:   00007f8ca25ee0e0 0000000000000000
0x00007f8ca316f9e0:   0000000000000000 00007f8ca316fab0
0x00007f8ca316f9f0:   0000000000000000 00007f8ca316faa0
0x00007f8ca316fa00:   00007f8c9c00a000 00007f8c9c00a000
0x00007f8ca316fa10:   00007f8c7dcbcb78 0000000000000000
0x00007f8ca316fa20:   00007f8ca316fa80 00007f8c8d015459
0x00007f8ca316fa30:   00007f8c9c00a000 00007f8c8d015422
0x00007f8ca316fa40:   00007f8ca316fa40 00007f8c7dcbcb78
0x00007f8ca316fa50:   00007f8ca316fb30 00007f8c7dcc02d8
0x00007f8ca316fa60:   0000000000000000 00007f8c7dcbcb78
0x00007f8ca316fa70:   0000000000000000 00007f8ca316faa0
0x00007f8ca316fa80:   00007f8ca316fb88 00007f8c8d0074a0
0x00007f8ca316fa90:   00000006c14d96a8 00007f8c8d00f2b7
0x00007f8ca316faa0:   00000006c0010ad0 0000000000000000
0x00007f8ca316fab0:   00000006c15aa230 0000000000000000

Instructions: (pc=0x00007f8c619d340a)
0x00007f8c619d33ea:   66 90 66 66 66 90 8b 06 55 41 b9 02 00 00 00 48
0x00007f8c619d33fa:   89 e5 ff c8 83 f8 01 77 44 48 8b 05 c6 49 10 00
0x00007f8c619d340a:   66 c7 07 49 bb 4c 89 47 0c 66 c7 47 0a 49 ba 48
0x00007f8c619d341a:   89 47 02 8b 46 1c 48 89 77 18 48 89 57 20 48 89

Register to memory mapping:

RAX=0x00007f8c619d38c2: ffi_closure_unix64+0 in /export/apps/elasticsearch/tmp/jna46830484485062808.tmp at 0x00007f8c619c1000
RBX=0x00007f8ca316fb28 is pointing into the stack for thread: 0x00007f8c9c00a000
RCX=0x00007f8c9c5667f0 is an unknown value
RDX=0x00007f8c619c8590: <offset 0x7590> in /export/apps/elasticsearch/tmp/jna46830484485062808.tmp at 0x00007f8c619c1000
RSP=0x00007f8ca316f8c0 is pointing into the stack for thread: 0x00007f8c9c00a000
RBP=0x00007f8ca316f8c0 is pointing into the stack for thread: 0x00007f8c9c00a000
RSI=0x00007f8c9c566810 is an unknown value
RDI=0x0000000000000000 is an unknown value
R8 =0x00007f8c9c00ab60 is an unknown value
R9 =0x0000000000000002 is an unknown value
R10=0x00007f8ca316f650 is pointing into the stack for thread: 0x00007f8c9c00a000
R11=0x00007f8c619d33f0: ffi_prep_closure_loc+0 in /export/apps/elasticsearch/tmp/jna46830484485062808.tmp at 0x00007f8c619c1000
R12=0x0000000000000000 is an unknown value
R13=0x0000000000000008 is an unknown value
R14=0x00007f8c9c5667f0 is an unknown value
R15=0x0000000000000004 is an unknown value


Stack: [0x00007f8ca3071000,0x00007f8ca3172000],  sp=0x00007f8ca316f8c0,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [jna46830484485062808.tmp+0x1240a]  ffi_prep_closure_loc+0x1a
C  [jna46830484485062808.tmp+0xd4dd]  Java_com_sun_jna_Native_registerMethod+0x45d
j  com.sun.jna.Native.registerMethod(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/String;[I[J[JIJJLjava/lang/Class;JIZ[Lcom/sun/jna/ToNativeConverter;Lcom/sun/jna/FromNativeConverter;Ljava/lang/String;)J+0
j  com.sun.jna.Native.register(Ljava/lang/Class;Lcom/sun/jna/NativeLibrary;)V+979
j  com.sun.jna.Native.register(Ljava/lang/Class;Ljava/lang/String;)V+28
j  com.sun.jna.Native.register(Ljava/lang/String;)V+7
j  org.elasticsearch.bootstrap.JNACLibrary.<clinit>()V+44
v  ~StubRoutines::call_stub
V  [libjvm.so+0x681a26]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x634fd7]  InstanceKlass::call_class_initializer_impl(instanceKlassHandle, Thread*)+0xd7
V  [libjvm.so+0x636c2c]  InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x1ac
V  [libjvm.so+0x636fe1]  InstanceKlass::initialize(Thread*)+0x41
V  [libjvm.so+0x7e9756]  LinkResolver::resolve_static_call(CallInfo&, KlassHandle&, Symbol*, Symbol*, KlassHandle, bool, bool, Thread*)+0x246
V  [libjvm.so+0x7e99df]  LinkResolver::resolve_invokestatic(CallInfo&, constantPoolHandle, int, Thread*)+0x23f
V  [libjvm.so+0x7e9f11]  LinkResolver::resolve_invoke(CallInfo&, Handle, constantPoolHandle, int, Bytecodes::Code, Thread*)+0x4f1
V  [libjvm.so+0x67a972]  InterpreterRuntime::resolve_invoke(JavaThread*, Bytecodes::Code)+0x1b2
j  org.elasticsearch.bootstrap.JNANatives.definitelyRunningAsRoot()Z+8
j  org.elasticsearch.bootstrap.Natives.definitelyRunningAsRoot()Z+22
j  org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Ljava/nio/file/Path;ZZZ)V+7
j  org.elasticsearch.bootstrap.Bootstrap.setup(ZLorg/elasticsearch/common/settings/Settings;Lorg/elasticsearch/env/Environment;)V+43
j  org.elasticsearch.bootstrap.Bootstrap.init([Ljava/lang/String;)V+235
j  org.elasticsearch.bootstrap.Elasticsearch.main([Ljava/lang/String;)V+1
v  ~StubRoutines::call_stub
V  [libjvm.so+0x681a26]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x6c3692]  jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)+0x362
V  [libjvm.so+0x6e009a]  jni_CallStaticVoidMethod+0x17a
C  [libjli.so+0x7bcc]  JavaMain+0x80c
C  [libpthread.so.0+0x7dc5]  start_thread+0xc5

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  com.sun.jna.Native.registerMethod(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/String;[I[J[JIJJLjava/lang/Class;JIZ[Lcom/sun/jna/ToNativeConverter;Lcom/sun/jna/FromNativeConverter;Ljava/lang/String;)J+0
j  com.sun.jna.Native.register(Ljava/lang/Class;Lcom/sun/jna/NativeLibrary;)V+979
j  com.sun.jna.Native.register(Ljava/lang/Class;Ljava/lang/String;)V+28
j  com.sun.jna.Native.register(Ljava/lang/String;)V+7
j  org.elasticsearch.bootstrap.JNACLibrary.<clinit>()V+44
v  ~StubRoutines::call_stub
j  org.elasticsearch.bootstrap.JNANatives.definitelyRunningAsRoot()Z+8
j  org.elasticsearch.bootstrap.Natives.definitelyRunningAsRoot()Z+22
j  org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Ljava/nio/file/Path;ZZZ)V+7
j  org.elasticsearch.bootstrap.Bootstrap.setup(ZLorg/elasticsearch/common/settings/Settings;Lorg/elasticsearch/env/Environment;)V+43
j  org.elasticsearch.bootstrap.Bootstrap.init([Ljava/lang/String;)V+235
j  org.elasticsearch.bootstrap.Elasticsearch.main([Ljava/lang/String;)V+1
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00007f8c9c131000 JavaThread "Service Thread" daemon [_thread_blocked, id=19950, stack(0x00007f8c7cc1e000,0x00007f8c7cd1f000)]
  0x00007f8c9c109800 JavaThread "C1 CompilerThread1" daemon [_thread_blocked, id=19949, stack(0x00007f8c7cd1f000,0x00007f8c7ce20000)]
  0x00007f8c9c107800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=19948, stack(0x00007f8c7ce20000,0x00007f8c7cf21000)]
  0x00007f8c9c105000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=19947, stack(0x00007f8c7cf21000,0x00007f8c7d022000)]
  0x00007f8c9c103000 JavaThread "Surrogate Locker Thread (Concurrent GC)" daemon [_thread_blocked, id=19946, stack(0x00007f8c7d022000,0x00007f8c7d123000)]
  0x00007f8c9c0d8800 JavaThread "Finalizer" daemon [_thread_blocked, id=19945, stack(0x00007f8c7d123000,0x00007f8c7d224000)]
  0x00007f8c9c0d6000 JavaThread "Reference Handler" daemon [_thread_blocked, id=19944, stack(0x00007f8c7d224000,0x00007f8c7d325000)]
=>0x00007f8c9c00a000 JavaThread "main" [_thread_in_native, id=19939, stack(0x00007f8ca3071000,0x00007f8ca3172000)]

Other Threads:
  0x00007f8c9c0d1000 VMThread [stack: 0x00007f8c7d325000,0x00007f8c7d426000] [id=19943]
  0x00007f8c9c134000 WatcherThread [stack: 0x00007f8c7cb1d000,0x00007f8c7cc1e000] [id=19951]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap:
 par new generation   total 153344K, used 24554K [0x00000006c0000000, 0x00000006ca660000, 0x00000006ca660000)
  eden space 136320K,  18% used [0x00000006c0000000, 0x00000006c17fa9b8, 0x00000006c8520000)
  from space 17024K,   0% used [0x00000006c8520000, 0x00000006c8520000, 0x00000006c95c0000)
  to   space 17024K,   0% used [0x00000006c95c0000, 0x00000006c95c0000, 0x00000006ca660000)
 concurrent mark-sweep generation total 4023936K, used 0K [0x00000006ca660000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 8237K, capacity 8360K, committed 8576K, reserved 1056768K
  class space    used 998K, capacity 1040K, committed 1152K, reserved 1048576K

Card table byte_map: [0x00007f8c8c7ff000,0x00007f8c8d000000] byte_map_base: 0x00007f8c891ff000

Marking Bits: (CMSBitMap*) 0x00007f8c9c0499f8
 Bits: [0x00007f8c84299000, 0x00007f8c87fff800)

Mod Union Table: (CMSBitMap*) 0x00007f8c9c049ab8
 Bits: [0x00007f8c841a3000, 0x00007f8c842989a0)

Polling page: 0x00007f8ca317e000

CodeCache: size=245760Kb used=2282Kb max_used=2282Kb free=243477Kb
 bounds [0x00007f8c8d000000, 0x00007f8c8d270000, 0x00007f8c9c000000]
 total_blobs=900 nmethods=552 adapters=263
 compilation: enabled

Compilation events (10 events):
Event: 0.576 Thread 0x00007f8c9c109800  545   !   3       java.util.zip.ZipFile$ZipFileInflaterInputStream::close (67 bytes)
Event: 0.577 Thread 0x00007f8c9c109800 nmethod 545 0x00007f8c8d238450 code [0x00007f8c8d238620, 0x00007f8c8d238e58]
Event: 0.577 Thread 0x00007f8c9c109800  546       3       java.util.zip.InflaterInputStream::close (34 bytes)
Event: 0.577 Thread 0x00007f8c9c109800 nmethod 546 0x00007f8c8d239210 code [0x00007f8c8d2393a0, 0x00007f8c8d239698]
Event: 0.577 Thread 0x00007f8c9c109800  547       3       java.util.zip.ZipFile::access$100 (6 bytes)
Event: 0.577 Thread 0x00007f8c9c109800 nmethod 547 0x00007f8c8d239790 code [0x00007f8c8d239900, 0x00007f8c8d239a68]
Event: 0.577 Thread 0x00007f8c9c109800  548   !   3       java.util.zip.ZipFile::releaseInflater (41 bytes)
Event: 0.578 Thread 0x00007f8c9c109800 nmethod 548 0x00007f8c8d239b10 code [0x00007f8c8d239cc0, 0x00007f8c8d23a3f8]
Event: 0.578 Thread 0x00007f8c9c109800  549   !   3       java.util.zip.Inflater::reset (69 bytes)
Event: 0.578 Thread 0x00007f8c9c109800 nmethod 549 0x00007f8c8d23a710 code [0x00007f8c8d23a8a0, 0x00007f8c8d23acf8]

GC Heap History (0 events):
No events

Deoptimization events (2 events):
Event: 0.154 Thread 0x00007f8c9c00a000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f8c8d151b40 method=java.lang.String.indexOf(II)I @ 49
Event: 0.501 Thread 0x00007f8c9c00a000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f8c8d1f8274 method=sun.misc.MetaIndex.mayContain(Ljava/lang/String;)Z @ 38

Internal exceptions (10 events):
Event: 0.492 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/LayoutCustomizer> (0x00000006c1385118) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.493 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/PatternLayoutCustomizer> (0x00000006c1390fb8) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.506 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/DailyRollingFileAppenderBeanInfo> (0x00000006c13cfdd0) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.506 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/FileAppenderBeanInfo> (0x00000006c13d6ff8) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.506 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/WriterAppenderBeanInfo> (0x00000006c13de3a0) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.506 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/AppenderSkeletonBeanInfo> (0x00000006c13e6058) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.507 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/AppenderSkeletonCustomizer> (0x00000006c13ee1e8) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.509 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/WriterAppenderCustomizer> (0x00000006c1407818) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.511 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/FileAppenderCustomizer> (0x00000006c141b250) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.513 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/DailyRollingFileAppenderCustomizer> (0x00000006c14308c0) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]

Events (10 events):
Event: 0.574 loading class com/sun/jna/Function
Event: 0.574 loading class com/sun/jna/Function done
Event: 0.575 loading class com/sun/jna/MethodParameterContext
Event: 0.575 loading class com/sun/jna/MethodParameterContext done
Event: 0.575 loading class com/sun/jna/FunctionParameterContext
Event: 0.575 loading class com/sun/jna/FunctionParameterContext done
Event: 0.576 loading class com/sun/jna/MethodResultContext
Event: 0.576 loading class com/sun/jna/MethodResultContext done
Event: 0.576 loading class com/sun/jna/FunctionResultContext
Event: 0.576 loading class com/sun/jna/FunctionResultContext done


Dynamic libraries:
00400000-00401000 r-xp 00000000 fd:00 135193                             /usr/java/jre1.8.0_45/bin/java
00600000-00601000 rw-p 00000000 fd:00 135193                             /usr/java/jre1.8.0_45/bin/java
023b6000-023d7000 rw-p 00000000 00:00 0                                  [heap]
6c0000000-7c0120000 rw-p 00000000 00:00 0
7c0120000-800000000 ---p 00000000 00:00 0
7f8c4c000000-7f8c4c021000 rw-p 00000000 00:00 0
7f8c4c021000-7f8c50000000 ---p 00000000 00:00 0
7f8c50000000-7f8c50021000 rw-p 00000000 00:00 0
7f8c50021000-7f8c54000000 ---p 00000000 00:00 0
7f8c54000000-7f8c5423f000 rw-p 00000000 00:00 0
7f8c5423f000-7f8c58000000 ---p 00000000 00:00 0
7f8c58000000-7f8c581a4000 rw-p 00000000 00:00 0
7f8c581a4000-7f8c5c000000 ---p 00000000 00:00 0
7f8c5c000000-7f8c5c021000 rw-p 00000000 00:00 0
7f8c5c021000-7f8c60000000 ---p 00000000 00:00 0
7f8c619c1000-7f8c619d8000 r-xp 00000000 fd:05 131152                     /export/apps/elasticsearch/tmp/jna46830484485062808.tmp (deleted)
7f8c619d8000-7f8c61ad7000 ---p 00017000 fd:05 131152                     /export/apps/elasticsearch/tmp/jna46830484485062808.tmp (deleted)
7f8c61ad7000-7f8c61ad9000 rw-p 00016000 fd:05 131152                     /export/apps/elasticsearch/tmp/jna46830484485062808.tmp (deleted)
7f8c61ad9000-7f8c68000000 r--p 00000000 fd:00 4384                       /usr/lib/locale/locale-archive
7f8c68000000-7f8c68021000 rw-p 00000000 00:00 0
7f8c68021000-7f8c6c000000 ---p 00000000 00:00 0
7f8c6c000000-7f8c6c021000 rw-p 00000000 00:00 0
7f8c6c021000-7f8c70000000 ---p 00000000 00:00 0
7f8c70000000-7f8c70021000 rw-p 00000000 00:00 0
7f8c70021000-7f8c74000000 ---p 00000000 00:00 0
7f8c74000000-7f8c74021000 rw-p 00000000 00:00 0
7f8c74021000-7f8c78000000 ---p 00000000 00:00 0
7f8c78000000-7f8c78021000 rw-p 00000000 00:00 0
7f8c78021000-7f8c7c000000 ---p 00000000 00:00 0
7f8c7c0bb000-7f8c7c0bd000 rw-p 00000000 00:00 0
7f8c7c0bd000-7f8c7c0c5000 r-xp 00000000 fd:00 135250                     /usr/java/jre1.8.0_45/lib/amd64/libmanagement.so
7f8c7c0c5000-7f8c7c2c5000 ---p 00008000 fd:00 135250                     /usr/java/jre1.8.0_45/lib/amd64/libmanagement.so
7f8c7c2c5000-7f8c7c2c6000 rw-p 00008000 fd:00 135250                     /usr/java/jre1.8.0_45/lib/amd64/libmanagement.so
7f8c7c2c6000-7f8c7c2d1000 r--s 00210000 fd:00 135273                     /usr/java/jre1.8.0_45/lib/ext/localedata.jar
7f8c7c2d1000-7f8c7c2f3000 r--s 0038e000 fd:00 135392                     /usr/java/jre1.8.0_45/lib/ext/cldrdata.jar
7f8c7c2f3000-7f8c7c304000 r-xp 00000000 fd:00 135253                     /usr/java/jre1.8.0_45/lib/amd64/libnio.so
7f8c7c304000-7f8c7c503000 ---p 00011000 fd:00 135253                     /usr/java/jre1.8.0_45/lib/amd64/libnio.so
7f8c7c503000-7f8c7c504000 rw-p 00010000 fd:00 135253                     /usr/java/jre1.8.0_45/lib/amd64/libnio.so
7f8c7c504000-7f8c7c508000 r--s 00085000 fd:00 135516                     /usr/java/jre1.8.0_45/lib/jsse.jar
7f8c7c508000-7f8c7c808000 rw-p 00000000 00:00 0
7f8c7c808000-7f8c7c81e000 r-xp 00000000 fd:00 135252                     /usr/java/jre1.8.0_45/lib/amd64/libnet.so
7f8c7c81e000-7f8c7ca1e000 ---p 00016000 fd:00 135252                     /usr/java/jre1.8.0_45/lib/amd64/libnet.so
7f8c7ca1e000-7f8c7ca1f000 rw-p 00016000 fd:00 135252                     /usr/java/jre1.8.0_45/lib/amd64/libnet.so
7f8c7ca1f000-7f8c7ca23000 r--s 00018000 fd:00 138818                     /usr/share/elasticsearch/lib/compiler-0.8.13.jar
7f8c7ca23000-7f8c7ca4a000 r--s 0021c000 fd:00 138832                     /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar
7f8c7ca4a000-7f8c7ca52000 r--s 00062000 fd:00 138831                     /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar
7f8c7ca52000-7f8c7ca5b000 r--s 00046000 fd:00 138826                     /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.2.jar
7f8c7ca5b000-7f8c7ca6c000 r--s 000b2000 fd:00 138830                     /usr/share/elasticsearch/lib/jts-1.13.jar
7f8c7ca6c000-7f8c7cb1d000 r--s 0084c000 fd:00 138820                     /usr/share/elasticsearch/lib/elasticsearch-2.3.2.jar
7f8c7cb1d000-7f8c7cb1e000 ---p 00000000 00:00 0
7f8c7cb1e000-7f8c7cc1e000 rw-p 00000000 00:00 0                          [stack:19951]
7f8c7cc1e000-7f8c7cc21000 ---p 00000000 00:00 0
7f8c7cc21000-7f8c7cd1f000 rw-p 00000000 00:00 0                          [stack:19950]
7f8c7cd1f000-7f8c7cd22000 ---p 00000000 00:00 0
7f8c7cd22000-7f8c7ce20000 rw-p 00000000 00:00 0                          [stack:19949]
7f8c7ce20000-7f8c7ce23000 ---p 00000000 00:00 0
7f8c7ce23000-7f8c7cf21000 rw-p 00000000 00:00 0                          [stack:19948]
7f8c7cf21000-7f8c7cf24000 ---p 00000000 00:00 0
7f8c7cf24000-7f8c7d022000 rw-p 00000000 00:00 0                          [stack:19947]
7f8c7d022000-7f8c7d025000 ---p 00000000 00:00 0
7f8c7d025000-7f8c7d123000 rw-p 00000000 00:00 0                          [stack:19946]
7f8c7d123000-7f8c7d126000 ---p 00000000 00:00 0
7f8c7d126000-7f8c7d224000 rw-p 00000000 00:00 0                          [stack:19945]
7f8c7d224000-7f8c7d227000 ---p 00000000 00:00 0
7f8c7d227000-7f8c7d325000 rw-p 00000000 00:00 0                          [stack:19944]
7f8c7d325000-7f8c7d326000 ---p 00000000 00:00 0
7f8c7d326000-7f8c7d426000 rw-p 00000000 00:00 0                          [stack:19943]
7f8c7d426000-7f8c7d5fe000 r--s 0341d000 fd:00 135563                     /usr/java/jre1.8.0_45/lib/rt.jar
7f8c7d5fe000-7f8c7dd3e000 rw-p 00000000 00:00 0
7f8c7dd3e000-7f8c7ddfe000 ---p 00000000 00:00 0
7f8c7ddfe000-7f8c7ddff000 ---p 00000000 00:00 0
7f8c7ddff000-7f8c80000000 rw-p 00000000 00:00 0                          [stack:19942]
7f8c80000000-7f8c80021000 rw-p 00000000 00:00 0
7f8c80021000-7f8c84000000 ---p 00000000 00:00 0
7f8c84000000-7f8c84002000 r--s 00019000 fd:00 135576                     /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar
7f8c84002000-7f8c84019000 r--s 00100000 fd:00 138822                     /usr/share/elasticsearch/lib/hppc-0.7.1.jar
7f8c84019000-7f8c8402c000 r--s 0016f000 fd:00 135585                     /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar
7f8c8402c000-7f8c88000000 rw-p 00000000 00:00 0
7f8c88000000-7f8c88021000 rw-p 00000000 00:00 0
7f8c88021000-7f8c8c000000 ---p 00000000 00:00 0
7f8c8c000000-7f8c8c001000 rw-p 00000000 00:00 0
7f8c8c001000-7f8c8c005000 r--s 00020000 fd:00 138834                     /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar
7f8c8c005000-7f8c8c009000 r--s 00035000 fd:00 138840                     /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar
7f8c8c009000-7f8c8c00b000 r--s 0000e000 fd:00 138829                     /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar
7f8c8c00b000-7f8c8c00e000 r--s 00018000 fd:00 138833                     /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar
7f8c8c00e000-7f8c8c027000 r--s 0012c000 fd:00 138844                     /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar
7f8c8c027000-7f8c8c052000 r--s 001fc000 fd:00 138821                     /usr/share/elasticsearch/lib/guava-18.0.jar
7f8c8c052000-7f8c8cfff000 rw-p 00000000 00:00 0
7f8c8cfff000-7f8c8d000000 rw-p 00000000 00:00 0
7f8c8d000000-7f8c8d270000 rwxp 00000000 00:00 0
7f8c8d270000-7f8c9c000000 ---p 00000000 00:00 0
7f8c9c000000-7f8c9c801000 rw-p 00000000 00:00 0
7f8c9c801000-7f8ca0000000 ---p 00000000 00:00 0
7f8ca0000000-7f8ca0002000 r--s 00008000 fd:00 135583                     /usr/share/elasticsearch/lib/joda-convert-1.2.jar
7f8ca0002000-7f8ca0008000 r--s 0003c000 fd:00 138846                     /usr/share/elasticsearch/lib/snakeyaml-1.15.jar
7f8ca0008000-7f8ca000c000 r--s 00027000 fd:00 138837                     /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar
7f8ca000c000-7f8ca0010000 r--s 0002d000 fd:00 138842                     /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar
7f8ca0010000-7f8ca0015000 r--s 00038000 fd:00 138843                     /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar
7f8ca0015000-7f8ca001a000 r--s 0003b000 fd:00 138841                     /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar
7f8ca001a000-7f8ca026b000 rw-p 00000000 00:00 0
7f8ca026b000-7f8ca026c000 ---p 00000000 00:00 0
7f8ca026c000-7f8ca036c000 rw-p 00000000 00:00 0                          [stack:19941]
7f8ca036c000-7f8ca036d000 ---p 00000000 00:00 0
7f8ca036d000-7f8ca0477000 rw-p 00000000 00:00 0                          [stack:19940]
7f8ca0477000-7f8ca082d000 ---p 00000000 00:00 0
7f8ca082d000-7f8ca0847000 r-xp 00000000 fd:00 135266                     /usr/java/jre1.8.0_45/lib/amd64/libzip.so
7f8ca0847000-7f8ca0a47000 ---p 0001a000 fd:00 135266                     /usr/java/jre1.8.0_45/lib/amd64/libzip.so
7f8ca0a47000-7f8ca0a48000 rw-p 0001a000 fd:00 135266                     /usr/java/jre1.8.0_45/lib/amd64/libzip.so
7f8ca0a48000-7f8ca0a54000 r-xp 00000000 fd:00 4084                       /usr/lib64/libnss_files-2.17.so
7f8ca0a54000-7f8ca0c53000 ---p 0000c000 fd:00 4084                       /usr/lib64/libnss_files-2.17.so
7f8ca0c53000-7f8ca0c54000 r--p 0000b000 fd:00 4084                       /usr/lib64/libnss_files-2.17.so
7f8ca0c54000-7f8ca0c55000 rw-p 0000c000 fd:00 4084                       /usr/lib64/libnss_files-2.17.so
7f8ca0c55000-7f8ca0c5b000 rw-p 00000000 00:00 0
7f8ca0c5b000-7f8ca0c85000 r-xp 00000000 fd:00 135231                     /usr/java/jre1.8.0_45/lib/amd64/libjava.so
7f8ca0c85000-7f8ca0e85000 ---p 0002a000 fd:00 135231                     /usr/java/jre1.8.0_45/lib/amd64/libjava.so
7f8ca0e85000-7f8ca0e87000 rw-p 0002a000 fd:00 135231                     /usr/java/jre1.8.0_45/lib/amd64/libjava.so
7f8ca0e87000-7f8ca0e94000 r-xp 00000000 fd:00 135265                     /usr/java/jre1.8.0_45/lib/amd64/libverify.so
7f8ca0e94000-7f8ca1094000 ---p 0000d000 fd:00 135265                     /usr/java/jre1.8.0_45/lib/amd64/libverify.so
7f8ca1094000-7f8ca1096000 rw-p 0000d000 fd:00 135265                     /usr/java/jre1.8.0_45/lib/amd64/libverify.so
7f8ca1096000-7f8ca109d000 r-xp 00000000 fd:00 4096                       /usr/lib64/librt-2.17.so
7f8ca109d000-7f8ca129c000 ---p 00007000 fd:00 4096                       /usr/lib64/librt-2.17.so
7f8ca129c000-7f8ca129d000 r--p 00006000 fd:00 4096                       /usr/lib64/librt-2.17.so
7f8ca129d000-7f8ca129e000 rw-p 00007000 fd:00 4096                       /usr/lib64/librt-2.17.so
7f8ca129e000-7f8ca139f000 r-xp 00000000 fd:00 4074                       /usr/lib64/libm-2.17.so
7f8ca139f000-7f8ca159e000 ---p 00101000 fd:00 4074                       /usr/lib64/libm-2.17.so
7f8ca159e000-7f8ca159f000 r--p 00100000 fd:00 4074                       /usr/lib64/libm-2.17.so
7f8ca159f000-7f8ca15a0000 rw-p 00101000 fd:00 4074                       /usr/lib64/libm-2.17.so
7f8ca15a0000-7f8ca224e000 r-xp 00000000 fd:00 135270                     /usr/java/jre1.8.0_45/lib/amd64/server/libjvm.so
7f8ca224e000-7f8ca244d000 ---p 00cae000 fd:00 135270                     /usr/java/jre1.8.0_45/lib/amd64/server/libjvm.so
7f8ca244d000-7f8ca2524000 rw-p 00cad000 fd:00 135270                     /usr/java/jre1.8.0_45/lib/amd64/server/libjvm.so
7f8ca2524000-7f8ca2568000 rw-p 00000000 00:00 0
7f8ca2568000-7f8ca271e000 r-xp 00000000 fd:00 4066                       /usr/lib64/libc-2.17.so
7f8ca271e000-7f8ca291e000 ---p 001b6000 fd:00 4066                       /usr/lib64/libc-2.17.so
7f8ca291e000-7f8ca2922000 r--p 001b6000 fd:00 4066                       /usr/lib64/libc-2.17.so
7f8ca2922000-7f8ca2924000 rw-p 001ba000 fd:00 4066                       /usr/lib64/libc-2.17.so
7f8ca2924000-7f8ca2929000 rw-p 00000000 00:00 0
7f8ca2929000-7f8ca292c000 r-xp 00000000 fd:00 4072                       /usr/lib64/libdl-2.17.so
7f8ca292c000-7f8ca2b2b000 ---p 00003000 fd:00 4072                       /usr/lib64/libdl-2.17.so
7f8ca2b2b000-7f8ca2b2c000 r--p 00002000 fd:00 4072                       /usr/lib64/libdl-2.17.so
7f8ca2b2c000-7f8ca2b2d000 rw-p 00003000 fd:00 4072                       /usr/lib64/libdl-2.17.so
7f8ca2b2d000-7f8ca2b42000 r-xp 00000000 fd:00 135209                     /usr/java/jre1.8.0_45/lib/amd64/jli/libjli.so
7f8ca2b42000-7f8ca2d42000 ---p 00015000 fd:00 135209                     /usr/java/jre1.8.0_45/lib/amd64/jli/libjli.so
7f8ca2d42000-7f8ca2d43000 rw-p 00015000 fd:00 135209                     /usr/java/jre1.8.0_45/lib/amd64/jli/libjli.so
7f8ca2d43000-7f8ca2d59000 r-xp 00000000 fd:00 4092                       /usr/lib64/libpthread-2.17.so
7f8ca2d59000-7f8ca2f59000 ---p 00016000 fd:00 4092                       /usr/lib64/libpthread-2.17.so
7f8ca2f59000-7f8ca2f5a000 r--p 00016000 fd:00 4092                       /usr/lib64/libpthread-2.17.so
7f8ca2f5a000-7f8ca2f5b000 rw-p 00017000 fd:00 4092                       /usr/lib64/libpthread-2.17.so
7f8ca2f5b000-7f8ca2f5f000 rw-p 00000000 00:00 0
7f8ca2f5f000-7f8ca2f80000 r-xp 00000000 fd:00 4059                       /usr/lib64/ld-2.17.so
7f8ca2f80000-7f8ca2f82000 r--s 00011000 fd:00 138825                     /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.2.jar
7f8ca2f82000-7f8ca2f84000 r--s 00007000 fd:00 138836                     /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar
7f8ca2f84000-7f8ca2f85000 r--s 00001000 fd:00 138845                     /usr/share/elasticsearch/lib/securesm-1.0.jar
7f8ca2f85000-7f8ca2f89000 r--s 000dc000 fd:00 138827                     /usr/share/elasticsearch/lib/jna-4.1.0.jar
7f8ca2f89000-7f8ca2f93000 r--s 00059000 fd:00 138839                     /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar
7f8ca2f93000-7f8ca2f95000 r--s 00012000 fd:00 138819                     /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar
7f8ca2f95000-7f8ca2f96000 r--s 0000c000 fd:00 135578                     /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar
7f8ca2f96000-7f8ca2f9d000 r--s 00037000 fd:00 138838                     /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar
7f8ca2f9d000-7f8ca2fa5000 r--s 00066000 fd:00 138817                     /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar
7f8ca2fa5000-7f8ca2fa8000 r--s 00022000 fd:00 138847                     /usr/share/elasticsearch/lib/spatial4j-0.5.jar
7f8ca2fa8000-7f8ca2fac000 r--s 0003c000 fd:00 138823                     /usr/share/elasticsearch/lib/jackson-core-2.6.2.jar
7f8ca2fac000-7f8ca2fbc000 r--s 00088000 fd:00 138828                     /usr/share/elasticsearch/lib/joda-time-2.8.2.jar
7f8ca2fbc000-7f8ca2fc5000 r--s 0006f000 fd:00 135584                     /usr/share/elasticsearch/lib/log4j-1.2.17.jar
7f8ca2fc5000-7f8ca3069000 rw-p 00000000 00:00 0
7f8ca3069000-7f8ca3071000 rw-s 00000000 fd:01 393218                     /tmp/hsperfdata_elasticsearch/19921
7f8ca3071000-7f8ca3074000 ---p 00000000 00:00 0
7f8ca3074000-7f8ca3176000 rw-p 00000000 00:00 0                          [stack:19939]
7f8ca3176000-7f8ca3178000 r--s 0000b000 fd:00 138848                     /usr/share/elasticsearch/lib/t-digest-3.0.jar
7f8ca3178000-7f8ca317c000 r--s 0001e000 fd:00 138835                     /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar
7f8ca317c000-7f8ca317d000 r--s 0000b000 fd:00 138824                     /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.2.jar
7f8ca317d000-7f8ca317e000 rw-p 00000000 00:00 0
7f8ca317e000-7f8ca317f000 r--p 00000000 00:00 0
7f8ca317f000-7f8ca3180000 rw-p 00000000 00:00 0
7f8ca3180000-7f8ca3181000 r--p 00021000 fd:00 4059                       /usr/lib64/ld-2.17.so
7f8ca3181000-7f8ca3182000 rw-p 00022000 fd:00 4059                       /usr/lib64/ld-2.17.so
7f8ca3182000-7f8ca3183000 rw-p 00000000 00:00 0
7ffcb0189000-7ffcb01aa000 rw-p 00000000 00:00 0                          [stack]
7ffcb01fc000-7ffcb01fe000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
jvm_args: -Xms4g -Xmx4g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Djna.debug_load.jna=true -Djna.debug_load=true -Djava.io.tmpdir=/export/apps/elasticsearch/tmp -Djna.tmpdir=/export/apps/elasticsearch/tmp -Des.path.home=/usr/share/elasticsearch
java_command: org.elasticsearch.bootstrap.Elasticsearch start -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch
java_class_path (initial): /usr/share/elasticsearch/lib/elasticsearch-2.3.2.jar:/usr/share/elasticsearch/lib/elasticsearch-2.3.2.jar:/usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.2.jar:/usr/share/elasticsearch/lib/log4j-1.2.17.jar:/usr/share/elasticsearch/lib/lucene-join-5.5.0.jar:/usr/share/elasticsearch/lib/joda-time-2.8.2.jar:/usr/share/elasticsearch/lib/jackson-core-2.6.2.jar:/usr/share/elasticsearch/lib/t-digest-3.0.jar:/usr/share/elasticsearch/lib/spatial4j-0.5.jar:/usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar:/usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar:/usr/share/elasticsearch/lib/commons-cli-1.3.1.jar:/usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar:/usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar:/usr/share/elasticsearch/lib/jna-4.1.0.jar:/usr/share/elasticsearch/lib/securesm-1.0.jar:/usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar:/usr/share/elasticsearch/lib/guava-18.0.jar:/usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar:/usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.2.jar:/usr/share/elasticsearch/lib/netty-3.10.5.Final.jar:/usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar:/usr/share/elasticsearch/lib/snakeyaml-1.15.jar:/usr/share/elasticsearch/lib/joda-convert-1.2.jar:/usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar:/usr/share/elasticsearch/lib/jsr166e-1.1.0.jar:/usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar:/usr/share/elasticsearch/lib/hppc-0.7.1.jar:/usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar:/usr/share/elasticsearch/lib/jts-1.13.jar:/usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.2.jar:/usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar:/usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-core-5.5.0.jar:/usr/share/elasticsearch/lib/c
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
SHELL=/sbin/nologin

Signal Handlers:
SIGSEGV: [libjvm.so+0xaad1e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xaad1e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x90b450], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x90b450], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x90b450], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x90b450], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x90ca90], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.so+0x90dde0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: [libjvm.so+0x90dde0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGTERM: [libjvm.so+0x90dde0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0x90dde0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO


---------------  S Y S T E M  ---------------

OS:CentOS Linux release 7.2.1511 (Core)

uname:Linux 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 31854, NOFILE 65535, AS infinity
load average:0.25 0.43 0.29

/proc/meminfo:
MemTotal:        7912632 kB
MemFree:         1366836 kB
MemAvailable:    7015480 kB
Buffers:          116304 kB
Cached:          5265648 kB
SwapCached:         4928 kB
Active:          1525396 kB
Inactive:        4327224 kB
Active(anon):     270016 kB
Inactive(anon):   248668 kB
Active(file):    1255380 kB
Inactive(file):  4078556 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        999420 kB
SwapFree:         935612 kB
Dirty:                72 kB
Writeback:             0 kB
AnonPages:        430132 kB
Mapped:            47044 kB
Shmem:             47952 kB
Slab:             607060 kB
SReclaimable:     568100 kB
SUnreclaim:        38960 kB
KernelStack:        3504 kB
PageTables:         8760 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4955736 kB
Committed_AS:    5185924 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       23160 kB
VmallocChunk:   34359710460 kB
HardwareCorrupted:     0 kB
AnonHugePages:    354304 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       43008 kB
DirectMap2M:     8345600 kB


CPU:total 2 (1 cores per cpu, 2 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, ht, tsc, bmi1, bmi2

/proc/cpuinfo:
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
stepping    : 2
microcode   : 0x2b
cpu MHz     : 2400.100
cache size  : 30720 KB
physical id : 0
siblings    : 2
core id     : 0
cpu cores   : 1
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt
bogomips    : 4800.20
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
stepping    : 2
microcode   : 0x2b
cpu MHz     : 2400.100
cache size  : 30720 KB
physical id : 0
siblings    : 2
core id     : 0
cpu cores   : 1
apicid      : 1
initial apicid  : 1
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt
bogomips    : 4800.20
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 7912632k(1366836k free), swap 999420k(935612k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.45-b02) for linux-amd64 JRE (1.8.0_45-b14), built on Apr 10 2015 10:07:45 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

time: Mon May 23 15:16:48 2016
elapsed time: 0 seconds (0d 0h 0m 0s)

[bubo77]

I've personally tried with setenforce 0 but nothing.
what is interesting that data dire and tmp points to the same directory and while data dir created and used without problem tmp cannot be for some reason