Add docs on JNA temp directory not being noexec (#35355)

If the underlying mount point for the JNA temporary directory is mounted
noexec on Linux, then the JVM will not be able to map the native code in
as executable. This will prevent JNA from executing and will prevent
Elasticsearch from being able to execute some functions that rely on
native code (e.g., memory locking, and installing system call
filters). We do not want to get into the business of catching exceptions
and parsing messages towards this because these exception messages can
change on us. We also do not want to jump through a lot of hoops to
check the underlying mount point for noexec. Instead, we will rely on
documentation to address this problem. This commit adds to the important
system configuration section of the docs that the JNA temporary
directory is not on a mount point with the noexec mount option.

Author: jasontedor

docs/reference/setup/sysconfig.asciidoc

@@ -13,6 +13,7 @@ The following settings *must* be considered before going to production:
 * <<vm-max-map-count,Ensure sufficient virtual memory>>
 * <<max-number-of-threads,Ensure sufficient threads>>
 * <<networkaddress-cache-ttl,JVM DNS cache settings>>
+* <<executable-jna-tmpdir,Temporary directory not mounted with `noexec`>>
 
 [[dev-vs-prod]]
 [float]
@@ -40,3 +41,5 @@ include::sysconfig/virtual-memory.asciidoc[]
 include::sysconfig/threads.asciidoc[]
 
 include::sysconfig/dns-cache.asciidoc[]
+
+include::sysconfig/executable-jna-tmpdir.asciidoc[]

docs/reference/setup/sysconfig/executable-jna-tmpdir.asciidoc

@@ -0,0 +1,24 @@
+[[executable-jna-tmpdir]]
+=== JNA temporary directory not mounted with `noexec`
+
+[NOTE]
+This is only relevant for Linux.
+
+Elasticsearch uses the Java Native Access (JNA) library for executing some
+platform-dependent native code. On Linux, the native code backing this library
+is extracted at runtime from the JNA archive. By default, this code is extracted
+to the Elasticsearch temporary directory which defaults to a sub-directory of
+`/tmp`. Alternatively, this location can be controlled with the JVM flag
+`-Djna.tmpdir=<path>`. As the native library is mapped into the JVM virtual
+address space as executable, the underlying mount point of the location that
+this code is extracted to must *not* be mounted with `noexec` as this prevents
+the JVM process from being able to map this code as executable. On some hardened
+Linux installations this is a default mount option for `/tmp`. One indication
+that the underlying mount is mounted with `noexec` is that at startup JNA will
+fail to load with a `java.lang.UnsatisfiedLinkerError` exception with a message
+along the lines of `failed to map segment from shared object`.  Note that the
+exception message can differ amongst JVM versions. Additionally, the components
+of Elasticsearch that rely on execution of native code via JNA will fail with
+messages indicating that it is `because JNA is not available`. If you are seeing
+such error messages, you must remount the temporary directory used for JNA to
+not be mounted with `noexec`.

docs/reference/setup/sysconfig/swap.asciidoc

@@ -95,13 +95,14 @@ Systems using `systemd`::
 
   Set `LimitMEMLOCK` to `infinity` in the <<systemd,systemd configuration>>.
 
-Another possible reason why `mlockall` can fail is that the temporary directory
-(usually `/tmp`) is mounted with the `noexec` option. This can be solved by
-specifying a new temp directory using the `ES_JAVA_OPTS` environment variable:
+Another possible reason why `mlockall` can fail is that
+<<executable-jna-tmpdir,the JNA temporary directory (usually a sub-directory of
+`/tmp`) is mounted with the `noexec` option>>. This can be solved by specifying
+a new temporary directory for JNA using the `ES_JAVA_OPTS` environment variable:
 
 [source,sh]
 --------------
-export ES_JAVA_OPTS="$ES_JAVA_OPTS -Djava.io.tmpdir=/path/to/temp/dir"
+export ES_JAVA_OPTS="$ES_JAVA_OPTS -Djna.tmpdir=<path>"
 ./bin/elasticsearch
 --------------