elasticsearch fails to start if temp dir path contains space

[sns-seb]

Elasticsearch version (bin/elasticsearch --version):

6.6.1

JVM version (java -version):

Java 8 (1.8.0_121), Java 9 (build 9+181), Java 10 (10.0.2), Java 11 (11.0.2)

OS version (uname -a if on a Unix-like system):

Ubuntu 18.04.1, but probably reproducible on any linux system

Description of the problem including expected versus actual behavior:

problem: If path to temp directory contains space, elasticsearch fails to start.
expected behavior: elastissearch starts and uses specified temp directory

Steps to reproduce:

Problem can be reproduced using both env variable ES_TMPDIR and jvm option java.io.tmpdir in jvm.options file

To reproduce with env variable:

1. unpack elastic tar package into /tmp/elasticsearch directory
2. create temp directory to be used by elasticsearch: mkdir -p "/tmp/has space/temp_for_es"
3. start ES specifying env variable ES_TMPDIR:

export ES_TMP_DIR='/tmp/temp_for_es' && /tmp/elasticsearch/bin/elasticsearch

4. startup fails with following error message:

Error: Could not find or load main class space.temp_for_es

To reproduce with jvm.options file:

1. unpack elastic tar package into /tmp/elasticsearch directory
2. create temp directory to be used by elasticsearch: mkdir -p "/tmp/has space/temp_for_es"
3. change value of property java.io.tmpdir in from -Djava.io.tmpdir=${ES_TMPDIR} to -Djava.io.tmpdir=/tmp/has space/temp_for_es
4. start ES:

/tmp/elasticsearch/bin/elasticsearch

4. startup fails with following error message:

Error: Could not find or load main class space.temp_for_es

Comment on the problem
The cause of the error is that the command line generated by the bin/elastissearch script does not correctly escape jvm options.

bin/elastissearch run with -x will display command line such as the following:

exec /opt/java/1.8/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/has space/temp_for_es -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/tmp/elasticsearch -Des.path.conf=/tmp/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=tar -cp '/tmp/elasticsearch/lib/*' org.elasticsearch.bootstrap.Elasticsearch

where we see that jvm option java.io.tmpDir is not escaped:
-Djava.io.tmpdir=/tmp/has space/temp_for_es.

This is interpreted by the java program as "/tmp/has" being the value of java.io.tmpdir option and space/temp_for_es the java class to be executed.

[elasticmachine]

Pinging @elastic/es-core-infra

[sns-seb]

Hi,

Since I had to write a work around to this problem to move on, I can share how I believe this can be fixed.

I don't have an extensive fix to share as I didn't write a full patch to elasticsearch given that, in my specific conditions, simply parsing the jvm.options file in bash was good enough. I could work on it later if the approach is approved.

Three actions are required:

1. JvmOptionsParser returns options separated by line return instead of space
   • this will of course have an impact on Windows batch scripts which parse the same output. I'm going to assume splitting based on line return can also be achieved on windows.
   • alternatively, JvmOptionsParser could generate a temporary file
2. output of JvmOptionsParser is parsed into an array with code such as the following:

# assuming here options are separated by line returns
JVM_OPTIONS_STRING=`"$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JvmOptionsParser "$ES_JVM_OPTIONS"`
JVM_OPTIONS=()
# "|| [[ "$line" ]]" is required to support string not ending with an empty line
while read -r line || [[ "$line" ]]; do
  # ignore empty or comment lines
  if [[ -z "$line" ]] || [[ $line =~ ^#.* ]]; then
    continue
  fi
  JVM_OPTIONS+=("${line//\$\{ES_TMPDIR\}/$ES_TMPDIR}")
done <<< "$JVM_OPTIONS_STRING"

3. concatenating ES_JAVA_OPTS to JVM_OPTIONS into variable ES_JAVA_OPTS is dropped and instead, both variable are added directly to exec. Most importantly, using syntax "${JVM_OPTIONS[@]}" will allow leveraging built-in escaping of each JVM option individually if needed:

  exec \
    "$JAVA" \
    "${JVM_OPTIONS[@]}" \
    ${ES_JAVA_OPTS} \
    -Des.path.home="$ES_HOME" \
    -Des.path.conf="$ES_PATH_CONF" \
    -Des.distribution.flavor="$ES_DISTRIBUTION_FLAVOR" \
    -Des.distribution.type="$ES_DISTRIBUTION_TYPE" \
    -cp "$ES_CLASSPATH" \
    org.elasticsearch.bootstrap.Elasticsearch \
    "$@"

[sagarpatr]

I am getting an error when I am trying to start sonar as a normat user. pfb error message
--> Wrapper Started as Daemon
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.

2020.01.16 15:06:03 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonar/temp
2020.01.16 15:06:03 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2020.01.16 15:06:03 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonar/elasticsearch]: /opt/sonar/elasticsearch/bin/elasticsearch
2020.01.16 15:06:03 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2020.01.16 15:06:03 INFO app[][o.e.p.PluginsService] no modules loaded
2020.01.16 15:06:03 INFO app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2020.01.16 15:06:07 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 1
2020.01.16 15:06:07 INFO app[][o.s.a.SchedulerImpl] Process [es] is stopped
2020.01.16 15:06:07 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped

[gwbrown]

I've confirmed that this is still present in 7.10.0, although by default the variable is ES_TMPDIR, not ES_TMP_DIR.

[ldematte]

We changed dramatically how we launch Elasticsearch in 8.3 (mid 2022): #85758
This issue should not be applicable anymore, but if you find any problem with the new java-based launcher, feel free to re-open (or open a new one). Thanks!