[SPARK-18645][Deploy] Fix spark-daemon.sh arguments error lead to throws Unrecognized option #16079

wangyum · 2016-11-30T10:50:51Z

What changes were proposed in this pull request?

spark-daemon.sh will lost single quotes around after #15338. as follows:

execute_command nice -n 0 bash /opt/cloudera/parcels/SPARK-2.1.0-cdh5.4.3.d20161129-21.04.38/lib/spark/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift JDBC/ODBC Server --conf spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp

With this fix, as follows:

execute_command nice -n 0 bash /opt/cloudera/parcels/SPARK-2.1.0-cdh5.4.3.d20161129-21.04.38/lib/spark/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name 'Thrift JDBC/ODBC Server' --conf 'spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp'

How was this patch tested?

Manual tests
Build the package and start-thriftserver.sh with --conf 'spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp'

srowen · 2016-11-30T12:19:30Z

sbin/spark-daemon.sh

  if [ -z ${SPARK_NO_DAEMONIZE+set} ]; then
-      nohup -- $command >> $log 2>&1 < /dev/null &
+      nohup -- "$@" >> $log 2>&1 < /dev/null &


It's OK by me, but why remove the local?

It will lost single quotes around if use local. You can verify as follows:

#!/usr/bin/env bash set -x execute_command() { echo "with local:" local command="$@" echo "no local:" "$@" } run_command() { execute_command ../bin/spark-submit "$@" } run_command "$@"

Save above code as SPARK-18645.sh and run sh SPARK-18645.sh --conf 'spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp'
you will see like this:

+ run_command --conf 'spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp' + execute_command ../bin/spark-submit --conf 'spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp' + echo 'with local:' with local: + local 'command=../bin/spark-submit --conf spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp' + echo 'no local:' no local: + ../bin/spark-submit --conf 'spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp' ...

Oh I see, OK.

srowen · 2016-11-30T12:19:52Z

sbin/spark-daemon.sh

      ;;

    (submit)
-      execute_command nice -n "$SPARK_NICENESS" bash "${SPARK_HOME}"/bin/spark-submit --class $command $@
+      execute_command nice -n "$SPARK_NICENESS" bash "${SPARK_HOME}"/bin/spark-submit --class $command "$@"


Can $command have spaces? not sure it can but might not hurt to quote it

Can not have spaces. I have quoted it and tested on my local machine.

SparkQA · 2016-11-30T12:46:35Z

Test build #69404 has finished for PR 16079 at commit 563645f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
execute_command nice -n \"$SPARK_NICENESS\" \"$
execute_command nice -n \"$SPARK_NICENESS\" bash \"$

srowen · 2016-11-30T13:20:30Z

Jenkins retest this please

SparkQA · 2016-11-30T15:56:01Z

Test build #69408 has finished for PR 16079 at commit 563645f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
execute_command nice -n \"$SPARK_NICENESS\" \"$
execute_command nice -n \"$SPARK_NICENESS\" bash \"$

SparkQA · 2016-11-30T17:31:20Z

Test build #69410 has finished for PR 16079 at commit 9f67fd8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
execute_command nice -n \"$SPARK_NICENESS\" \"$
execute_command nice -n \"$SPARK_NICENESS\" bash \"$

…ows Unrecognized option ## What changes were proposed in this pull request? spark-daemon.sh will lost single quotes around after #15338. as follows: ``` execute_command nice -n 0 bash /opt/cloudera/parcels/SPARK-2.1.0-cdh5.4.3.d20161129-21.04.38/lib/spark/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift JDBC/ODBC Server --conf spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp ``` With this fix, as follows: ``` execute_command nice -n 0 bash /opt/cloudera/parcels/SPARK-2.1.0-cdh5.4.3.d20161129-21.04.38/lib/spark/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name 'Thrift JDBC/ODBC Server' --conf 'spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp' ``` ## How was this patch tested? - Manual tests - Build the package and start-thriftserver.sh with `--conf 'spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp'` Author: Yuming Wang <wgyumg@gmail.com> Closes #16079 from wangyum/SPARK-18645. (cherry picked from commit 2ab8551) Signed-off-by: Sean Owen <sowen@cloudera.com>

srowen · 2016-12-01T13:26:45Z

Merged to master/2.1. This doesn't actually affect earlier versions because it seems to be introduced by https://issues.apache.org/jira/browse/SPARK-11653 I didn't realize the 'local' would have this effect either.

…ows Unrecognized option ## What changes were proposed in this pull request? spark-daemon.sh will lost single quotes around after apache#15338. as follows: ``` execute_command nice -n 0 bash /opt/cloudera/parcels/SPARK-2.1.0-cdh5.4.3.d20161129-21.04.38/lib/spark/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift JDBC/ODBC Server --conf spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp ``` With this fix, as follows: ``` execute_command nice -n 0 bash /opt/cloudera/parcels/SPARK-2.1.0-cdh5.4.3.d20161129-21.04.38/lib/spark/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name 'Thrift JDBC/ODBC Server' --conf 'spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp' ``` ## How was this patch tested? - Manual tests - Build the package and start-thriftserver.sh with `--conf 'spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp'` Author: Yuming Wang <wgyumg@gmail.com> Closes apache#16079 from wangyum/SPARK-18645.

Fix Unrecognized option

563645f

srowen reviewed Nov 30, 2016

View reviewed changes

Quote $command

9f67fd8

asfgit closed this in 2ab8551 Dec 1, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-18645][Deploy] Fix spark-daemon.sh arguments error lead to throws Unrecognized option #16079

[SPARK-18645][Deploy] Fix spark-daemon.sh arguments error lead to throws Unrecognized option #16079

Uh oh!

wangyum commented Nov 30, 2016

Uh oh!

srowen Nov 30, 2016

Uh oh!

wangyum Nov 30, 2016 •

edited

Loading

Uh oh!

srowen Nov 30, 2016

Uh oh!

srowen Nov 30, 2016

Uh oh!

wangyum Nov 30, 2016

Uh oh!

SparkQA commented Nov 30, 2016

Uh oh!

srowen commented Nov 30, 2016

Uh oh!

SparkQA commented Nov 30, 2016

Uh oh!

SparkQA commented Nov 30, 2016

Uh oh!

srowen commented Dec 1, 2016

Uh oh!

Uh oh!

[SPARK-18645][Deploy] Fix spark-daemon.sh arguments error lead to throws Unrecognized option #16079

[SPARK-18645][Deploy] Fix spark-daemon.sh arguments error lead to throws Unrecognized option #16079

Uh oh!

Conversation

wangyum commented Nov 30, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

srowen Nov 30, 2016

Choose a reason for hiding this comment

Uh oh!

wangyum Nov 30, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

srowen Nov 30, 2016

Choose a reason for hiding this comment

Uh oh!

srowen Nov 30, 2016

Choose a reason for hiding this comment

Uh oh!

wangyum Nov 30, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 30, 2016

Uh oh!

srowen commented Nov 30, 2016

Uh oh!

SparkQA commented Nov 30, 2016

Uh oh!

SparkQA commented Nov 30, 2016

Uh oh!

srowen commented Dec 1, 2016

Uh oh!

Uh oh!

wangyum Nov 30, 2016 •

edited

Loading