Skip to content

Commit e8c7bfc

Browse files
committed
remove export PYSPARK_PYTHON in spark submit
1 parent 3334169 commit e8c7bfc

File tree

1 file changed

+0
-16
lines changed

1 file changed

+0
-16
lines changed

bin/spark-submit

Lines changed: 0 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -37,22 +37,6 @@ done
3737

3838
DEPLOY_MODE=${DEPLOY_MODE:-"client"}
3939

40-
41-
# This is a hack to make DStream.pyprint work.
42-
# This will be removed after pyprint is moved to PythonDStream.
43-
# Problem is that print function is in (Scala)DStream.
44-
# Whenever python code is executed, we call PythonDStream which passes
45-
# pythonExec(which python Spark should execute). pythonExec is used to call python.
46-
# Since pyprint is located in DStream, Spark does not know which python should use.
47-
# In that case, get python path from PYSPARK_PYTHON, environmental variable.
48-
49-
# Figure out which Python executable to use
50-
if [[ -z "$PYSPARK_PYTHON" ]]; then
51-
PYSPARK_PYTHON="python"
52-
fi
53-
export PYSPARK_PYTHON
54-
55-
5640
if [ -n "$DRIVER_MEMORY" ] && [ $DEPLOY_MODE == "client" ]; then
5741
export SPARK_DRIVER_MEMORY=$DRIVER_MEMORY
5842
fi

0 commit comments

Comments
 (0)