We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
1 parent 72bfc66 commit a7a0b5cCopy full SHA for a7a0b5c
bin/spark-submit
@@ -37,6 +37,16 @@ done
37
38
DEPLOY_MODE=${DEPLOY_MODE:-"client"}
39
40
+
41
+# This is a hack to make DStream.pyprint work.
42
+# This will be removed after pyprint is moved to PythonDStream.
43
+# Problem is that print function is in (Scala)DStream.
44
+# Whenever python code is executed, we call PythonDStream which passes
45
+# pythonExec(which python Spark should execute).
46
+# Since pyprint is located in DStream, Spark does not know which python should use.
47
+# In that case, get python path from PYSPARK_PYTHON, environmental variable.
48
+# This fix is ongoing in print branch in my repo.
49
50
# Figure out which Python executable to use
51
if [[ -z "$PYSPARK_PYTHON" ]]; then
52
PYSPARK_PYTHON="python"
0 commit comments