File tree Expand file tree Collapse file tree 1 file changed +13
-0
lines changed Expand file tree Collapse file tree 1 file changed +13
-0
lines changed Original file line number Diff line number Diff line change @@ -10,6 +10,19 @@ as it leverages the Docker image
10
10
[ dclong/python-portable] ( https://github.com/dclong/docker-python-portable )
11
11
to build portable Python environments.
12
12
13
+ ## Tips on Portable Python Environment for PySpark
14
+
15
+ The Python library ` pyspark ` is not needed as Spark has a pyspark library come with it
16
+ and it is used when you submit PySpark applications to the cluster.
17
+ Some Python libraries might have the Python library ` pyspark ` as one of its dependencies.
18
+ If you have such a library installed
19
+ and thus have the Python library ` pyspark ` installed into the portable Python environment,
20
+ you can simply remove the ` pyspark ` library.
21
+
22
+ :::bash
23
+ rm -rf /opt/python/lib/site-packages/pyspark*
24
+
25
+
13
26
## Pre-developed Scripts for Building and Packing Portable Python
14
27
15
28
| Script | Build Command | Output Env | Comments |
You can’t perform that action at this time.
0 commit comments