Skip to content

Commit 46b7c3e

Browse files
authored
Update readme.md
1 parent 2cf5ea3 commit 46b7c3e

File tree

1 file changed

+13
-0
lines changed

1 file changed

+13
-0
lines changed

readme.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,19 @@ as it leverages the Docker image
1010
[dclong/python-portable](https://github.com/dclong/docker-python-portable)
1111
to build portable Python environments.
1212

13+
## Tips on Portable Python Environment for PySpark
14+
15+
The Python library `pyspark` is not needed as Spark has a pyspark library come with it
16+
and it is used when you submit PySpark applications to the cluster.
17+
Some Python libraries might have the Python library `pyspark` as one of its dependencies.
18+
If you have such a library installed
19+
and thus have the Python library `pyspark` installed into the portable Python environment,
20+
you can simply remove the `pyspark` library.
21+
22+
:::bash
23+
rm -rf /opt/python/lib/site-packages/pyspark*
24+
25+
1326
## Pre-developed Scripts for Building and Packing Portable Python
1427

1528
| Script | Build Command | Output Env | Comments |

0 commit comments

Comments
 (0)