Skip to content

Commit

Permalink
[SPARK-45706][PYTHON][DOCS] Makes entire Binder build fails fast duri…
Browse files Browse the repository at this point in the history
…ng setting up

### What changes were proposed in this pull request?

This PR proposes to make entire Binder build fails fast during setting up to prevent the Binder image to be successfully built which it cannot be rebuilt later on the same commit.

### Why are the changes needed?

Binder build is currently broken for Spark 3.5.0:

https://mybinder.org/v2/gh/apache/spark/ce5ddad9903?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart_df.ipynb

Seems like we uploaded PySpark late into PyPI, and the installation steps just slightly ignored the failure (a user triggered the first docker image for Binder, and that's being reused at that time PySpark wasn't uploaded to PyPI).

![Screenshot 2023-10-27 at 5 42 26 PM](https://github.com/apache/spark/assets/6477701/9030e4a1-2afa-43a2-aee0-dda01abb46ce)

### Does this PR introduce _any_ user-facing change?

Yes, it fixes the user-facing live notebooks (at https://spark.apache.org/docs/latest/api/python/index.html).

### How was this patch tested?

Manually tested in my fork:
- https://mybinder.org/v2/gh/HyukjinKwon/spark/ce5ddad9903?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart_df.ipynb
- https://mybinder.org/v2/gh/HyukjinKwon/spark/ce5ddad9903?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart_connect.ipynb
- https://mybinder.org/v2/gh/HyukjinKwon/spark/ce5ddad9903?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart_ps.ipynb
### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#43553 from HyukjinKwon/SPARK-45706.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
HyukjinKwon committed Oct 27, 2023
1 parent efa891c commit f1d1dc1
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions binder/postBuild
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@
# This file is used for Binder integration to install PySpark available in
# Jupyter notebook.

# SPARK-45706: Should fail fast. Otherwise, the Binder image is successfully
# built, and it cannot be rebuilt.
set -o pipefail
set -e

VERSION=$(python -c "exec(open('python/pyspark/version.py').read()); print(__version__)")
TAG=$(git describe --tags --exact-match 2>/dev/null)

Expand Down

0 comments on commit f1d1dc1

Please sign in to comment.