Skip to content

Commit

Permalink
[SPARK-49720][PYTHON][INFRA] Add a script to clean up PySpark temp files
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
Add a script to clean up PySpark temp files

### Why are the changes needed?
Sometimes I encounter weird issues due to the out-dated `pyspark.zip` file, and removing it can result in expected behavior.
So I think we can add such a script.

### Does this PR introduce _any_ user-facing change?
no, dev-only

### How was this patch tested?
manually test

### Was this patch authored or co-authored using generative AI tooling?
no

Closes apache#48167 from zhengruifeng/py_infra_cleanup.

Authored-by: Ruifeng Zheng <ruifengz@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
zhengruifeng authored and attilapiros committed Oct 4, 2024
1 parent 75d96ad commit 232e941
Showing 1 changed file with 31 additions and 0 deletions.
31 changes: 31 additions & 0 deletions dev/py-cleanup
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/usr/bin/env bash

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# Utility for temporary files cleanup in 'python'.
# usage: ./dev/py-cleanup

set -ex

SPARK_HOME="$(cd "`dirname $0`"/..; pwd)"
cd "$SPARK_HOME"

rm -rf python/target
rm -rf python/lib/pyspark.zip
rm -rf python/docs/build
rm -rf python/docs/source/reference/*/api

0 comments on commit 232e941

Please sign in to comment.