Skip to content

Commit 2f812c0

Browse files
committed
[SPARK-39861][PYTHON][DOCS] Deprecate Python 3.7 Support
### What changes were proposed in this pull request? This PR aims to deprecate Python 3.7 support at Apache Spark 3.4.0. ### Why are the changes needed? Apache Spark 3.4 will be released on February 2023 and will be supported for next 18 months. - https://spark.apache.org/versioning-policy.html Before Spark 3.4.1, Python 3.7 is going to reach `End Of Support` on 2023-06-27. Although Apache Spark 3.4 will work with Python 3.7 still for a while, there is no proper and official way to support Python 3.7 from that time. - https://www.python.org/downloads/ ### Does this PR introduce _any_ user-facing change? Yes, but the users will see only a deprecation warning log and docs. ### How was this patch tested? Pass the CIs. Closes #37279 from dongjoon-hyun/SPARK-39861. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 8628c15 commit 2f812c0

File tree

3 files changed

+10
-0
lines changed

3 files changed

+10
-0
lines changed

docs/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ source, visit [Building Spark](building-spark.html).
4040
Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS), and it should run on any platform that runs a supported version of Java. This should include JVMs on x86_64 and ARM64. It's easy to run locally on one machine --- all you need is to have `java` installed on your system `PATH`, or the `JAVA_HOME` environment variable pointing to a Java installation.
4141

4242
Spark runs on Java 8/11/17, Scala 2.12/2.13, Python 3.7+ and R 3.5+.
43+
Python 3.7 support is deprecated as of Spark 3.4.0.
4344
Java 8 prior to version 8u201 support is deprecated as of Spark 3.2.0.
4445
For the Scala API, Spark {{site.SPARK_VERSION}}
4546
uses Scala {{site.SCALA_BINARY_VERSION}}. You will need to use a compatible Scala version

docs/rdd-programming-guide.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,7 @@ so C libraries like NumPy can be used. It also works with PyPy 7.3.6+.
106106

107107
Python 2, 3.4 and 3.5 supports were removed in Spark 3.1.0.
108108
Python 3.6 support was removed in Spark 3.3.0.
109+
Python 3.7 support is deprecated as of Spark 3.4.0.
109110

110111
Spark applications in Python can either be run with the `bin/spark-submit` script which includes Spark at runtime, or by including it in your setup.py as:
111112

python/pyspark/context.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -306,6 +306,14 @@ def _do_init(
306306
self.pythonExec = os.environ.get("PYSPARK_PYTHON", "python3")
307307
self.pythonVer = "%d.%d" % sys.version_info[:2]
308308

309+
if sys.version_info[:2] < (3, 8):
310+
with warnings.catch_warnings():
311+
warnings.simplefilter("once")
312+
warnings.warn(
313+
"Python 3.7 support is deprecated in Spark 3.4.",
314+
FutureWarning
315+
)
316+
309317
# Broadcast's __reduce__ method stores Broadcast instances here.
310318
# This allows other code to determine which Broadcast instances have
311319
# been pickled, so it can determine which Java broadcast objects to

0 commit comments

Comments
 (0)