[SPARK-32017][PYTHON][BUILD] Make Pyspark Hadoop 3.2+ Variant available in PyPI #29703

HyukjinKwon · 2020-09-10T04:29:24Z

What changes were proposed in this pull request?

This PR proposes to add a way to select Hadoop and Hive versions in pip installation.
Users can select Hive or Hadoop versions as below:

HADOOP_VERSION=3.2 pip install pyspark
HIVE_VERSION=1.2 pip install pyspark
HIVE_VERSION=1.2 HADOOP_VERSION=2.7 pip install pyspark

When the environment variables are set, internally it downloads the corresponding Spark version and then sets the Spark home to it. Also this PR exposes a mirror to set as an environment variable, PYSPARK_RELEASE_MIRROR.

Please NOTE that:

We cannot currently leverage pip's native installation option, for example:
```
pip install pyspark --install-option="hadoop3.2"
```
This is because of a limitation and bug in pip itself. Once they fix this issue, we can switch from the environment variables to the proper installation options, see SPARK-32837.

It IS possible to workaround but very ugly or hacky with a big change. See this PR as an example.
In pip installation, we pack the relevant jars together. This PR does not touch existing packaging way in order to prevent any behaviour changes.

Once this experimental way is proven to be safe, we can avoid packing the relevant jars together (and keep only the relevant Python scripts). And downloads the Spark distribution as this PR proposes.
This way is sort of consistent with SparkR:

SparkR provides a method SparkR::install.spark to support CRAN installation. This is fine because SparkR is provided purely as a R library. For example, sparkr script is not packed together.

PySpark cannot take this approach because PySpark packaging ships relevant executable script together, e.g.) pyspark shell.

If PySpark has a method such as pyspark.install_spark, users cannot call it in pyspark because pyspark already assumes relevant Spark is installed, JVM is launched, etc.
There looks no way to release that contains different Hadoop or Hive to PyPI due to the version semantics. This is not an option.

The usual way looks either --install-option above with hacks or environment variables given my investigation.

Why are the changes needed?

To provide users the options to select Hadoop and Hive versions.

Does this PR introduce any user-facing change?

Yes, users will be able to select Hive and Hadoop version as below when they install it from pip;

HADOOP_VERSION=3.2 pip install pyspark
HIVE_VERSION=1.2 pip install pyspark
HIVE_VERSION=1.2 HADOOP_VERSION=2.7 pip install pyspark

How was this patch tested?

Unit tests were added. I also manually tested in Mac and Windows (after building Spark with python/dist/pyspark-3.1.0.dev0.tar.gz):

./build/mvn -DskipTests -Phive-thriftserver clean package

Mac:

SPARK_VERSION=3.0.1 HADOOP_VERSION=3.2 pip install pyspark-3.1.0.dev0.tar.gz

Windows:

set HADOOP_VERSION=3.2
set SPARK_VERSION=3.0.1
pip install pyspark-3.1.0.dev0.tar.gz

HyukjinKwon · 2020-09-10T04:32:18Z

python/pyspark/install.py

+    return spark_version, hadoop_version, hive_version
+
+
+def install_spark(dest, spark_version, hadoop_version, hive_version):


I basically referred to

spark/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala

Lines 70 to 111 in b84ed41

private def tryDownloadSpark(version: String, path: String): Unit = {

// Try a few mirrors first; fall back to Apache archive

val mirrors =

(0 until 2).flatMap { _ =>

try {

Some(getStringFromUrl("https://www.apache.org/dyn/closer.lua?preferred=true"))

} catch {

// If we can't get a mirror URL, skip it. No retry.

case _: Exception => None

}

}

val sites =

mirrors.distinct :+ "https://archive.apache.org/dist" :+ PROCESS_TABLES.releaseMirror

logInfo(s"Trying to download Spark $version from $sites")

for (site <- sites) {

val filename = s"spark-$version-bin-hadoop2.7.tgz"

val url = s"$site/spark/spark-$version/$filename"

logInfo(s"Downloading Spark $version from $url")

try {

getFileFromUrl(url, path, filename)

val downloaded = new File(sparkTestingDir, filename).getCanonicalPath

val targetDir = new File(sparkTestingDir, s"spark-$version").getCanonicalPath

Seq("mkdir", targetDir).!

val exitCode = Seq("tar", "-xzf", downloaded, "-C", targetDir, "--strip-components=1").!

Seq("rm", downloaded).!

// For a corrupted file, `tar` returns non-zero values. However, we also need to check

// the extracted file because `tar` returns 0 for empty file.

val sparkSubmit = new File(sparkTestingDir, s"spark-$version/bin/spark-submit")

if (exitCode == 0 && sparkSubmit.exists()) {

return

} else {

Seq("rm", "-rf", targetDir).!

}

} catch {

case ex: Exception =>

logWarning(s"Failed to download Spark $version from $url: ${ex.getMessage}")

}

}

fail(s"Unable to download Spark $version")

}

and

spark/R/pkg/R/install.R

Lines 68 to 161 in f53d8c6

install.spark <- function(hadoopVersion = "2.7", mirrorUrl = NULL,

localDir = NULL, overwrite = FALSE) {

sparkHome <- Sys.getenv("SPARK_HOME")

if (isSparkRShell()) {

stopifnot(nchar(sparkHome) > 0)

message("Spark is already running in sparkR shell.")

return(invisible(sparkHome))

} else if (!is.na(file.info(sparkHome)$isdir)) {

message("Spark package found in SPARK_HOME: ", sparkHome)

return(invisible(sparkHome))

}

version <- paste0("spark-", packageVersion("SparkR"))

hadoopVersion <- tolower(hadoopVersion)

hadoopVersionName <- hadoopVersionName(hadoopVersion)

packageName <- paste(version, "bin", hadoopVersionName, sep = "-")

localDir <- ifelse(is.null(localDir), sparkCachePath(),

normalizePath(localDir, mustWork = FALSE))

if (is.na(file.info(localDir)$isdir)) {

dir.create(localDir, recursive = TRUE)

}

if (overwrite) {

message("Overwrite = TRUE: download and overwrite the tar file",

"and Spark package directory if they exist.")

}

releaseUrl <- Sys.getenv("SPARKR_RELEASE_DOWNLOAD_URL")

if (releaseUrl != "") {

packageName <- basenameSansExtFromUrl(releaseUrl)

}

packageLocalDir <- file.path(localDir, packageName)

# can use dir.exists(packageLocalDir) under R 3.2.0 or later

if (!is.na(file.info(packageLocalDir)$isdir) && !overwrite) {

if (releaseUrl != "") {

message(packageName, " found, setting SPARK_HOME to ", packageLocalDir)

} else {

message(version, " for Hadoop ",

if (hadoopVersion == "without") "Free build" else hadoopVersion,

" found, setting SPARK_HOME to ", packageLocalDir)

}

Sys.setenv(SPARK_HOME = packageLocalDir)

return(invisible(packageLocalDir))

} else {

message("Spark not found in the cache directory. Installation will start.")

}

packageLocalPath <- paste0(packageLocalDir, ".tgz")

tarExists <- file.exists(packageLocalPath)

if (tarExists && !overwrite) {

message("tar file found.")

} else {

if (releaseUrl != "") {

message("Downloading from alternate URL:\n- ", releaseUrl)

success <- downloadUrl(releaseUrl, packageLocalPath)

if (!success) {

unlink(packageLocalPath)

stop("Fetch failed from ", releaseUrl)

}

} else {

robustDownloadTar(mirrorUrl, version, hadoopVersion, packageName, packageLocalPath)

}

}

message("Installing to ", localDir)

# There are two ways untar can fail - untar could stop() on errors like incomplete block on file

# or, tar command can return failure code

success <- tryCatch(untar(tarfile = packageLocalPath, exdir = localDir) == 0,

error = function(e) {

message(e, "\n")

FALSE

},

warning = function(w) {

message(w, "\n")

FALSE

})

if (!tarExists || overwrite || !success) {

unlink(packageLocalPath)

if (success) {

# if tar file was not there before (or it was, but we are told to overwrite it),

# and untar is successful - set a flag that we have downloaded (and untar) Spark package.

assign(".sparkDownloaded", TRUE, envir = .sparkREnv)

}

}

if (!success) stop("Extract archive failed.")

message("DONE.")

Sys.setenv(SPARK_HOME = packageLocalDir)

message("SPARK_HOME set to ", packageLocalDir)

invisible(packageLocalDir)

}

HyukjinKwon · 2020-09-10T04:32:51Z

cc @srowen, @dongjoon-hyun, @holdenk, @BryanCutler, @viirya, @ueshin FYI

dongjoon-hyun · 2020-09-10T04:47:38Z

Thank you for pinging me, @HyukjinKwon .

cc @gatorsmile . Do you have any opinion on Hive 1.2 at Apache Spark 3.1.0?

HyukjinKwon · 2020-09-10T04:54:00Z

python/pyspark/install.py

+            print("Downloaded %d of %d bytes (%0.2f%%)" % (
+                bytes_so_far,
+                total_size,
+                round(float(bytes_so_far) / total_size * 100, 2)))


The purpose of showing the progress are two:

The print out will be seen when pip install ... -v.

The output from plan pip without -v shows like it's in progress (otherwise it looks like it hangs). For example:

Building wheel for pyspark (setup.py) ... - Building wheel for pyspark (setup.py) ... \ Building wheel for pyspark (setup.py) ... |

HyukjinKwon · 2020-09-10T05:02:00Z

dev/create-release/release-build.sh

@@ -275,6 +275,8 @@ if [[ "$1" == "package" ]]; then
  # In dry run mode, only build the first one. The keys in BINARY_PKGS_ARGS are used as the
  # list of packages to be built, so it's ok for things to be missing in BINARY_PKGS_EXTRA.

+  # NOTE: Don't forget to update the valid combinations of distributions at
+  #   'python/pyspark.install.py' if you're changing them.
  declare -A BINARY_PKGS_ARGS
  BINARY_PKGS_ARGS["hadoop3.2"]="-Phadoop-3.2 $HIVE_PROFILES"


If we happen to drop Hive 1.2 (or add other combinations of profiles in the distributions), we'll have to change this and here. I believe this could be done separately later.

python/setup.py

### What changes were proposed in this pull request? PyPI and CRAN did not change because of the concern about selecting Hadoop and Hive versions. For PyPI, now there is a PR open at #29703 For CRAN, we can already select Hadoop and Hive versions via `SparkR::install.spark`. ### Why are the changes needed? To keep the default profiles consistent in distributions ### Does this PR introduce _any_ user-facing change? Yes, the default distributions will use Hadoop 3.2. ### How was this patch tested? Jenkins tests. Closes #29704 from HyukjinKwon/SPARK-32058. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

srowen · 2020-09-10T13:02:25Z

I think exposing the option is fine.

nchammas

Some high-level questions:

How does this interact with the ability to specify dependencies in a requirements file? It seems odd to have to do something like HADOOP_VERSION=3.2 pip install -r requirements.txt, because, kinda like with --install-option, we've now modified pip's behavior across all the libraries it's going to install.

I also wonder if this plays well with tools like pip-tools that compile down requirements files into the full list of their transitive dependencies. I'm guessing users will need to manually preserve the environment variables, because they will not be reflected in the compiled requirements.
Have you considered publishing these alternate builds under different package names? e.g. pyspark-hadoop3.2. This avoids the need to mess with environment variables, and delivers a more vanilla install experience. But it will also push us to define upfront what combinations to publish builds for to PyPI.
Are you sure it's OK to point at archive.apache.org? Everyone installing a non-current version of PySpark with alternate versions of Hadoop / Hive specified will hit the archive. Unlike PyPI, the Apache archive is not backed by a generous CDN:

Do note that a daily limit of 5GB per IP is being enforced on archive.apache.org, to prevent abuse.

In Flintrock, I never touch the archive out of fear of being an "abusive user". This is another argument for publishing alternate packages to PyPI.

nchammas · 2020-09-11T00:56:09Z

python/pyspark/install.py

+            pass
+
+    default_sites = [
+        "https://archive.apache.org/dist", "https://dist.apache.org/repos/dist/release"]


All non-current versions of Spark will hit the archive, since the mirrors only maintain the latest version. I don't think the archive will be able to handle the volume of traffic that will eventually come its way from various people downloading (and re-downloading) Spark, e.g. as part of CI setup.

When they install, I think it will likely be the latest in most of cases. I guess that's the reason why we moved the old versions into these archive and keep the the latest versions in the mirrors.

People are already using this to download old versions or to setup the CI. This PR just makes it easier to do it.

HyukjinKwon · 2020-09-11T01:33:54Z

How does this interact with the ability to specify dependencies in a requirements file? It seems odd to have to do something like HADOOP_VERSION=3.2 pip install -r requirements.txt, because, kinda like with --install-option, we've now modified pip's behavior across all the libraries it's going to install.

I also wonder if this plays well with tools like pip-tools that compile down requirements files into the full list of their transitive dependencies. I'm guessing users will need to manually preserve the environment variables, because they will not be reflected in the compiled requirements.

I agree that it doesn't look very pip friendly. That's why I had to investigate a lot and write down what I checked in the PR description.

--instal-option is supported via requirement.txt so once pip provides a proper way to configure this, we will switch to this (at SPARK-32837). We can't use this option for now due to pypa/pip#1883 (see also pypa/pip#5771). There seems no other ways possible given my investigation.

We can just keep this as an experimental mode for the time being in this way, and switch it to the proper pip installation option once they support in the future.

Have you considered publishing these alternate builds under different package names? e.g. pyspark-hadoop3.2. This avoids the need to mess with environment variables, and delivers a more vanilla install experience. But it will also push us to define upfront what combinations to publish builds for to PyPI.

I have thought about this option too but ..

I think we'll end up with having multiple packages per the profiles we support.
I still think using pip's native configuration is the ideal way. By using environment variables, we can easily switch it to use pip's option in the future.
Minor but .. It will be difficult to track the usage (https://pypistats.org/packages/pyspark)

Are you sure it's OK to point at archive.apache.org? Everyone installing a non-current version of PySpark with alternate versions of Hadoop / Hive specified will hit the archive. Unlike PyPI, the Apache archive is not backed by a generous CDN:

Do note that a daily limit of 5GB per IP is being enforced on archive.apache.org, to prevent abuse.

In Flintrock, I never touch the archive out of fear of being an "abusive user". This is another argument for publishing alternate packages to PyPI.

Yeah, I understand this can be a valid concern. But this is already available to use and people use it. Also it's being used in our own CI:

spark/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala

Line 82 in b84ed41

    
           mirrors.distinct :+ "https://archive.apache.org/dist" :+ PROCESS_TABLES.releaseMirror

The PR makes it easier to use them to download old versions. We can make it configurable as well via exposing an environment variable.

SparkQA · 2020-09-14T13:13:34Z

Test build #128639 has finished for PR 29703 at commit 83815e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-09-15T00:34:55Z

I documented it. I believe this is ready for a review.

HyukjinKwon · 2020-09-17T02:56:08Z

I proofread, tested again and fixed some docs.

python/docs/source/getting_started/installation.rst

viirya · 2020-09-17T03:15:59Z

python/pyspark/install.py

+    if hadoop_version == "without":
+        hadoop_version = "without-hadoop"


Is "without-hadoop" also supported as special keyword? Seems not see it is matched here?

It is verified below if hadoop_version not in SUPPORTED_HADOOP_VERSIONS: later. There's a test case here: https://github.com/apache/spark/pull/29703/files/033a33ee515b95342e8c5a74e63054d915661579#diff-e23af4eb5cc3bf6af4bc26cb801b7e84R69 and https://github.com/apache/spark/pull/29703/files/033a33ee515b95342e8c5a74e63054d915661579#diff-e23af4eb5cc3bf6af4bc26cb801b7e84R88

Users can specify the Hadoop and Hive versions such as hadoop3.2 and hive2.3 as well but I didn't document this. These keywords are actually ported from SparkR as are SparkR::install.spark.

SparkQA · 2020-09-17T06:11:21Z

Test build #128789 has finished for PR 29703 at commit 033a33e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-09-17T07:05:02Z

Test build #128793 has finished for PR 29703 at commit 09997b7.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-09-20T05:10:16Z

Test build #128903 has finished for PR 29703 at commit 058e61a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin

LGTM otherwise.

python/pyspark/find_spark_home.py

HyukjinKwon · 2020-09-22T04:20:59Z

I tested again on both Windows and Mac.

SparkQA · 2020-09-22T04:27:06Z

Test build #128956 has finished for PR 29703 at commit 33594e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-09-22T07:05:01Z

Test build #128961 has finished for PR 29703 at commit 20491e0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-09-22T07:05:02Z

Test build #128960 has finished for PR 29703 at commit 1e3507f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-09-22T09:08:39Z

retest this please

SparkQA · 2020-09-22T11:43:49Z

Test build #128973 has finished for PR 29703 at commit 20491e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-09-23T00:30:03Z

Merged to master.

HyukjinKwon · 2020-09-23T00:30:51Z

Thanks @viirya @ueshin @nchammas, @srowen and @dongjoon-hyun for reviewing this.

…OOP_VERSION in pip installation option ### What changes were proposed in this pull request? This PR is a followup of #29703. It renames `HADOOP_VERSION` environment variable to `PYSPARK_HADOOP_VERSION` in case `HADOOP_VERSION` is already being used somewhere. Arguably `HADOOP_VERSION` is a pretty common name. I see here and there: - https://www.ibm.com/support/knowledgecenter/SSZUMP_7.2.1/install_grid_sym/understanding_advanced_edition.html - https://cwiki.apache.org/confluence/display/ARROW/HDFS+Filesystem+Support - http://crs4.github.io/pydoop/_pydoop1/installation.html ### Why are the changes needed? To avoid the environment variables is unexpectedly conflicted. ### Does this PR introduce _any_ user-facing change? It renames the environment variable but it's not released yet. ### How was this patch tested? Existing unittests will test. Closes #31028 from HyukjinKwon/SPARK-32017-followup. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

…OOP_VERSION in pip installation option ### What changes were proposed in this pull request? This PR is a followup of #29703. It renames `HADOOP_VERSION` environment variable to `PYSPARK_HADOOP_VERSION` in case `HADOOP_VERSION` is already being used somewhere. Arguably `HADOOP_VERSION` is a pretty common name. I see here and there: - https://www.ibm.com/support/knowledgecenter/SSZUMP_7.2.1/install_grid_sym/understanding_advanced_edition.html - https://cwiki.apache.org/confluence/display/ARROW/HDFS+Filesystem+Support - http://crs4.github.io/pydoop/_pydoop1/installation.html ### Why are the changes needed? To avoid the environment variables is unexpectedly conflicted. ### Does this PR introduce _any_ user-facing change? It renames the environment variable but it's not released yet. ### How was this patch tested? Existing unittests will test. Closes #31028 from HyukjinKwon/SPARK-32017-followup. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit 329850c) Signed-off-by: HyukjinKwon <gurwls223@apache.org>

probot-autolabeler bot added BUILD CORE INFRA PYTHON labels Sep 10, 2020

HyukjinKwon commented Sep 10, 2020

View reviewed changes

HyukjinKwon mentioned this pull request Sep 10, 2020

[SPARK-32841][BUILD] Use Apache Hadoop 3.2.0 for PyPI and CRAN #29704

Closed

HyukjinKwon commented Sep 10, 2020

View reviewed changes

viirya reviewed Sep 10, 2020

View reviewed changes

python/setup.py Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

nchammas reviewed Sep 11, 2020

View reviewed changes

HyukjinKwon force-pushed the SPARK-32017 branch from 75dbcaf to 83815e0 Compare September 14, 2020 10:10

probot-autolabeler bot added the DOCS label Sep 14, 2020

viirya reviewed Sep 17, 2020

View reviewed changes

HyukjinKwon added 4 commits September 20, 2020 11:02

Make Pyspark Hadoop 3.2+ Variant available in PyPI

b28e284

Add test_install_spark into modules.py

498f188

Add docs and address a comment

0ac1e02

Address nits and fix documentations to be more clear

058e61a

HyukjinKwon force-pushed the SPARK-32017 branch from 09997b7 to 058e61a Compare September 20, 2020 02:04

ueshin approved these changes Sep 21, 2020

View reviewed changes

ueshin reviewed Sep 22, 2020

View reviewed changes

python/pyspark/find_spark_home.py Show resolved Hide resolved

Address comment

33594e1

Cleanup and add an env

1e3507f

Add a new env to select a mirror

20491e0

HyukjinKwon closed this in 942f577 Sep 23, 2020

HyukjinKwon deleted the SPARK-32017 branch December 7, 2020 02:05

HyukjinKwon mentioned this pull request Jan 5, 2021

[SPARK-32017][PYTHON][FOLLOW-UP] Rename HADOOP_VERSION to PYSPARK_HADOOP_VERSION in pip installation option #31028

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-32017][PYTHON][BUILD] Make Pyspark Hadoop 3.2+ Variant available in PyPI #29703

[SPARK-32017][PYTHON][BUILD] Make Pyspark Hadoop 3.2+ Variant available in PyPI #29703

HyukjinKwon commented Sep 10, 2020 •

edited

Loading

HyukjinKwon Sep 10, 2020

HyukjinKwon commented Sep 10, 2020

dongjoon-hyun commented Sep 10, 2020 •

edited

Loading

HyukjinKwon Sep 10, 2020

HyukjinKwon Sep 10, 2020

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

srowen commented Sep 10, 2020

nchammas left a comment •

edited

Loading

nchammas Sep 11, 2020

HyukjinKwon Sep 11, 2020

HyukjinKwon commented Sep 11, 2020 •

edited

Loading

SparkQA commented Sep 14, 2020

HyukjinKwon commented Sep 15, 2020

HyukjinKwon commented Sep 17, 2020

viirya Sep 17, 2020

HyukjinKwon Sep 17, 2020

SparkQA commented Sep 17, 2020

SparkQA commented Sep 17, 2020

SparkQA commented Sep 20, 2020

ueshin left a comment •

edited

Loading

HyukjinKwon commented Sep 22, 2020

SparkQA commented Sep 22, 2020

SparkQA commented Sep 22, 2020

SparkQA commented Sep 22, 2020

HyukjinKwon commented Sep 22, 2020

SparkQA commented Sep 22, 2020

HyukjinKwon commented Sep 23, 2020

HyukjinKwon commented Sep 23, 2020

		return spark_version, hadoop_version, hive_version


		def install_spark(dest, spark_version, hadoop_version, hive_version):

	private def tryDownloadSpark(version: String, path: String): Unit = {
	// Try a few mirrors first; fall back to Apache archive
	val mirrors =
	(0 until 2).flatMap { _ =>
	try {
	Some(getStringFromUrl("https://www.apache.org/dyn/closer.lua?preferred=true"))
	} catch {
	// If we can't get a mirror URL, skip it. No retry.
	case _: Exception => None
	}
	}
	val sites =
	mirrors.distinct :+ "https://archive.apache.org/dist" :+ PROCESS_TABLES.releaseMirror
	logInfo(s"Trying to download Spark $version from $sites")
	for (site <- sites) {
	val filename = s"spark-$version-bin-hadoop2.7.tgz"
	val url = s"$site/spark/spark-$version/$filename"
	logInfo(s"Downloading Spark $version from $url")
	try {
	getFileFromUrl(url, path, filename)
	val downloaded = new File(sparkTestingDir, filename).getCanonicalPath
	val targetDir = new File(sparkTestingDir, s"spark-$version").getCanonicalPath

	Seq("mkdir", targetDir).!
	val exitCode = Seq("tar", "-xzf", downloaded, "-C", targetDir, "--strip-components=1").!
	Seq("rm", downloaded).!

	// For a corrupted file, `tar` returns non-zero values. However, we also need to check
	// the extracted file because `tar` returns 0 for empty file.
	val sparkSubmit = new File(sparkTestingDir, s"spark-$version/bin/spark-submit")
	if (exitCode == 0 && sparkSubmit.exists()) {
	return
	} else {
	Seq("rm", "-rf", targetDir).!
	}
	} catch {
	case ex: Exception =>
	logWarning(s"Failed to download Spark $version from $url: ${ex.getMessage}")
	}
	}
	fail(s"Unable to download Spark $version")
	}

	install.spark <- function(hadoopVersion = "2.7", mirrorUrl = NULL,
	localDir = NULL, overwrite = FALSE) {
	sparkHome <- Sys.getenv("SPARK_HOME")
	if (isSparkRShell()) {
	stopifnot(nchar(sparkHome) > 0)
	message("Spark is already running in sparkR shell.")
	return(invisible(sparkHome))
	} else if (!is.na(file.info(sparkHome)$isdir)) {
	message("Spark package found in SPARK_HOME: ", sparkHome)
	return(invisible(sparkHome))
	}

	version <- paste0("spark-", packageVersion("SparkR"))
	hadoopVersion <- tolower(hadoopVersion)
	hadoopVersionName <- hadoopVersionName(hadoopVersion)
	packageName <- paste(version, "bin", hadoopVersionName, sep = "-")
	localDir <- ifelse(is.null(localDir), sparkCachePath(),
	normalizePath(localDir, mustWork = FALSE))

	if (is.na(file.info(localDir)$isdir)) {
	dir.create(localDir, recursive = TRUE)
	}

	if (overwrite) {
	message("Overwrite = TRUE: download and overwrite the tar file",
	"and Spark package directory if they exist.")
	}

	releaseUrl <- Sys.getenv("SPARKR_RELEASE_DOWNLOAD_URL")
	if (releaseUrl != "") {
	packageName <- basenameSansExtFromUrl(releaseUrl)
	}

	packageLocalDir <- file.path(localDir, packageName)

	# can use dir.exists(packageLocalDir) under R 3.2.0 or later
	if (!is.na(file.info(packageLocalDir)$isdir) && !overwrite) {
	if (releaseUrl != "") {
	message(packageName, " found, setting SPARK_HOME to ", packageLocalDir)
	} else {
	message(version, " for Hadoop ",
	if (hadoopVersion == "without") "Free build" else hadoopVersion,
	" found, setting SPARK_HOME to ", packageLocalDir)
	}
	Sys.setenv(SPARK_HOME = packageLocalDir)
	return(invisible(packageLocalDir))
	} else {
	message("Spark not found in the cache directory. Installation will start.")
	}

	packageLocalPath <- paste0(packageLocalDir, ".tgz")
	tarExists <- file.exists(packageLocalPath)

	if (tarExists && !overwrite) {
	message("tar file found.")
	} else {
	if (releaseUrl != "") {
	message("Downloading from alternate URL:\n- ", releaseUrl)
	success <- downloadUrl(releaseUrl, packageLocalPath)
	if (!success) {
	unlink(packageLocalPath)
	stop("Fetch failed from ", releaseUrl)
	}
	} else {
	robustDownloadTar(mirrorUrl, version, hadoopVersion, packageName, packageLocalPath)
	}
	}

	message("Installing to ", localDir)
	# There are two ways untar can fail - untar could stop() on errors like incomplete block on file
	# or, tar command can return failure code
	success <- tryCatch(untar(tarfile = packageLocalPath, exdir = localDir) == 0,
	error = function(e) {
	message(e, "\n")
	FALSE
	},
	warning = function(w) {
	message(w, "\n")
	FALSE
	})
	if (!tarExists \|\| overwrite \|\| !success) {
	unlink(packageLocalPath)
	if (success) {
	# if tar file was not there before (or it was, but we are told to overwrite it),
	# and untar is successful - set a flag that we have downloaded (and untar) Spark package.
	assign(".sparkDownloaded", TRUE, envir = .sparkREnv)
	}
	}
	if (!success) stop("Extract archive failed.")
	message("DONE.")
	Sys.setenv(SPARK_HOME = packageLocalDir)
	message("SPARK_HOME set to ", packageLocalDir)
	invisible(packageLocalDir)
	}

		if hadoop_version == "without":
		hadoop_version = "without-hadoop"

[SPARK-32017][PYTHON][BUILD] Make Pyspark Hadoop 3.2+ Variant available in PyPI #29703

[SPARK-32017][PYTHON][BUILD] Make Pyspark Hadoop 3.2+ Variant available in PyPI #29703

Conversation

HyukjinKwon commented Sep 10, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

HyukjinKwon Sep 10, 2020

Choose a reason for hiding this comment

HyukjinKwon commented Sep 10, 2020

dongjoon-hyun commented Sep 10, 2020 • edited Loading

HyukjinKwon Sep 10, 2020

Choose a reason for hiding this comment

HyukjinKwon Sep 10, 2020

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

srowen commented Sep 10, 2020

nchammas left a comment • edited Loading

Choose a reason for hiding this comment

nchammas Sep 11, 2020

Choose a reason for hiding this comment

HyukjinKwon Sep 11, 2020

Choose a reason for hiding this comment

HyukjinKwon commented Sep 11, 2020 • edited Loading

SparkQA commented Sep 14, 2020

HyukjinKwon commented Sep 15, 2020

HyukjinKwon commented Sep 17, 2020

viirya Sep 17, 2020

Choose a reason for hiding this comment

HyukjinKwon Sep 17, 2020

Choose a reason for hiding this comment

SparkQA commented Sep 17, 2020

SparkQA commented Sep 17, 2020

SparkQA commented Sep 20, 2020

ueshin left a comment • edited Loading

Choose a reason for hiding this comment

HyukjinKwon commented Sep 22, 2020

SparkQA commented Sep 22, 2020

SparkQA commented Sep 22, 2020

SparkQA commented Sep 22, 2020

HyukjinKwon commented Sep 22, 2020

SparkQA commented Sep 22, 2020

HyukjinKwon commented Sep 23, 2020

HyukjinKwon commented Sep 23, 2020

HyukjinKwon commented Sep 10, 2020 •

edited

Loading

dongjoon-hyun commented Sep 10, 2020 •

edited

Loading

nchammas left a comment •

edited

Loading

HyukjinKwon commented Sep 11, 2020 •

edited

Loading

ueshin left a comment •

edited

Loading