[SPARK-33214][TEST][HIVE] Stop HiveExternalCatalogVersionsSuite from using a hard-coded location to store localized Spark binaries.

[xkrogen]

What changes were proposed in this pull request?

This PR changes HiveExternalCatalogVersionsSuite to, by default, use a standard temporary directory to store the Spark binaries that it localizes. It additionally adds a new System property, spark.test.cache-dir, which can be used to define a static location into which the Spark binary will be localized to allow for sharing between test executions. If the System property is used, the downloaded binaries won't be deleted after the test runs.

Why are the changes needed?

In SPARK-22356 (PR #19579), the sparkTestingDir used by HiveExternalCatalogVersionsSuite became hard-coded to enable re-use of the downloaded Spark tarball between test executions:

  // For local test, you can set `sparkTestingDir` to a static value like `/tmp/test-spark`, to
  // avoid downloading Spark of different versions in each run.
  private val sparkTestingDir = new File("/tmp/test-spark")

However this doesn't work, since it gets deleted every time:

  override def afterAll(): Unit = {
    try {
      Utils.deleteRecursively(wareHousePath)
      Utils.deleteRecursively(tmpDataDir)
      Utils.deleteRecursively(sparkTestingDir)
    } finally {
      super.afterAll()
    }
  }

It's bad that we're hard-coding to a /tmp directory, as in some cases this is not the proper place to store temporary files. We're not currently making any good use of it.

Does this PR introduce any user-facing change?

Developer-facing changes only, as this is in a test.

How was this patch tested?

The test continues to execute as expected.

[xkrogen]

Ping @cloud-fan @gatorsmile who were involved in the original PR.

[cloud-fan]

ok to test

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35188/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35188/

[SparkQA]

Test build #130588 has finished for PR 30122 at commit c56cf1f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master!

[xkrogen]

Thanks @cloud-fan !