Allow creating SparkContext in executors when doing so in order to score Spark models via mlflow.pyfunc.spark_udf

[smurching]

Signed-off-by: Sid Murching sid.murching@databricks.com

What changes are proposed in this pull request?

Flips flag (introduced in apache/spark#28986) to allow creating SparkContext on the executors.

We depend on this behavior (which is being disabled, allowable via a flag, in Spark 3.1) when scoring Spark models via MLflow's mlflow.pyfunc.spark_udf API. In particular, when scoring a Spark model via spark_udf, the underlying pandas_udf we define for scoring constructs a SparkContext in order to create a Spark DataFrame out of the passed-in pandas DataFrame, and then scores the Spark ML model on the dataframe. The pandas_udf runs on the executors, hence we need to be able to create a SparkContext on the executors.

How is this patch tested?

Manual testing against Spark 3.1
(Details)

Release Notes

Is this a user-facing change?

• No. You can skip the rest of this section.
• Yes. Give a description of this change to be included in the release notes for MLflow users.

(Details in 1-2 sentences. You can just refer to another PR with a description if this PR is part of a larger change.)

What component(s), interfaces, languages, and integrations does this PR affect?

Components

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/projects: MLproject format, project running backends
• area/scoring: Local serving, model deployment tools, spark UDFs
• area/server-infra: MLflow server, JavaScript dev server
• area/tracking: Tracking Service, tracking client APIs, autologging

Interface

• area/uiux: Front-end, user experience, JavaScript, plotting
• area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

Language

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

Integrations

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

• rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
• rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
• rn/feature - A new user-facing feature worth mentioning in the release notes
• rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
• rn/documentation - A user-facing documentation change worth mentioning in the release notes