[SPARK-30617][SQL] Stop check values of spark.sql.catalogImplementation to improve expansibility #27349

WeiWenda · 2020-01-24T01:28:09Z

What changes were proposed in this pull request?

When user config spark.sql.catalogImplementation with value not in in-memory/hive, check if below properties is configured. If configured then instantiate SessionState with provided Class, or else throw Exception as usual.

spark.sql.catalogImplementation.[value of spark.sql.catalogImplementation].builder
spark.sql.catalogImplementation.[value of spark.sql.catalogImplementation].externalCatalog

For example:

spark.sql.catalogImplementation = qihoo
spark.sql.catalogImplementation.qihoo.builder = org.apache.spark.sql.qihoo.QihooSessionStateBuilder
spark.sql.catalogImplementation.qihoo.externalCatalog = org.apache.spark.sql.qihoo.QihooExternalCatalog

Why are the changes needed?

We have implemented a complex ExternalCatalog which is used for retrieving multi isomerism database's metadata(sush as elasticsearch、postgresql), so that we can make a mixture query between hive and our online data. But as spark require that value of spark.sql.catalogImplementation must be one of in-memory/hive, we have to modify SparkSession and rebuild spark to make our project work.
Finally, we hope spark removing above restriction, so that it's will be much easier to let us keep pace with new spark version. Thanks!

Does this PR introduce any user-facing change?

no

How was this patch tested?

no

…proach to let user define themselves metadata catalog

AmplabJenkins · 2020-01-24T01:32:59Z

Can one of the admins verify this patch?

dongjoon-hyun · 2020-01-24T03:53:28Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala

@@ -36,9 +36,13 @@ object StaticSQLConf {
    .createWithDefault(Utils.resolveURI("spark-warehouse").toString)

  val CATALOG_IMPLEMENTATION = buildStaticConf("spark.sql.catalogImplementation")
+    .doc("Spark built-in implementation is in-memory and hive," +


nit. ," -> , ".

dongjoon-hyun · 2020-01-24T03:54:30Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala

+      "when set to yourImpl(can be any words), " +
+      "you need also provide following configuration to make everything going" +
+      "spark.sql.catalogImplementation.yourImpl.builder" +
+      "spark.sql.catalogImplementation.yourImpl.externalCatalog")
    .internal()


Technically, this is .internal() configuration. This PR seems to propose make this external conf.

I find that internal didn't effect anything. So I have removed .internal() in latest commit.

dongjoon-hyun · 2020-01-24T03:56:18Z

sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala

+      case other => conf.getOption(s"spark.sql.catalogImplementation.$other.builder")
+          .getOrElse {
+        throw new IllegalArgumentException(
+          "You need to configure spark.sql.catalogImplementation.xx.builder when xx configured by" +


The following will be better.

- "You need to configure spark.sql.catalogImplementation.xx.builder when xx configured by" + + s"You need to configure spark.sql.catalogImplementation.$other.builder when $other configured by" +

dongjoon-hyun · 2020-01-24T03:56:35Z

sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala

+      case other => conf.getOption(s"spark.sql.catalogImplementation.$other.externalCatalog")
+          .getOrElse {
+        throw new IllegalArgumentException(
+          "You need to configure spark.sql.catalogImplementation.xx.externalCatalog when xx " +


dongjoon-hyun · 2020-01-24T04:00:26Z

Technically, there exist previous JIRAs and PRs. The first one is similar to this.

At 2.0.0, [SPARK-17767][SQL] Support user-provided ExternalCatalog
At 2.1.0, SPARK-19632 Allow configuring non-hive and non-local SessionState and ExternalCatalog
At 3.0.0, SPARK-23443 Spark with Glue as external catalog

dongjoon-hyun

Thank you for making a PR, @WeiWenda .

The recent community decision was using new CatalogPlugin for this kind of needs.

Since this is requested again, I'll ping the senior PMCs once more.

Hi, @rxin , @gatorsmile , @cloud-fan .
Can we extend spark.sql.catalogImplementation in 3.0.0?

WeiWenda · 2020-01-24T10:52:07Z

@dongjoon-hyun Thanks for your help. SPARK-17767 and SPARK-19632's solution is so old that cannot be used in spark 3.0.0 anymore.

The recent community decision was using new CatalogPlugin for this kind of needs.

If there still has a long way to go with new CatalogPlugin, maybe cancelling checkValues of spark.sql. catalogImplementation is lowest-cost way to consider user-defined catalog.

dongjoon-hyun · 2020-01-24T20:18:42Z

I don't think so. I intentionally didn't reopen it. (I'm the author of the oldest PR on SPARK-17767 and have been keeping track this requirement exactly since Apache Spark 2.0.0. 😄 )

SPARK-19632's solution is so old that cannot be used in spark 3.0.0 anymore.

CatalogPlugin is already available at 3.0.0-preview and 3.0.0-preview2 via https://issues.apache.org/jira/browse/SPARK-24252 . At 3.0.0, it will be a much stable feature.

If there still has a long way to go with new CatalogPlugin

WeiWenda · 2020-01-25T02:45:51Z

Yeah, i noticed your SPARK-17767, you are foresight and responsible. I just look at #23915, it can solve our problem more gently. Is spark 3.0.0 will release in recent month?

dongjoon-hyun · 2020-01-25T04:10:59Z

According to the plan, we will proceed feature freeze first by creating branch-3.0 in this month.

https://spark.apache.org/versioning-policy.html

Then, QA period will be started. For major version, it takes some time.

In any way, your patch cannot be released faster than 3.0.0 because new feature is not backported to branch-2.4. (2.4.5 RC1 vote was held last week and we are receiving some more correctness patches. 2.4.5 RC2 vote will start soon. 2.4.5 will be released before 3.0.0.)

dongjoon-hyun · 2020-01-25T04:13:16Z

BTW, if you agree with CatalogPlugin in 3.0.0, could you close the PR and JIRA please?

WeiWenda · 2020-01-25T04:20:27Z

BTW, if you agree with CatalogPlugin in 3.0.0, could you close the PR and JIRA please?

All right, we will waiting for spark 3.0.0. Thank you anyway.

dongjoon-hyun · 2020-01-25T04:33:07Z

Thank you, @WeiWenda .

stop check values of spark.sql.catalogImplementation to provide an ap…

ee868b9

…proach to let user define themselves metadata catalog

WeiWenda mentioned this pull request Jan 24, 2020

[SPARK-30617][SQL] Stop check values of spark.sql.catalogImplementation to improve expansibility #27338

Closed

dongjoon-hyun reviewed Jan 24, 2020

View reviewed changes

dongjoon-hyun added the SQL label Jan 24, 2020

code reformat

e01b352

WeiWenda closed this Jan 25, 2020

[SPARK-30617][SQL] Stop check values of spark.sql.catalogImplementation to improve expansibility #27349

[SPARK-30617][SQL] Stop check values of spark.sql.catalogImplementation to improve expansibility #27349

Uh oh!

Conversation

WeiWenda commented Jan 24, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AmplabJenkins commented Jan 24, 2020

Uh oh!

dongjoon-hyun Jan 24, 2020

Choose a reason for hiding this comment

Uh oh!

WeiWenda Jan 24, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jan 24, 2020

Choose a reason for hiding this comment

Uh oh!

WeiWenda Jan 24, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jan 24, 2020

Choose a reason for hiding this comment

Uh oh!

WeiWenda Jan 24, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jan 24, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jan 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WeiWenda commented Jan 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jan 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WeiWenda commented Jan 25, 2020

Uh oh!

dongjoon-hyun commented Jan 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jan 25, 2020

Uh oh!

WeiWenda commented Jan 25, 2020

Uh oh!

dongjoon-hyun commented Jan 25, 2020

Uh oh!

Uh oh!

dongjoon-hyun commented Jan 24, 2020 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

WeiWenda commented Jan 24, 2020 •

edited

Loading

dongjoon-hyun commented Jan 24, 2020 •

edited

Loading

dongjoon-hyun commented Jan 25, 2020 •

edited

Loading