[SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive #25542

sandeep-katta · 2019-08-21T14:55:29Z

What changes were proposed in this pull request?

Spark loads the jars to custom class loader which is returned by getSubmitClassLoader .
Spark code

In 1.2.1.spark2 version of Hive

HiveConf.getClassLoader returns same the class loader which is set by the spark

In Hive 2.3.5
HiveConf.getClassLoader returns the UDFClassLoader which is created by Hive. Because of this spark cannot find the jars as class loader got changed
Hive code

Why are the changes needed?

Before creating CliSessionState object save the current class loader object in some reference.
After SessionState.start() reset back class Loader to the one which saved earlier.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added Test case and also Manually tested

Before Fix

After Fix

wangyum

Is it a standard usage? We support --jars one.jar:

build/sbt clean package -Phive -Phadoop-2.7 -Phive-thriftserver
export SPARK_PREPEND_CLASSES=true
bin/spark-sql --jars /root/.ivy2/cache/org.spark-project.hive/hive-contrib/jars/hive-contrib-1.2.1.spark2.jar -e "CREATE TEMPORARY FUNCTION example_max AS 'org.apache.hadoop.hive.contrib.udaf.example.UDAFExampleMax'"

sandeep-katta · 2019-08-21T15:58:32Z

Is it a standard usage? We support --jars one.jar:

build/sbt clean package -Phive -Phadoop-2.7 -Phive-thriftserver
export SPARK_PREPEND_CLASSES=true
bin/spark-sql --jars /root/.ivy2/cache/org.spark-project.hive/hive-contrib/jars/hive-contrib-1.2.1.spark2.jar -e "CREATE TEMPORARY FUNCTION example_max AS 'org.apache.hadoop.hive.contrib.udaf.example.UDAFExampleMax'"

Yes we can use one or multiple jars. Sorry I didn't get you, why you gave the above command ?

wangyum · 2019-08-21T16:08:26Z

Our example always uses --jars one.jar,two.jar. It seems --jars=one.jar,two.jar is not a standard usage.

http://spark.apache.org/docs/latest/spark-standalone.html#launching-spark-applications
http://spark.apache.org/docs/latest/running-on-yarn.html#adding-other-jars
http://spark.apache.org/docs/latest/rdd-programming-guide.html#using-the-shell
http://spark.apache.org/docs/latest/sql-data-sources-jdbc.html

sandeep-katta · 2019-08-21T17:36:00Z

I got it. even if you use like below also there is a problem
./bin/spark-sql --master local --jars /opt/BigdataTools/summation.jar

dongjoon-hyun · 2019-08-21T18:16:44Z

ok to test

dongjoon-hyun · 2019-08-21T18:25:23Z

Retest this please.

SparkQA · 2019-08-21T18:57:37Z

Test build #109510 has finished for PR 25542 at commit 7e89f55.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-21T18:59:55Z

Test build #109509 has finished for PR 25542 at commit 7e89f55.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-08-22T02:54:15Z

It seems --jars sql/hive/src/test/resources/SPARK-21101-1.0.jar works without this patch:

build/sbt clean package -Phive -Phadoop-2.7 -Phive-thriftserver
export SPARK_PREPEND_CLASSES=true
bin/spark-sql --jars sql/hive/src/test/resources/SPARK-21101-1.0.jar

sandeep-katta · 2019-08-22T03:04:06Z

please use hadoop-3.2 as it comes with Hive 2.3.5

wangyum · 2019-08-22T03:16:26Z

Yes. I can reproduce this issue. Please add [test-hadoop3.2] to the PR title.

wangyum · 2019-08-22T03:19:46Z

retest this please

dongjoon-hyun · 2019-08-22T03:25:31Z

Hi, guys. I already triggered both profiles.
Test build #109509 has finished for PR 25542 at commit 7e89f55. was hadoop-3.2.

SparkQA · 2019-08-22T03:47:05Z

Test build #109547 has finished for PR 25542 at commit 7e89f55.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-08-22T06:56:01Z

This issue caused by HIVE-11878.

wangyum · 2019-08-22T10:16:38Z

...e-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala

      // components.
      // See also: code in ExecDriver.java
-      var loader = conf.getClassLoader
+      var loader = orginalClassLoader


I think this is a correct fix. Another approach is add the Spark jars to Utilities.addToClassPath to make UDFClassLoader work:

val sparkJars = sparkConf.get(org.apache.spark.internal.config.JARS) if (sparkJars.nonEmpty || StringUtils.isNotBlank(auxJars)) { loader = Utilities.addToClassPath(loader, sparkJars.toArray ++ StringUtils.split(auxJars, ",")) }

cc @juliuszsompolski

wangyum · 2019-08-22T10:17:30Z

...e-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala

-      var loader = conf.getClassLoader
+      var loader = orginalClassLoader
      val auxJars = HiveConf.getVar(conf, HiveConf.ConfVars.HIVEAUXJARS)
      if (StringUtils.isNotBlank(auxJars)) {


Please add another test case to cover Utilities.addToClassPath(loader, StringUtils.split(auxJars, ",")):

spark-sql --jars one.jar --conf spark.hadoop.hive.aux.jars.path=two.jar

I see already test case is present for this test("Support hive.aux.jars.path") . Do you want me to add one more combination with --jars and --conf both ?

Yes. Thank you.

I see already test case is present for this test("Support hive.aux.jars.path") . Do you want me to add one more combination with --jars and --conf both ?

is the test case written for -- conf

juliuszsompolski · 2019-08-22T14:38:41Z

...e-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala

        cliConf.set(k, v)
    }

+    val orginalClassLoader = Thread.currentThread().getContextClassLoader


nit: orginal -> original

SparkQA · 2019-08-22T15:26:39Z

Test build #109579 has finished for PR 25542 at commit bc87633.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-22T15:42:20Z

Test build #109581 has finished for PR 25542 at commit 82795ad.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-08-24T16:12:30Z

Retest this please.

SparkQA · 2019-08-24T16:37:05Z

Test build #109669 has finished for PR 25542 at commit 82795ad.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-10T15:39:26Z

Test build #110422 has finished for PR 25542 at commit 533eeeb.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-10T17:00:12Z

Test build #110427 has finished for PR 25542 at commit 8eb6310.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-09-10T17:46:13Z

Test build #110430 has finished for PR 25542 at commit cf13bfa.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-10T18:03:05Z

Test build #110431 has finished for PR 25542 at commit 47f8632.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-11T03:38:15Z

Test build #110455 has finished for PR 25542 at commit 4a79580.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-11T07:09:46Z

retest this please

SparkQA · 2019-09-11T07:38:16Z

Test build #110462 has finished for PR 25542 at commit 4a79580.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-12T10:55:10Z

Merged to master.

… avoided as it returns the UDFClassLoader which is created by Hive ### What changes were proposed in this pull request? Spark loads the jars to custom class loader which is returned by `getSubmitClassLoader` . [Spark code](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L337) **In 1.2.1.spark2 version of Hive** `HiveConf.getClassLoader` returns same the class loader which is set by the spark **In Hive 2.3.5** `HiveConf.getClassLoader` returns the UDFClassLoader which is created by Hive. Because of this spark cannot find the jars as class loader got changed [Hive code](https://github.com/apache/hive/blob/rel/release-2.3.5/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L395) ### Why are the changes needed? Before creating `CliSessionState` object save the current class loader object in some reference. After SessionState.start() reset back class Loader to the one which saved earlier. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Added Test case and also Manually tested **Before Fix** ![b4Fix](https://user-images.githubusercontent.com/35216143/63442838-6789f400-c451-11e9-9529-ccf4ea9621b9.png) **After Fix** ![afterFix](https://user-images.githubusercontent.com/35216143/63442860-707ac580-c451-11e9-8012-2b70934d55f3.png) Closes apache#25542 from sandeep-katta/jarIssue. Lead-authored-by: sandeep katta <sandeep.katta2007@gmail.com> Co-authored-by: angerszhu <angers.zhu@gmail.com> Signed-off-by: Yuming Wang <wgyumg@gmail.com>

…ing SessionState for built-in Hive 2.3 ### What changes were proposed in this pull request? Hive 2.3 will set a new UDFClassLoader to hiveConf.classLoader when initializing SessionState since HIVE-11878, and 1. ADDJarCommand will add jars to clientLoader.classLoader. 2. --jar passed jar will be added to clientLoader.classLoader 3. jar passed by hive conf `hive.aux.jars` [SPARK-28954](#25653) [SPARK-28840](#25542) will be added to clientLoader.classLoader too For these reason we cannot load the jars added by ADDJarCommand because of class loader got changed. We reset it to clientLoader.ClassLoader here. ### Why are the changes needed? support for jdk11 ### Does this PR introduce any user-facing change? NO ### How was this patch tested? UT ``` export JAVA_HOME=/usr/lib/jdk-11.0.3 export PATH=$JAVA_HOME/bin:$PATH build/sbt -Phive-thriftserver -Phadoop-3.2 hive/test-only *HiveSparkSubmitSuite -- -z "SPARK-8368: includes jars passed in through --jars" hive-thriftserver/test-only *HiveThriftBinaryServerSuite -- -z "test add jar" ``` Closes #25775 from AngersZhuuuu/SPARK-29015-STS-JDK11. Authored-by: angerszhu <angers.zhu@gmail.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>

wangyum reviewed Aug 21, 2019

View reviewed changes

dongjoon-hyun added the SQL label Aug 21, 2019

wangyum reviewed Aug 22, 2019

View reviewed changes

juliuszsompolski reviewed Aug 22, 2019

View reviewed changes

sandeep-katta force-pushed the jarIssue branch 2 times, most recently from 330bda1 to 82795ad Compare August 22, 2019 15:11

sandeep-katta force-pushed the jarIssue branch from 533eeeb to 8eb6310 Compare September 10, 2019 16:54

sandeep-katta force-pushed the jarIssue branch from 8eb6310 to cf13bfa Compare September 10, 2019 17:40

Fix the class loader issue

3c06e89

AngersZhuuuu and others added 3 commits September 10, 2019 23:25

spark-28954 Use SessionResourceLoader to cover HIVEAUXJARS conf jars

be6d653

fix review comments

30ff7a2

avoild using ClassOf[HiveCode]

47f8632

sandeep-katta force-pushed the jarIssue branch from cf13bfa to 47f8632 Compare September 10, 2019 17:58

avoild using ClassOf[HiveCode]

4a79580

wangyum approved these changes Sep 11, 2019

View reviewed changes

wangyum closed this in 7e61425 Sep 12, 2019

AngersZhuuuu mentioned this pull request Sep 12, 2019

[SPARK-29015][SQL][test-hadoop3.2]Reset class loader after initializing SessionState for built-in Hive 2.3 #25775

Closed

[SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive #25542

[SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive #25542

Conversation

sandeep-katta commented Aug 21, 2019 • edited by HyukjinKwon Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

wangyum left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sandeep-katta commented Aug 21, 2019

Uh oh!

wangyum commented Aug 21, 2019

Uh oh!

sandeep-katta commented Aug 21, 2019

Uh oh!

dongjoon-hyun commented Aug 21, 2019

Uh oh!

dongjoon-hyun commented Aug 21, 2019

Uh oh!

SparkQA commented Aug 21, 2019

Uh oh!

SparkQA commented Aug 21, 2019

Uh oh!

wangyum commented Aug 22, 2019

Uh oh!

sandeep-katta commented Aug 22, 2019

Uh oh!

wangyum commented Aug 22, 2019

Uh oh!

wangyum commented Aug 22, 2019

Uh oh!

dongjoon-hyun commented Aug 22, 2019

Uh oh!

SparkQA commented Aug 22, 2019

Uh oh!

wangyum commented Aug 22, 2019

Uh oh!

wangyum Aug 22, 2019

Choose a reason for hiding this comment

Uh oh!

wangyum Aug 22, 2019

Choose a reason for hiding this comment

Uh oh!

wangyum Aug 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sandeep-katta Aug 22, 2019

Choose a reason for hiding this comment

Uh oh!

wangyum Aug 22, 2019

Choose a reason for hiding this comment

Uh oh!

pvk2727 Aug 27, 2019

Choose a reason for hiding this comment

Uh oh!

sandeep-katta Sep 3, 2019

Choose a reason for hiding this comment

Uh oh!

juliuszsompolski Aug 22, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 22, 2019

Uh oh!

SparkQA commented Aug 22, 2019

Uh oh!

dongjoon-hyun commented Aug 24, 2019

Uh oh!

SparkQA commented Aug 24, 2019

Uh oh!

SparkQA commented Sep 10, 2019

Uh oh!

SparkQA commented Sep 10, 2019

Uh oh!

SparkQA commented Sep 10, 2019

sandeep-katta commented Aug 21, 2019 •

edited by HyukjinKwon

Loading

wangyum left a comment •

edited

Loading

wangyum Aug 22, 2019 •

edited

Loading