[SPARK-16349][sql] Fall back to isolated class loader when classes not found. #14020
Conversation
…t found. Some Hadoop classes needed by the Hive metastore client jars are not present in Spark's packaging (for example, "org/apache/hadoop/mapred/MRVersion"). So if the parent class loader fails to find a class, try to load it from the isolated class loader, in case it's available there. I also took the opportunity to remove the HIVE_EXECUTION_VERSION constant since it's not used anywhere. Tested by setting spark.sql.hive.metastore.jars to local paths with Hive/Hadoop libraries and verifying that Spark can talk to the metastore.
|
Test build #61640 has finished for PR 14020 at commit
|
|
retest this please |
|
Test build #61645 has finished for PR 14020 at commit
|
|
Might be a legitimate test failure, will look later. |
| @@ -264,7 +270,7 @@ private[hive] class IsolatedClientLoader( | |||
| throw new ClassNotFoundException( | |||
| s"$cnf when creating Hive client using classpath: ${execJars.mkString(", ")}\n" + | |||
| "Please make sure that jars for your version of hive and hadoop are included in the " + | |||
There was a problem hiding this comment.
Just a nitpick...should 'hive' be Hive as the line above + Hadoop?
|
Test build #61768 has finished for PR 14020 at commit
|
|
ping |
|
I'm not familiar with this part, cc @liancheng to take a look |
|
I'll leave this open until EOD then push the change. |
|
Will putting that jar in Spark's classpath work? Seems so? |
| baseClassLoader.loadClass(name) | ||
| } catch { | ||
| case _: ClassNotFoundException => | ||
| super.loadClass(name, resolve) |
There was a problem hiding this comment.
Which classloader does this call delegate to?
There was a problem hiding this comment.
This will delegate to the loader using the user-provided jars from the metastore jars configuration.
|
also cc @marmbrus |
Yes but the whole point of the configuration is to not pollute Spark's class loader. |
|
lgtm. Merging to master |
Some Hadoop classes needed by the Hive metastore client jars are not present
in Spark's packaging (for example, "org/apache/hadoop/mapred/MRVersion"). So
if the parent class loader fails to find a class, try to load it from the
isolated class loader, in case it's available there.
Tested by setting spark.sql.hive.metastore.jars to local paths with Hive/Hadoop
libraries and verifying that Spark can talk to the metastore.