Uh oh!
There was an error while loading. Please reload this page.
[SPARK-16349][sql] Fall back to isolated class loader when classes not found. - #14020
[SPARK-16349][sql] Fall back to isolated class loader when classes not found.#14020vanzin wants to merge 2 commits into
Conversation
…t found. Some Hadoop classes needed by the Hive metastore client jars are not present in Spark's packaging (for example, "org/apache/hadoop/mapred/MRVersion"). So if the parent class loader fails to find a class, try to load it from the isolated class loader, in case it's available there. I also took the opportunity to remove the HIVE_EXECUTION_VERSION constant since it's not used anywhere. Tested by setting spark.sql.hive.metastore.jars to local paths with Hive/Hadoop libraries and verifying that Spark can talk to the metastore.
SparkQA
commented
Jul 1, 2016
Test build #61640 has finished for PR 14020 at commit
|
vanzin
commented
Jul 2, 2016
retest this please |
SparkQA
commented
Jul 2, 2016
Test build #61645 has finished for PR 14020 at commit
|
vanzin
commented
Jul 2, 2016
Might be a legitimate test failure, will look later. |
| @@ -264,7 +270,7 @@ private[hive] class IsolatedClientLoader( | |||
| throw new ClassNotFoundException( | |||
| s"$cnf when creating Hive client using classpath: ${execJars.mkString(", ")}\n" + | |||
| "Please make sure that jars for your version of hive and hadoop are included in the " + | |||
There was a problem hiding this comment.
Just a nitpick...should 'hive' be Hive as the line above + Hadoop?
SparkQA
commented
Jul 5, 2016
Test build #61768 has finished for PR 14020 at commit
|
vanzin
commented
Jul 5, 2016
vanzin
commented
Jul 6, 2016
ping |
cloud-fan
commented
Jul 8, 2016
I'm not familiar with this part, cc @liancheng to take a look |
vanzin
commented
Jul 11, 2016
I'll leave this open until EOD then push the change. |
yhuai
commented
Jul 11, 2016
Will putting that jar in Spark's classpath work? Seems so? |
| baseClassLoader.loadClass(name) | ||
| } catch { | ||
| case _: ClassNotFoundException => | ||
| super.loadClass(name, resolve) |
There was a problem hiding this comment.
Which classloader does this call delegate to?
There was a problem hiding this comment.
This will delegate to the loader using the user-provided jars from the metastore jars configuration.
yhuai
commented
Jul 11, 2016
also cc @marmbrus |
vanzin
commented
Jul 11, 2016
Yes but the whole point of the configuration is to not pollute Spark's class loader. |
yhuai
commented
Jul 11, 2016
lgtm. Merging to master |
Some Hadoop classes needed by the Hive metastore client jars are not present
in Spark's packaging (for example, "org/apache/hadoop/mapred/MRVersion"). So
if the parent class loader fails to find a class, try to load it from the
isolated class loader, in case it's available there.
Tested by setting spark.sql.hive.metastore.jars to local paths with Hive/Hadoop
libraries and verifying that Spark can talk to the metastore.