Uh oh!
There was an error while loading. Please reload this page.
SPARK-1877: ClassNotFoundException when loading RDD with serialized obje... - #821
SPARK-1877: ClassNotFoundException when loading RDD with serialized obje...#821ghidi wants to merge 2 commits into
Conversation
AmplabJenkins
commented
May 19, 2014
Can one of the admins verify this patch? |
mateiz
commented
May 19, 2014
Jenkins, this is ok to test |
AmplabJenkins
commented
May 19, 2014
Merged build triggered. |
AmplabJenkins
commented
May 19, 2014
Merged build started. |
AmplabJenkins
commented
May 19, 2014
Merged build finished. |
AmplabJenkins
commented
May 19, 2014
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15086/ |
tdas
commented
May 19, 2014
Is it possible to give a unit test that highlights this problem? Since it works locally, you can try using "local-cluster" mode (see DistributedSuite if you dont know what it is) to reproduce and test it. |
There was a problem hiding this comment.
Could you change this to Utils.getContextOrSparkClassLoader? It's not always the case that the context classloader is set, and the utility function deals with that correctly.
pwendell
commented
May 20, 2014
LGTM - I pending a small change (we have a utility function to get the classloader). |
…objects Updated version of #821 Author: Tathagata Das <tathagata.das1565@gmail.com> Author: Ghidireac <bogdang@u448a5b0a73d45358d94a.ant.amazon.com> Closes#835 from tdas/SPARK-1877 and squashes the following commits: f346f71 [Tathagata Das] Addressed Patrick's comments. fee0c5d [Ghidireac] SPARK-1877: ClassNotFoundException when loading RDD with serialized objects (cherry picked from commit 52eb54d) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
…objects Updated version of #821 Author: Tathagata Das <tathagata.das1565@gmail.com> Author: Ghidireac <bogdang@u448a5b0a73d45358d94a.ant.amazon.com> Closes#835 from tdas/SPARK-1877 and squashes the following commits: f346f71 [Tathagata Das] Addressed Patrick's comments. fee0c5d [Ghidireac] SPARK-1877: ClassNotFoundException when loading RDD with serialized objects
…with serialized objects
AmplabJenkins
commented
May 20, 2014
Merged build triggered. |
AmplabJenkins
commented
May 20, 2014
Merged build started. |
ghidi
commented
May 20, 2014
I changed Thread.getContextClassLoader with Utils.getContextOrSparkClassLoader. |
tdas
commented
May 20, 2014
AmplabJenkins
commented
May 20, 2014
Merged build finished. All automated tests passed. |
AmplabJenkins
commented
May 20, 2014
All automated tests passed. |
…objects Updated version of apache#821 Author: Tathagata Das <tathagata.das1565@gmail.com> Author: Ghidireac <bogdang@u448a5b0a73d45358d94a.ant.amazon.com> Closesapache#835 from tdas/SPARK-1877 and squashes the following commits: f346f71 [Tathagata Das] Addressed Patrick's comments. fee0c5d [Ghidireac] SPARK-1877: ClassNotFoundException when loading RDD with serialized objects
When I load a RDD that has custom serialized objects, Spark throws ClassNotFoundException. This happens only when Spark is deployed as a standalone cluster, it works fine when Spark is local.
I debugged the issue and I noticed that ObjectInputStream.resolveClass does not use ExecutorURLClassLoader set by SparkSubmit. You have to explicitly set the classloader in SparkContext.objectFile for ObjectInputStream when deserializing objects.