Uh oh!
There was an error while loading. Please reload this page.
SPARK-1929 DAGScheduler suspended by local task OOM - #883
Conversation
AmplabJenkins
commented
May 26, 2014
Can one of the admins verify this patch? |
rxin
commented
May 26, 2014
Jenkins, add to whitelist. |
AmplabJenkins
commented
May 26, 2014
Merged build triggered. |
AmplabJenkins
commented
May 26, 2014
Merged build started. |
AmplabJenkins
commented
May 26, 2014
Merged build finished. All automated tests passed. |
AmplabJenkins
commented
May 26, 2014
All automated tests passed. |
There was a problem hiding this comment.
When it is actually OOM, should we try to avoid allocating new objects to make sure it can recover gracefully?
There was a problem hiding this comment.
Yes, maybe trying to catch the OOM error is not a good idea, but here we can't distinguish the exception whether thrown by local task or by driver itself. And we just try to recover DAG scheduler from the previous situation.
Any advice would be appreciated!
There was a problem hiding this comment.
What if instead of allocating more stuff, you just put the following:
valexception=newSparkException("Out of memory exception", oom)There was a problem hiding this comment.
Thanks @rxin, I have removed the redundant memory allocations.
AmplabJenkins
commented
May 27, 2014
Merged build triggered. |
AmplabJenkins
commented
May 27, 2014
Merged build started. |
There was a problem hiding this comment.
Can you change the error message to "Local job aborted due to out of memory error"
rxin
commented
May 27, 2014
Actually never mind I will just do that when I commit the change. Merging this into master. Thanks! |
rxin
commented
May 27, 2014
AmplabJenkins
commented
May 27, 2014
Merged build finished. |
AmplabJenkins
commented
May 27, 2014
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15219/ |
DAGScheduler does not handle local task OOM properly, and will wait for the job result forever. Author: Zhen Peng <zhenpeng01@baidu.com> Closesapache#883 from zhpengg/bugfix-dag-scheduler-oom and squashes the following commits: 76f7eda [Zhen Peng] remove redundant memory allocations aa63161 [Zhen Peng] SPARK-1929 DAGScheduler suspended by local task OOM
DAGScheduler does not handle local task OOM properly, and will wait for the job result forever. Author: Zhen Peng <zhenpeng01@baidu.com> Closesapache#883 from zhpengg/bugfix-dag-scheduler-oom and squashes the following commits: 76f7eda [Zhen Peng] remove redundant memory allocations aa63161 [Zhen Peng] SPARK-1929 DAGScheduler suspended by local task OOM
Co-authored-by: Egor Krivokon <>
Co-authored-by: Egor Krivokon <>
Co-authored-by: Egor Krivokon <>
DAGScheduler does not handle local task OOM properly, and will wait for the job result forever.