Uh oh!
There was an error while loading. Please reload this page.
[SPARK-20900][YARN] Catch IllegalArgumentException thrown by new Path in ApplicationMaster - #18124
[SPARK-20900][YARN] Catch IllegalArgumentException thrown by new Path in ApplicationMaster#18124nonsleepr wants to merge 1 commit into
Conversation
srowen
commented
May 26, 2017
Why? instead of check the value of the env variable directly. |
AmplabJenkins
commented
May 26, 2017
Can one of the admins verify this patch? |
nonsleepr
commented
May 26, 2017
@srowen Because |
I'm not sure how this could be happened, "SPARK_YARN_STAGING_DIR" is a Spark internal environment variable which should not be empty, unless you deliberately unset it. In the JIRA you mentioned you directly running BTW seems you submitted a PR against branch 2.1, why not against master branch? |
fe9d709 to
3c746eaComparenonsleepr
commented
May 30, 2017
@jerryshao You're right, it won't be empty if I run the app using spark-submit. The whole reason I encountered this bug is that in my project I'm running Spark job remotely (via YARN's REST API) from the node which doesn't have Spark distro. |
srowen
commented
May 30, 2017
I don't think we'd change this then if it's kind of complicating the code and for a non-standard usage anyway. |
# What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017Closesapache#11459Closesapache#13833Closesapache#13720Closesapache#12506Closesapache#12456Closesapache#12252Closesapache#17689Closesapache#17791Closesapache#18163Closesapache#17640Closesapache#17926Closesapache#18163Closesapache#12506Closesapache#18044Closesapache#14036Closesapache#15831Closesapache#14461Closesapache#17638Closesapache#18222 Added: Closesapache#18045Closesapache#18061Closesapache#18010Closesapache#18041Closesapache#18124Closesapache#18130Closesapache#12217 Added: Closesapache#16291Closesapache#17480Closesapache#14995 Added: Closesapache#12835Closesapache#17141 ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#18223 from HyukjinKwon/close-stale-prs.
What changes were proposed in this pull request?
Catch
IllegalArgumentExceptionwhich might be thrown ifSPARK_YARN_STAGING_DIRis not set or empty.How was this patch tested?
Manually tested by running
ApplicationMasterwithSPARK_YARN_STAGING_DIRunset.