Skip to content

[SPARK-20900][YARN] Catch IllegalArgumentException thrown by new Path in ApplicationMaster - #18124

Closed
nonsleepr wants to merge 1 commit into
apache:masterfrom
nonsleepr:spark-yarn-staging-dir
Closed

[SPARK-20900][YARN] Catch IllegalArgumentException thrown by new Path in ApplicationMaster#18124
nonsleepr wants to merge 1 commit into
apache:masterfrom
nonsleepr:spark-yarn-staging-dir

Conversation

@nonsleepr

Copy link
Copy Markdown

What changes were proposed in this pull request?

Catch IllegalArgumentException which might be thrown if SPARK_YARN_STAGING_DIR is not set or empty.

How was this patch tested?

Manually tested by running ApplicationMaster with SPARK_YARN_STAGING_DIR unset.

@srowen

Copy link
Copy Markdown
Member

Why? instead of check the value of the env variable directly.

@AmplabJenkins

Copy link
Copy Markdown

Can one of the admins verify this patch?

@nonsleepr

Copy link
Copy Markdown
Author

@srowen Because Path will perform multiple checks. E.g. check for null or empty string and then URI syntax exception.

@jerryshao

jerryshao commented May 27, 2017

Copy link
Copy Markdown
Contributor

I'm not sure how this could be happened, "SPARK_YARN_STAGING_DIR" is a Spark internal environment variable which should not be empty, unless you deliberately unset it.

In the JIRA you mentioned you directly running ApplicationMaster, so that you met this issue, I'm not sure why you need to directly call ApplicationMaster?

BTW seems you submitted a PR against branch 2.1, why not against master branch?

@nonsleepr
nonsleeprforce-pushed the spark-yarn-staging-dir branch from fe9d709 to 3c746eaCompareMay 30, 2017 14:47
@nonsleepr
nonsleepr changed the base branch from branch-2.1 to masterMay 30, 2017 14:48
@nonsleepr

Copy link
Copy Markdown
Author

@jerryshao You're right, it won't be empty if I run the app using spark-submit. The whole reason I encountered this bug is that in my project I'm running Spark job remotely (via YARN's REST API) from the node which doesn't have Spark distro.
My motivation for running it is another story but in short: I don't want to upload my keytab to HDFS to run spark-submit and instead I request delegation tokens for the services my job uses.

@srowen

Copy link
Copy Markdown
Member

I don't think we'd change this then if it's kind of complicating the code and for a non-standard usage anyway.

@srowensrowen mentioned this pull request Jun 7, 2017
@asfgitasfgit closed this in b771fedJun 8, 2017
zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
# What changes were proposed in this pull request?
This PR proposes to close stale PRs, mostly the same instances with apache#18017Closesapache#11459Closesapache#13833Closesapache#13720Closesapache#12506Closesapache#12456Closesapache#12252Closesapache#17689Closesapache#17791Closesapache#18163Closesapache#17640Closesapache#17926Closesapache#18163Closesapache#12506Closesapache#18044Closesapache#14036Closesapache#15831Closesapache#14461Closesapache#17638Closesapache#18222
Added:
Closesapache#18045Closesapache#18061Closesapache#18010Closesapache#18041Closesapache#18124Closesapache#18130Closesapache#12217
Added:
Closesapache#16291Closesapache#17480Closesapache#14995
Added:
Closesapache#12835Closesapache#17141
## How was this patch tested?
N/A
Author: hyukjinkwon <gurwls223@gmail.com>
Closesapache#18223 from HyukjinKwon/close-stale-prs.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@nonsleepr@srowen@AmplabJenkins@jerryshao