[SPARK-21855][YARN] Should print error when upload same file multiple tim…

[caneGuy]

…es to yarn

What changes were proposed in this pull request?

Now when submit job with yarn,and upload same file multiple times.We will throw exception but logging level is warn.
This pr change logging level to error,since warning is misleading.

How was this patch tested?

N/A

[AmplabJenkins]

Can one of the admins verify this patch?

[jerryshao]

@caneGuy why do you think it is misleading?

[caneGuy]

@jerryshao There is a user of our cluster who first saw warning he ignore it,if we print error it may be more clearly.

[jerryshao]

But as I remembered the same name file will be ignored when met again. This should not be a fatal issue, right?

[caneGuy]

Job submitting will fail since exception threw.I think this is fatal issue.

[jerryshao]

I tried locally with same name jar uploaded twice, the spark application can be started. Can you please paste your exception here?

[caneGuy]

17/08/29 10:06:26 INFO yarn.Client: Deleting staging directory /spark/xxx/staging/hdfs_admin/.sparkStaging/application_1500536001739_0003 Exception in thread "main" java.lang.IllegalArgumentException: requirement failed at scala.Predef$.require(Predef.scala:221) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6$$anonfun$apply$3.apply(Client.scala:516) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6$$anonfun$apply$3.apply(Client.scala:514) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6.apply(Client.scala:514) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6.apply(Client.scala:512) at scala.collection.immutable.List.foreach(List.scala:318) at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:512) at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:771) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:144) at org.apache.spark.deploy.yarn.Client.run(Client.scala:1088) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1148) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:749) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
Cluster version is 1.6,i have checked the logic of master,it has not been changed.

[srowen]

This is not an error condition. 'someone ignored it' is not sufficient reason.

[jerryshao]

@caneGuy can you please check this JIRA (https://issues.apache.org/jira/browse/SPARK-14423), I remembered I fixed this issue before.

Can you please first try latest version of Spark, if the issue occurs please describe the step to reproduce this issue in JIRA. Thanks!

[caneGuy]

The actual reason i want change the log level is that,if you threw an exception ,then this indicate it is a fatal error,we should not just print a warning. @srowen

[caneGuy]

Ok i will try later,thanks @jerryshao

[caneGuy]

@jerryshao
reproduce step is below:
1、add --files when submit jobs
2、add file with same name in --files then exception will throw
After looking into your pr,that fix this issue.
I will backport 14423 to our inner branch,and can you backport your fix to master。

[jerryshao]

I think it is already in master branch @caneGuy .

[caneGuy]

Yes from git blame i found that.And i found a follow-up(maybe follow-up?) pr spark-18099

[jerryshao]

Yes, maybe. You can take a try locally.