Uh oh!
There was an error while loading. Please reload this page.
[SPARK-14914] Fix Command too long for windows. Especially for test cases. - #12694
[SPARK-14914] Fix Command too long for windows. Especially for test cases. #12694taoli91 wants to merge 11 commits into
Conversation
AmplabJenkins
commented
Apr 26, 2016
Can one of the admins verify this patch? |
| * Create a jar file at the given path, containing a manifest with a classpath | ||
| * that references all specified entries. | ||
| */ | ||
| def createShortClassPath(tempDir: File, classPath: String) : String = { |
There was a problem hiding this comment.
Maybe, def createShortClassPath(tempDir: File, classPath: String): String = {
vanzin
commented
Aug 5, 2016
I'm not a fan of this approach. This means that every invocation of the launcher code will create temp files and have to deal with cleaning it up; except the launcher cannot do it, because the jvm running SparkSubmit needs it. And now SparkSubmit needs to know to clean it up. That's adding a lot of complexity. Setting the driver / executor classpaths in tests should be mostly unnecessary at this point, since the root pom file sets For the generic use case it is possible for someone could create a command line that exceeds the limit; but I think a different approach would be better. Maybe generating a temp config file, with the contents of both the user-provided config and the defaults from Spark's conf dir, but that has the same problem with "who deletes the temp conf file". On a side note, there's a ton of style issues with your patch, please try to follow the style of the code you're modifying. |
vanzin
commented
Aug 5, 2016
Update: I thought about using argument files, but they're only supported by javac (
|
srowen
commented
Aug 15, 2016
Let's close this PR |
Closesapache#10995Closesapache#13658Closesapache#14505Closesapache#14536Closesapache#12753Closesapache#14449Closesapache#12694Closesapache#12695Closesapache#14810
What changes were proposed in this pull request?
Windows has a limitation of 8192 characters on command line. This limitation will fail the test cases related to command builder in some test cases. This pull request adopt the method in Hadoop, which create a Jar that ref all the dependencies in the manifest to shorten the class path passing to the command line.
How was this patch tested?
Unit tests on windows and Linux. Note that this commit can't fix all the unit test errors on windows.