[SPARK-2815]: Compilation failed upon the hadoop version 2.0.0-cdh4.5.0

[witgo]

No description provided.

[SparkQA]

QA tests have started for PR 1754. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17815/consoleFull

[srowen]

This piece of code in the build is legacy, as the note in the following line says. I don't think it needs to be changed. The OP is building with the wrong command; that's the problem.

[witgo]

This patch is to be consistent with the 1.0 version.
See #772.

[SparkQA]

QA results for PR 1754:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17815/consoleFull

[pwendell]

I think this is an intermediate YARN version that is different from both the yarn-alpha and yarn-stable API's. @witgo what if you apply the patch here - does it work?

https://github.com/apache/spark/pull/151/files

In terms of merging this - I'm not sure we want to support 3 different YARN API's in the build and I think CDH itself said that YARN was not stable/supported here, so I'm not sure if we want to merge that patch. I am curious though whether it fixes it for you.

[witgo]

This PR makes sbt' s behavior is consistent with building-with-maven.md description.

YARN version	Profile required
0.23.x to 2.1.x	yarn-alpha
2.2.x and later	yarn

[srowen]

If there is ever a Hadoop 2.10, this pattern would consider it a YARN alpha version. Better to match against 2\\.[01]\\.[0-9]+ or something. This won't address the actual issue in the OP. It looks like a bit of cleanup, but in a deprecated code path. It would let Hadoop 2.0 / 2.1 be "supported" but they aren't actually.

[witgo]

@pwendell#151 compilation fails.
There seems to be infinite loop:
SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 SPARK_YARN=true SPARK_HIVE=true sbt/sbt clean assembly ->

java.lang.StackOverflowError
at scala.reflect.internal.HasFlags$class.isSynthetic(HasFlags.scala:115)
at scala.reflect.internal.Symbols$Symbol.isSynthetic(Symbols.scala:112)
at xsbt.ExtractUsedNames.eligibleAsUsedName(ExtractUsedNames.scala:121)
at xsbt.ExtractUsedNames.handleClassicTreeNode$1(ExtractUsedNames.scala:87)
at xsbt.ExtractUsedNames.xsbt$ExtractUsedNames$$handleTreeNode$1(ExtractUsedNames.scala:97)
at xsbt.ExtractUsedNames$$anonfun$handleMacroExpansion$1$2.apply(ExtractUsedNames.scala:64)
at xsbt.ExtractUsedNames$$anonfun$handleMacroExpansion$1$2.apply(ExtractUsedNames.scala:64)
at scala.reflect.internal.Trees$ForeachTreeTraverser.traverse(Trees.scala:1489)
at scala.reflect.internal.Trees$ForeachTreeTraverser.traverse(Trees.scala:1487)
at scala.reflect.internal.Trees$class.itraverse(Trees.scala:1174)
at scala.reflect.internal.SymbolTable.itraverse(SymbolTable.scala:13)
at scala.reflect.internal.SymbolTable.itraverse(SymbolTable.scala:13)
at scala.reflect.api.Trees$Traverser.traverse(Trees.scala:2825)
at scala.reflect.internal.Trees$ForeachTreeTraverser.traverse(Trees.scala:1490)
at scala.reflect.internal.Trees$TreeContextApiImpl.foreach(Trees.scala:80)
at xsbt.ExtractUsedNames.handleMacroExpansion$1(ExtractUsedNames.scala:64)
at xsbt.ExtractUsedNames.xsbt$ExtractUsedNames$$handleTreeNode$1(ExtractUsedNames.scala:95)
at xsbt.ExtractUsedNames$$anonfun$handleMacroExpansion$1$2.apply(ExtractUsedNames.scala:64)
at xsbt.ExtractUsedNames$$anonfun$handleMacroExpansion$1$2.apply(ExtractUsedNames.scala:64)
at scala.reflect.internal.Trees$ForeachTreeTraverser.traverse(Trees.scala:1489)
at scala.reflect.internal.Trees$ForeachTreeTraverser.traverse(Trees.scala:1487)
at scala.reflect.api.Trees$Traverser.traverseTrees(Trees.scala:2829)
at scala.reflect.internal.Trees$class.itraverse(Trees.scala:1174)
at scala.reflect.internal.SymbolTable.itraverse(SymbolTable.scala:13)
at scala.reflect.internal.SymbolTable.itraverse(SymbolTable.scala:13)
at scala.reflect.api.Trees$Traverser.traverse(Trees.scala:2825)
at scala.reflect.internal.Trees$ForeachTreeTraverser.traverse(Trees.scala:1490)
at scala.reflect.internal.Trees$TreeContextApiImpl.foreach(Trees.scala:80)
at xsbt.ExtractUsedNames.handleMacroExpansion$1(ExtractUsedNames.scala:64)
at xsbt.ExtractUsedNames.xsbt$ExtractUsedNames$$handleTreeNode$1(ExtractUsedNames.scala:95)
at xsbt.ExtractUsedNames$$anonfun$handleMacroExpansion$1$2.apply(ExtractUsedNames.scala:64)
at xsbt.ExtractUsedNames$$anonfun$handleMacroExpansion$1$2.apply(ExtractUsedNames.scala:64)
at scala.reflect.internal.Trees$ForeachTreeTraverser.traverse(Trees.scala:1489)
at scala.reflect.internal.Trees$ForeachTreeTraverser.traverse(Trees.scala:1487)
at scala.reflect.api.Trees$Traverser.traverseTrees(Trees.scala:2829)
at scala.reflect.internal.Trees$class.itraverse(Trees.scala:1174)
at scala.reflect.internal.SymbolTable.itraverse(SymbolTable.scala:13)
at scala.reflect.internal.SymbolTable.itraverse(SymbolTable.scala:13)
at scala.reflect.api.Trees$Traverser.traverse(Trees.scala:2825)
at scala.reflect.internal.Trees$ForeachTreeTraverser.traverse(Trees.scala:1490)
at scala.reflect.internal.Trees$TreeContextApiImpl.foreach(Trees.scala:80)
at xsbt.ExtractUsedNames.handleMacroExpansion$1(ExtractUsedNames.scala:64)
at xsbt.ExtractUsedNames.xsbt$ExtractUsedNames$$handleTreeNode$1(ExtractUsedNames.scala:95)
at xsbt.ExtractUsedNames$$anonfun$handleMacroExpansion$1$2.apply(ExtractUsedNames.scala:64)
at xsbt.ExtractUsedNames$$anonfun$handleMacroExpansion$1$2.apply(ExtractUsedNames.scala:64)
at scala.reflect.internal.Trees$ForeachTreeTraverser.traverse(Trees.scala:1489)

[witgo]

We need to explicitly pointed out that spark does not support the version 2.0.x and 2.1.x of yarn ?

[srowen]

@witgo I don't think #151 is to be committed, if I understand correctly. It's not 100% clear which versions of YARN 2.0.x actually work with yarn-alpha, and which if any work with yarn. If anything it's worth a note that the pre-stable YARN versions are not guaranteed to work, but might.

[pwendell]

I don't mind putting this one in (it's simple enough and might lower the bar for anyone trying to go this route). But the regex needs to be fixed, otherwise it will match 2.10.X.

[witgo]

@pwendell I have updated the regular expression.

[SparkQA]

QA tests have started for PR 1754 at commit 727062e.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1754 at commit 727062e.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.