[SPARK-20343][BUILD] Avoid Unidoc build only if Hadoop 2.6 is explicitly set in SBT build

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes two things as below:

• Avoid Unidoc build only if Hadoop 2.6 is explicitly set in SBT build

  Due to a different dependency resolution in SBT & Unidoc by an unknown reason, the documentation build fails on a specific machine & environment in Jenkins but it was unable to reproduce.

  So, this PR just checks an environment variable AMPLAB_JENKINS_BUILD_PROFILE that is set in Hadoop 2.6 SBT build against branches on Jenkins, and then disables Unidoc build. Note that PR builder will still build it with Hadoop 2.6 & SBT.

  ========================================================================
  Building Unidoc API Documentation
  ========================================================================
  [info] Building Spark unidoc (w/Hive 1.2.1) using SBT with these arguments: -Phadoop-2.6 -Pmesos -Pkinesis-asl -Pyarn -Phive-thriftserver -Phive unidoc
  Using /usr/java/jdk1.8.0_60 as default JAVA_HOME.
  ...
  

  I checked the environment variables from the logs (first bit) as below:

  • spark-master-test-sbt-hadoop-2.6 (this one is being failed) - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/lastBuild/consoleFull

  JAVA_HOME=/usr/java/jdk1.8.0_60
  JAVA_7_HOME=/usr/java/jdk1.7.0_79
  SPARK_BRANCH=master
  AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.6 <- I use this variable
  AMPLAB_JENKINS="true"
  

  • spark-master-test-sbt-hadoop-2.7 - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/lastBuild/consoleFull

  JAVA_HOME=/usr/java/jdk1.8.0_60
  JAVA_7_HOME=/usr/java/jdk1.7.0_79
  SPARK_BRANCH=master
  AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.7
  AMPLAB_JENKINS="true"
  

  • spark-master-test-maven-hadoop-2.6 - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.6/lastBuild/consoleFull

  JAVA_HOME=/usr/java/jdk1.8.0_60
  JAVA_7_HOME=/usr/java/jdk1.7.0_79
  HADOOP_PROFILE=hadoop-2.6
  HADOOP_VERSION=
  SPARK_BRANCH=master
  AMPLAB_JENKINS="true"
  

  • spark-master-test-maven-hadoop-2.7 - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/lastBuild/consoleFull

  JAVA_HOME=/usr/java/jdk1.8.0_60
  JAVA_7_HOME=/usr/java/jdk1.7.0_79
  HADOOP_PROFILE=hadoop-2.7
  HADOOP_VERSION=
  SPARK_BRANCH=master
  AMPLAB_JENKINS="true"
  

  • PR builder - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75843/consoleFull

  JENKINS_MASTER_HOSTNAME=amp-jenkins-master
  JAVA_HOME=/usr/java/jdk1.8.0_60
  JAVA_7_HOME=/usr/java/jdk1.7.0_79
  

  Assuming from other logs in branch-2.1

  • SBT & Hadoop 2.6 against branch-2.1 https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.1-test-sbt-hadoop-2.6/lastBuild/consoleFull

    JAVA_HOME=/usr/java/jdk1.8.0_60
    JAVA_7_HOME=/usr/java/jdk1.7.0_79
    SPARK_BRANCH=branch-2.1
    AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.6
    AMPLAB_JENKINS="true"
    

  • Maven & Hadoop 2.6 against branch-2.1 https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.1-test-maven-hadoop-2.6/lastBuild/consoleFull

    JAVA_HOME=/usr/java/jdk1.8.0_60
    JAVA_7_HOME=/usr/java/jdk1.7.0_79
    HADOOP_PROFILE=hadoop-2.6
    HADOOP_VERSION=
    SPARK_BRANCH=branch-2.1
    AMPLAB_JENKINS="true"
    

  We have been using the same convention for those variables. These are actually being used in run-tests.py script - here https://github.com/apache/spark/blob/master/dev/run-tests.py#L519-L520

• Revert the previous try

  After [SPARK-20343][BUILD] Force Avro 1.7.7 in sbt build to resolve build failure in SBT Hadoop 2.6 master on Jenkins #17651, it seems the build still fails on SBT Hadoop 2.6 master.

  I am unable to reproduce this - [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc build on Jenkins #17477 (comment) and the reviewer was too. So, this got merged as it looks the only way to verify this is to merge it currently (as no one seems able to reproduce this).

How was this patch tested?

I only checked is_hadoop_version_2_6 = os.environ.get("AMPLAB_JENKINS_BUILD_PROFILE") == "hadoop2.6" is working fine as expected as below:

>>>importcollections>>>os=collections.namedtuple('os', 'environ')(environ={"AMPLAB_JENKINS_BUILD_PROFILE": "hadoop2.6"})
>>>print(notos.environ.get("AMPLAB_JENKINS_BUILD_PROFILE") =="hadoop2.6")
False>>>os=collections.namedtuple('os', 'environ')(environ={"AMPLAB_JENKINS_BUILD_PROFILE": "hadoop2.7"})
>>>print(notos.environ.get("AMPLAB_JENKINS_BUILD_PROFILE") =="hadoop2.6")
True>>>os=collections.namedtuple('os', 'environ')(environ={})
>>>print(notos.environ.get("AMPLAB_JENKINS_BUILD_PROFILE") =="hadoop2.6")
True

I tried many ways but I was unable to reproduce this in my local. Sean also tried the way I did but he was also unable to reproduce this.

Please refer the comments in #17477 (comment)

[SparkQA]

Test build #75894 has finished for PR 17669 at commit f81b9fc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #75929 has started for PR 17669 at commit 0c9dd50.

[HyukjinKwon]

@srowen, could you take a look please? I think this is a simplest temp fix but might look a bit funny though.

[HyukjinKwon]

retest this please

[srowen]

Yeah looks like the best hack solution to get this working for now.

I am not sure why the build fails, but it's not Hadoop 2.6 or SBT per se, and we can't reproduce it outside Jenkins either way. Therefore it seems reasonable to work around it in Jenkins, because building the docs is pretty valuable.

[SparkQA]

Test build #75931 has finished for PR 17669 at commit 0c9dd50.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master, let's see if it works

[srowen]

Looks good. I back-ported this to 2.2 as well to match up with the other logical changes.

[HyukjinKwon]

Thank you @srowen.