Skip to content

[SPARK-32058][BUILD][SQL][test-hive1.2][test-hadoop2.7][FOLLOWUP] Set hadoop 2.7.4 for hive 1.2 profile - #29006

Closed
yaooqinn wants to merge 3 commits into
apache:masterfrom
yaooqinn:hive12
Closed

[SPARK-32058][BUILD][SQL][test-hive1.2][test-hadoop2.7][FOLLOWUP] Set hadoop 2.7.4 for hive 1.2 profile#29006
yaooqinn wants to merge 3 commits into
apache:masterfrom
yaooqinn:hive12

Conversation

@yaooqinn

@yaooqinnyaooqinn commented Jul 6, 2020•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR set Hadoop 2.7.4 for hive 1.2 profile, because the default Hadoop 3.2.0 is incompatiable

Why are the changes needed?

The test-hive1.2 tag works wrongly these days

[info] Building Spark using SBT with these arguments: -Phadoop-3.2 -Phive-1.2 -Phadoop-cloud -Phive -Pkinesis-asl -Pkubernetes -Phive-thriftserver -Pyarn -Pspark-ganglia-lgpl -Pmesos test:package streaming-kinesis-asl-assembly/assembly

It blocks #28912 (comment) and #28963 (comment)

Does this PR introduce any user-facing change?

No

How was this patch tested?

passing Jenkins with test-hive-1.2 tag

@yaooqinn

yaooqinn commented Jul 6, 2020•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

cc @dongjoon-hyun@cloud-fan@maropu thanks

@yaooqinnyaooqinn mentioned this pull request Jul 6, 2020
Closed
HyukjinKwon
HyukjinKwon reviewed Jul 6, 2020
View reviewed changes
Comment threadpom.xml Outdated
<datanucleus-core.version>3.2.10</datanucleus-core.version>
<hadoop.version>2.7.4</hadoop.version>
<curator.version>2.7.1</curator.version>
<commons-io.version>2.4</commons-io.version>

@HyukjinKwonHyukjinKwonJul 6, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean the builds now pass even with -Phadoop-3.2 -Phive-1.2 but setting the Hadoop version to 2.7 implicitly?

@dongjoon-hyundongjoon-hyunJul 6, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya. This looks completely wrong.

HyukjinKwon
HyukjinKwon reviewed Jul 6, 2020
View reviewed changes
Comment threaddev/run-tests-jenkins.py Outdated
# Switch the Hive profile based on the PR title:
if "test-hive1.2" in ghprb_pull_title:
os.environ["AMPLAB_JENKINS_BUILD_HIVE_PROFILE"] = "hive1.2"
os.environ["AMPLAB_JENKINS_BUILD_PROFILE"] = "hadoop2.7"

@HyukjinKwonHyukjinKwonJul 6, 2020•
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's don't do this but rather fail with an explicit error message. It will be confusing what Hive 1.2 uses its Hadoop version as.

@dongjoon-hyundongjoon-hyunJul 6, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya. +1 for @HyukjinKwon 's advice.

@yaooqinnyaooqinnJul 6, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. thanks for the advice

@HyukjinKwon

HyukjinKwon commented Jul 6, 2020

Copy link
Copy Markdown
Member

I think you can give a shot to rather ban the profile combination with an explicit error message, for example, by using https://maven.apache.org/enforcer/enforcer-rules/bannedDependencies.html.

dongjoon-hyun
dongjoon-hyun requested changes Jul 6, 2020
View reviewed changes

@dongjoon-hyundongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @yaooqinn .
Sorry, but I'm -1 for this PR. hive-1.2 is not a super profile. It should not override other profile.

@dongjoon-hyun

dongjoon-hyun commented Jul 6, 2020

Copy link
Copy Markdown
Member

One more thing, @yaooqinn ~
Apache Spark community strongly discourages hive-1.2 profile which is pointing unofficial Apache Hive 1.2 fork. It's because it's simply a violation of global Apache project policy. Apache Spark community should not maintain another Apache project's fork.

@cloud-fan

cloud-fan commented Jul 6, 2020•
edited
Loading

Copy link
Copy Markdown
Contributor

@dongjoon-hyun shall we remove the [test-hive1.2] support for PR builders?

@HyukjinKwon

HyukjinKwon commented Jul 6, 2020

Copy link
Copy Markdown
Member

Nope, I don't think we should remove it now - it should be best to remove it when we completely remove test-hive1.2. Having [test-hive1.2] is only a dev purpose so shouldn't be a big deal.

@cloud-fan

cloud-fan commented Jul 6, 2020

Copy link
Copy Markdown
Contributor

If a PR wants to test hive 1.2, it must add both [test-hive1.2] and [test-hadoop2.7]?

@dongjoon-hyun

dongjoon-hyun commented Jul 6, 2020

Copy link
Copy Markdown
Member

Yes, @cloud-fan ~

@SparkQA

SparkQA commented Jul 6, 2020

Copy link
Copy Markdown

Test build #125016 has finished for PR 29006 at commit 5fed5a2.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yaooqinnyaooqinn changed the title [SPARK-32058][BUILD][SQL][test-hive1.2][FOLLOWUP] Set hadoop 2.7.4 for hive 1.2 profile[SPARK-32058][BUILD][SQL][test-hive1.2][test-hadoop2.7][FOLLOWUP] Set hadoop 2.7.4 for hive 1.2 profileJul 6, 2020
@yaooqinn

yaooqinn commented Jul 6, 2020

Copy link
Copy Markdown
MemberAuthor

retest this please

@SparkQA

SparkQA commented Jul 6, 2020

Copy link
Copy Markdown

Test build #125044 has started for PR 29006 at commit e1db207.

dongjoon-hyun
dongjoon-hyun reviewed Jul 6, 2020
View reviewed changes
Comment threadpom.xml
<rules>
<bannedDependencies>
<excludes>
<exclude>org.apache.hadoop:*:[3.2.0,)</exclude>

@dongjoon-hyundongjoon-hyunJul 6, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this work in sbt run? It seems that PR builder is banned by this.

[info] Building Spark using SBT with these arguments: -Phadoop-2.7 -Phive-1.2 -Pspark-ganglia-lgpl -Pyarn -Pkubernetes -Phadoop-cloud -Phive-thriftserver -Phive -Pkinesis-asl -Pmesos test:package streaming-kinesis-asl-assembly/assembly

@yaooqinnyaooqinnJul 7, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not work for sbt runner, and I don't know how to enable this for it. Any advice？

@HyukjinKwonHyukjinKwonJul 7, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just drop this PR then. We'll deprecate and remove Hive 1.2 profile in the near future anyway.

@yaooqinnyaooqinnJul 7, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I'll close this

@dongjoon-hyun

dongjoon-hyun commented Jul 6, 2020

Copy link
Copy Markdown
Member

BTW, @yaooqinn . Please use a new JIRA ID because banning a dependency is orthogonal to SPARK-32058.

@yaooqinn

yaooqinn commented Jul 7, 2020

Copy link
Copy Markdown
MemberAuthor

retest this please

@yaooqinnyaooqinn closed this Jul 7, 2020
@AmplabJenkins

AmplabJenkins commented Jul 7, 2020

Copy link
Copy Markdown

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125182/
Test FAILed.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HyukjinKwonHyukjinKwonHyukjinKwon left review comments
@dongjoon-hyundongjoon-hyundongjoon-hyun requested changes

Assignees

No one assigned

Labels

BUILDINFRA

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@yaooqinn@HyukjinKwon@dongjoon-hyun@cloud-fan@SparkQA@AmplabJenkins