HIVE-23980: Shade Guava from hive-exec in Hive branch-2

[viirya]

What changes were proposed in this pull request?

This PR proposes to shade Guava from hive-exec in Hive branch-2. This is basically for triggering test for #1356.

Why are the changes needed?

When trying to upgrade Guava in Spark, found the following error. A Guava method became package-private since Guava version 20. So there is incompatibility with Guava versions > 19.0.

sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IllegalAccessError: tried to access method com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator; from class org.apache.hadoop.hive.ql.exec.FetchOperator
at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:108)
at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)

This is a problem for downstream clients. Hive project noticed that problem too in HIVE-22126, however that only targets 4.0.0. It'd be nicer if we can also shade Guava from current Hive versions, e.g. Hive 2.3 line.

Does this PR introduce any user-facing change?

Yes. Guava will be shaded from hive-exec.

How was this patch tested?

Built Hive locally and checked jar content.

[viirya]

cc @sunchao

[sunchao]

@viirya seems there is no new testing failing which is good.

[viirya]

@sunchao But I saw some test failures?

There are 0 new tests failing, 636 existing failing and 231 skipped.

What existing failing means?

[sunchao]

@viirya it probably means those tests were already failing previously. However I was not able to find a history for this. According to this comment, it seems many tests were failing after we enabled CI for branch-2.

cc @belugabehr - wonder if you have any info on this. Thanks.

[viirya]

Any idea about which tests I should take a look at?

[sunchao]

@viirya I see most of these test failures are related to Calcite so seems not related, and they were failing previously. Maybe we can trigger the tests again by force push. Also it'd be great if we can verify locally by selecting a few test before and after the PR.

[viirya]

I tried to run failed tests locally.

mvn clean -Dtest=org.apache.hive.hcatalog.pig.TestParquetHCatStorer test

------------------------------------------------------- T E S T S ------------------------------------------------------- OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0 Running org.apache.hive.hcatalog.pig.TestParquetHCatStorer Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 106.145 sec - in org.apache.hive.hcatalog.pig.TestParquetHCatStorer Results :
Tests run: 27, Failures: 0, Errors: 0, Skipped: 0

mvn clean -Dtest=org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned test

-------------------------------------------------------
T E S T S
-------------------------------------------------------
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
Running org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned
Tests run: 14, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 143.976 sec - in org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned
Results :
Tests run: 14, Failures: 0, Errors: 0, Skipped: 4

But I cannot reproduce the test failure.

[viirya]

@sunchao Can you reproduce the test failure locally?

[sunchao]

@sunchao Can you reproduce the test failure locally?

NO I took two failed tests and can't reproduce them locally, with and without your PR.

[viirya]

@sunchao Can you reproduce the test failure locally?

NO I took two failed tests and can't reproduce them locally, with and without your PR.

Hm, it makes harder to debug...

[sunchao]

Yeah, however I think this PR is trivial enough and I don't see the tests are related.

[sunchao]

LGTM