HIVE-28191: Upgrade Hadoop and Hbase Client version

[linghengqian]

What changes were proposed in this pull request?

• Upgrade Hadoop and Hbase Client version.

Why are the changes needed?

• Since HIVE-28191. Upgrade Hadoop Version to 3.4.0. #5187 is no longer being updated, I need to open the current PR.

Does this PR introduce any user-facing change?

• Null.

Is the change a dependency upgrade?

• Yes.
• dependency-tree.txt

How was this patch tested?

• Jenkins

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[linghengqian]

• I have opened https://issues.apache.org/jira/browse/HIVE-28552 to discuss the unit test bug for the master branch.

[zhangbutao]

https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-5481/3/tests Hbase storage handler related tests failed again:

java.io.IOException: Shutting down
at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:257)
at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:110)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1185)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1232)
at org.apache.hadoop.hive.hbase.HBaseTestSetup.setUpFixtures(HBaseTestSetup.java:112)
at org.apache.hadoop.hive.hbase.HBaseTestSetup.preTest(HBaseTestSetup.java:78)
at org.apache.hadoop.hive.ql.QTestMiniClusters.setup(QTestMiniClusters.java:240)
at org.apache.hadoop.hive.ql.QTestUtil.<init>(QTestUtil.java:222)
at org.apache.hadoop.hive.hbase.HBaseQTestUtil.<init>(HBaseQTestUtil.java:44)
at org.apache.hadoop.hive.cli.control.CoreHBaseNegativeCliDriver.beforeClass(CoreHBaseNegativeCliDriver.java:49)
at org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:80)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
Caused by: java.lang.RuntimeException: Master not initialized after 200000ms
at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:221)
at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:190)
at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:407)
at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:250)
... 21 more

But the info is not very clear. I think you can refer to this ticket https://issues.apache.org/jira/browse/HBASE-28061 to explore the real failed exception details. You can write a simple java test to test the MiniHbaseCluster on Hadoop 3.4.0

[linghengqian]

But the info is not very clear. I think you can refer to this ticket https://issues.apache.org/jira/browse/HBASE-28061 to explore the real failed exception details. You can write a simple java test to test the MiniHbaseCluster on Hadoop 3.4.0

@zhangbutao

• I've opened https://issues.apache.org/jira/browse/HIVE-28559 and I can't even start any relevant unit tests locally, which prevents me from even debugging. Unit tests that failed to execute include org.apache.hadoop.hive.metastore.TestRemoteHiveMetastoreWithHttpJwt, org.apache.hive.service.auth.jwt.TestHttpJwtAuthentication and org.apache.hadoop.hive.cli.TestHBaseCliDriver.
• I'm not sure what happened to the master branch, it doesn't seem like something documented at https://cwiki.apache.org/confluence/display/Hive/HowToContribute . Either Hive's master branch can't be compiled with JDK8, or Hive's master branch can't be compiled with Maven 3.9.9. And Hive hasn't used Maven Wrapper to unify the environment for contributors yet.

[zhangbutao]

1. Failed Hbase Handler tests are related to HBASE-28908

2. Failed tests TestRemoteHiveMetastoreWithHttpJwt#testValidJWT and TestHttpJwtAuthentication#testAuthorizedUser is related Hadoop3.4.0 change HADOOP-17009: Embrace Immutability of Java Collections hadoop#1974
   https://github.com/apache/hadoop/pull/1974/files#diff-372a0d25bcccd88b409a8149949628abd7d3472a1798bebe813e7617b0ef73c7L918-L920

Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.ProcessEnvironment$Variable
at java.lang.ProcessEnvironment$StringEnvironment.toEnvironmentBlock(ProcessEnvironment.java:273) ~[?:1.8.0_221]
at java.lang.ProcessEnvironment.toEnvironmentBlock(ProcessEnvironment.java:298) ~[?:1.8.0_221]
at java.lang.ProcessImpl.start(ProcessImpl.java:86) ~[?:1.8.0_221]
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_221]
at org.apache.hadoop.util.Shell.runCommand(Shell.java:998) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.util.Shell.run(Shell.java:959) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1282) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1377) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1359) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1535) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNonNativeIO(RawLocalFileSystem.java:1000) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:991) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:952) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.hive.ql.exec.Utilities.ensurePathIsWritable(Utilities.java:4954) ~[classes/:?]
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:843) ~[classes/:?]
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:784) ~[classes/:?]
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:708) ~[classes/:?]
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:669) ~[classes/:?]
at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:182) ~[classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_221]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_221]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_221]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_221]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[classes/:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[classes/:?]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[classes/:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_221]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_221]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[classes/:?]
at com.sun.proxy.$Proxy58.open(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:485) ~[classes/:?]
... 40 more

[slfan1989]

@linghengqian@zhangbutao Thank you for your attention to this JIRA! I will close my PR(#5187) and let’s work together to resolve the issue in this PR.

[linghengqian]

@linghengqian@zhangbutao Thank you for your attention to this JIRA! I will close my PR(#5187) and let’s work together to resolve the issue in this PR.

• Sorry, my outlook mailbox mistakenly classified the emails from the ASF JIRA issue into spam. I just saw the launch method for executing integration tests, I am still investigating on my local machine.

[zhangbutao]

Superseded by #5500

[linghengqian]

Superseded by #5500

No problem, I'll close the current PR.