HIVE-24484: Upgrade Hadoop to 3.3.1 And Tez to 0.10.2

[ayushtkn]

Upgrade to Hadoop-3.3.1 & Tez to 0.10.2

[ayushtkn]

My Last run here had 4 errors, of which I think I have fixed 3 more. The one remaining is some XML parsing error, which I think might get auto resolved or may be an after affect.

@kgyrtkirk I have sorted the JLine issue here as well, which you told in the previous PR. Can you give a check once

[ayushtkn]

Thanx @abstractdog We have a green build here, so I have updated the PR title, while merging they will get squashed automatically :-)

[abstractdog]

Thanx @abstractdog We have a green build here, so I have updated the PR title, while merging they will get squashed automatically :-)

makes sense, thanks! I wish we could merge this now, hopefully we can release tez in 2 weeks

I can see the PTF boolean patch in the commits, it's not intentional I guess

[suryanshagnihotri]

@ayushtkn Did you not face this compilation error. I do not see any change in FairSchedulerShim.java. I compiled hive 3.1.2 with hadoop 3.3.1.
Compilation failure [ERROR] /Users/suryansh/Documents/BDS/apache_hive/shims/scheduler/src/main/java/org/apache/hadoop/hive/schshim/FairSchedulerShim.java:[31,68] org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy is not public in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair; cannot be accessed from outside package

[ayushtkn]

@suryanshagnihotri If this was the case, the build would have failed, which it didn't. So, we are cool here. Nothing pending here apart from awaiting an official Tez release here..

[suryanshagnihotri]

@ayushtkn your base branch is does not seems to be based on 3.1.x release branch. It does not have FairSchedulerShim.javahttps://github.com/ayushtkn/hive/tree/HIVE-24484/shims/scheduler.
It fails if base branch is cut from 3.1.x...

Check https://github.com/apache/hive/blob/release-3.1.3-rc3/shims/scheduler/src/main/java/org/apache/hadoop/hive/schshim/FairSchedulerShim.java#L31 makes reference to QueuePlacementPolicy which is package protected final class https://github.com/apache/hadoop/blame/release-3.3.3-RC1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java#L54

Is there any plan to add support in 3.1.x version of hive?

[ayushtkn]

@suryanshagnihotri Nopes, not chasing that branch, this is only for hive-master

[sujith71955]

@ayushtkn your base branch is does not seems to be based on 3.1.x release branch. It does not have FairSchedulerShim.javahttps://github.com/ayushtkn/hive/tree/HIVE-24484/shims/scheduler. It fails if base branch is cut from 3.1.x...

Check https://github.com/apache/hive/blob/release-3.1.3-rc3/shims/scheduler/src/main/java/org/apache/hadoop/hive/schshim/FairSchedulerShim.java#L31 makes reference to QueuePlacementPolicy which is package protected final class https://github.com/apache/hadoop/blame/release-3.3.3-RC1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java#L54

Is there any plan to add support in 3.1.x version of hive?

But Hive 3.1.x version is not very old and 4.x still looks like in alpha so we may not able to upgrade. so with this PR still we have compatibility issues with Hive 3.x version. any suggestions? thanks
cc @sankarh

[ayushtkn]

But Hive 3.1.x version is not very old and 4.x still looks like in alpha so we may not able to upgrade. so with this PR still we have compatibility issues with Hive 3.x version. any suggestions? thanks

@sujith71955 Unfortunately, I don't have a use case for 3.x line, but that should be doable but would requires changes across Hadoop, Hive & Tez. We did the same here as well....

It is certainly doable, if you folks have a use case, feel free to create a Jira for 3.1.x line. Running busy so couldn't spare time to check the problem you folks stated above..

[ayushtkn]

Got a new Test failure due to jetty. TestSSL, Fixed in the latest commit.

[abstractdog]

minor note, otherwise LGTM +1

[abstractdog]

please include hadoop upgrade as the cause in this comment, otherwise looks good to me

[abstractdog]

LGTM +1

[ayushtkn]

Merged. Thanx @abstractdog, @kgyrtkirk and @steveloughran for helping with reviews. :-)

[abstractdog]

also, let me grab the opportunity to thank @belugabehr who put enormous efforts into the hadoop upgrade in the early days!