TEZ-4420 : Upgrade Hadoop to 3.3.4 by amanraj2520 · Pull Request #272 · apache/tez

amanraj2520 · 2023-02-21T06:47:59Z

Upgraded Hadoop to 3.3.4 and some dependency imports for tests.
Have successfully tested in my local running all the test cases.

…e main idea is to give priority to Mockito than Powermock.

amanraj2520 · 2023-02-21T06:48:52Z

@abstractdog I have tested all tests in my local. The only test failure was TestLocalMode previously which I see you have fixed recently in master. Can you please review this.

…tFixes

amanraj2520 · 2023-02-22T09:37:38Z

Just FYI why I made this change in tez-tests
This issue comes when PowerMock library conflicts with Mockito as described here https://stackoverflow.com/questions/71973762/java-lang-nosuchmethoderror-org-mockito-answers-getlorg-mockito-stubbing-answ

amanraj2520 · 2023-02-22T09:48:06Z

@abstractdog Can you please review this now

…ez-tests

amanraj2520 · 2023-02-23T09:30:54Z

@abstractdog Can you please review this.

abstractdog · 2023-02-23T09:42:04Z

Just FYI why I made this change in tez-tests This issue comes when PowerMock library conflicts with Mockito as described here https://stackoverflow.com/questions/71973762/java-lang-nosuchmethoderror-org-mockito-answers-getlorg-mockito-stubbing-answ

I cannot see powermock used by tez, so assuming that the powermock used by hadoop conflicts with the mockito used by tez? is that right? if so, theoretically, this conflict can be resolved in the future in case of a mockito version change in tez or rather a powermock version change in hadoop (as mockito is quite new one in tez since TEZ-4383)
please clarify this scenario here by an xml comment: pom.xmls are full of dependencies without any context and hence extremely easy to break, I would like to see the explanation here (any comment that clarifies my points above is welcome :) )

amanraj2520 · 2023-02-23T11:13:03Z

Hi @abstractdog Sorry my bad. I checked the Hadoop 3.3.4 version in my local repo. It had the use of powermock libraries which I was using to debug and mock some functionalities for branch-3 (Hive) upgrade with Hadoop 3.3.4. These were unnecessary changes. I refreshed my local and deleted all the jars. Now it downloaded oss Hadoop 3.3.4 and built correctly. We only need to include mockito-core for the tez-tests that's it. I have pushed my commit. Kindly check it once.

amanraj2520 · 2023-02-23T11:14:09Z

@abstractdog Added a snapshot of a sample successful test which was failing earlier :

amanraj2520 · 2023-02-23T11:19:22Z

Just for everyone's knowledge, if we do not include mockito-core, the tests will fail with the following errors:
`[INFO] Running org.apache.tez.mapreduce.TestMRRJobsDAGApi
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.92 s <<< FAILURE! - in org.apache.tez.mapreduce.TestMRRJobsDAGApi
[ERROR] org.apache.tez.mapreduce.TestMRRJobsDAGApi Time elapsed: 2.914 s <<< ERROR!
java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
at org.apache.tez.mapreduce.TestMRRJobsDAGApi.setup(TestMRRJobsDAGApi.java:156)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
at org.apache.tez.mapreduce.TestMRRJobsDAGApi.setup(TestMRRJobsDAGApi.java:156)

[INFO] Running org.apache.tez.mapreduce.TestMRRJobs
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.416 s <<< FAILURE! - in org.apache.tez.mapreduce.TestMRRJobs
[ERROR] org.apache.tez.mapreduce.TestMRRJobs Time elapsed: 3.412 s <<< ERROR!
java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
at org.apache.tez.mapreduce.TestMRRJobs.setup(TestMRRJobs.java:71)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
at org.apache.tez.mapreduce.TestMRRJobs.setup(TestMRRJobs.java:71)

[INFO] Running org.apache.tez.test.TestAM
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.019 s <<< FAILURE! - in org.apache.tez.test.TestAM
[ERROR] org.apache.tez.test.TestAM Time elapsed: 3.015 s <<< ERROR!
java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
at org.apache.tez.test.TestAM.setup(TestAM.java:66)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
at org.apache.tez.test.TestAM.setup(TestAM.java:66)

[INFO] Running org.apache.tez.test.TestAMRecovery
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.463 s <<< FAILURE! - in org.apache.tez.test.TestAMRecovery
[ERROR] org.apache.tez.test.TestAMRecovery Time elapsed: 3.46 s <<< ERROR!
java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
at org.apache.tez.test.TestAMRecovery.beforeClass(TestAMRecovery.java:109)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
at org.apache.tez.test.TestAMRecovery.beforeClass(TestAMRecovery.java:109)`

tez-yetus · 2023-02-23T13:40:35Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	6m 36s	Maven dependency ordering for branch
+1 💚	mvninstall	10m 9s	master passed
+1 💚	compile	2m 47s	master passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04
+1 💚	compile	2m 32s	master passed with JDK Private Build-1.8.0_352-8u352-ga-1~22.04-b08
+1 💚	javadoc	2m 35s	master passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04
+1 💚	javadoc	2m 6s	master passed with JDK Private Build-1.8.0_352-8u352-ga-1~22.04-b08
		_ Patch Compile Tests _
+0 🆗	mvndep	1m 15s	Maven dependency ordering for patch
+1 💚	mvninstall	4m 31s	the patch passed
+1 💚	compile	2m 47s	the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04
+1 💚	javac	2m 47s	the patch passed
+1 💚	compile	2m 31s	the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~22.04-b08
+1 💚	javac	2m 31s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	javadoc	2m 24s	the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04
+1 💚	javadoc	2m 2s	the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~22.04-b08
		_ Other Tests _
+1 💚	unit	39m 47s	tez-tests in the patch passed.
+1 💚	unit	67m 27s	root in the patch passed.
+1 💚	asflicense	0m 50s	The patch does not generate ASF License warnings.
		152m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-272/6/artifact/out/Dockerfile
GITHUB PR	#272
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux 5d791a3b716f 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `be99489`
Default Java	Private Build-1.8.0_352-8u352-ga-1~22.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~22.04-b08
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-272/6/testReport/
Max. process+thread count	2090 (vs. ulimit of 5500)
modules	C: tez-tests . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-272/6/console
versions	git=2.34.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

abstractdog

looks much cleaner, thanks @amanraj2520
+1

steveloughran · 2023-02-24T15:19:28Z

nice. well done

abstractdogand others added 2 commits November 26, 2022 08:30

TEZ-4420: Upgrade to Hadoop 3.3.4
e83c806

Added dependency version and order in tez-tests pom.xml. Note that th…
944476e
…e main idea is to give priority to Mockito than Powermock.

This comment was marked as outdated.
Sign in to view

amanraj2520 added 3 commits February 22, 2023 02:55

Merge remote-tracking branch 'oss/master' into TEZ-4420TestFixes
56408a2

Merge remote-tracking branch 'oss/TEZ-4420TestFixes' into TEZ-4420Tes…
29979aa
…tFixes

Removed mockito-core version to use project's version
f7d1c84

abstractdog reviewed Feb 22, 2023
View reviewed changes

Comment threadtez-tests/pom.xml Outdated

amanraj2520 force-pushed the TEZ-4420TestFixes branch from e82492c to f7d1c84Compare February 22, 2023 09:47

steveloughran reviewed Feb 22, 2023
View reviewed changes

Comment threadtez-tests/pom.xml Outdated

This comment was marked as outdated.
Sign in to view

Declared dependency in main pom.xml and just imported dependency in t…
449a85c
…ez-tests

This comment was marked as outdated.
Sign in to view

Removed non-used dependencies
98f8455

abstractdog self-requested a review February 24, 2023 07:37

abstractdog approved these changes Feb 24, 2023
View reviewed changes

abstractdog merged commit db9eb1e into apache:masterFeb 24, 2023

abstractdog mentioned this pull request Feb 24, 2023
TEZ-4420: Upgrade to Hadoop 3.3.4 #213
Closed

Uh oh!

Conversation

amanraj2520 commented Feb 21, 2023

Uh oh!

amanraj2520 commented Feb 21, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

amanraj2520 commented Feb 22, 2023

Uh oh!

amanraj2520 commented Feb 22, 2023

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

amanraj2520 commented Feb 23, 2023

Uh oh!

abstractdog commented Feb 23, 2023

Uh oh!

amanraj2520 commented Feb 23, 2023

Uh oh!

amanraj2520 commented Feb 23, 2023

Uh oh!

amanraj2520 commented Feb 23, 2023

Uh oh!

tez-yetus commented Feb 23, 2023

Uh oh!

abstractdog left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Feb 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

amanraj2520 commented Feb 21, 2023•
edited
Loading