HBASE-29838 Run Hadoop Check as a GitHub Action

[ndimiduk]

No description provided.

[ndimiduk]

Looks like we occupied the runner for 6h and then it was aborted.

[Apache9]

https://infra.apache.org/github-actions-policy.html

The policy here does not say about the 6 hours timeout...

We can ask infra about the rules and the size of the github runners, our jenkins runners finished in "343m 23s", which was very close to 6 hours, so if the machine of the github runner is weaker, the build will be very easy to cost more than 6 hours...

[ndimiduk]

My action itself has timeout-minutes: 600 . i'll try bumping this up.

[ndimiduk]

or not. 6h is GH's hard limit, https://docs.github.com/en/actions/reference/limits

[Apache9]

Then maybe we should try self hosted github runners? For self hosted runners the execution time limit is 5 days...

[ndimiduk]

Then maybe we should try self hosted github runners? For self hosted runners the execution time limit is 5 days...

Yes we should bring this back to our CI discussions with Infra. Maybe we can borrow from the pool of new Jenkins workers while we continue to build this out.

Yetus is supposed to provide smart, selective detection of module changes when it decided which tests to run. I think the new .github directory broke that for this run, so I've pushed a change to exclude it, maybe that will help.

I'm also going to see if I can manually parallelize the unit test runs -- maybe break out three separate checks for the three main unit test groups or something like that.

[Apache9]

Then maybe we should try self hosted github runners? For self hosted runners the execution time limit is 5 days...

Yes we should bring this back to our CI discussions with Infra. Maybe we can borrow from the pool of new Jenkins workers while we continue to build this out.

Yetus is supposed to provide smart, selective detection of module changes when it decided which tests to run. I think the new .github directory broke that for this run, so I've pushed a change to exclude it, maybe that will help.

I'm also going to see if I can manually parallelize the unit test runs -- maybe break out three separate checks for the three main unit test groups or something like that.

Maybe we could split the UTs run as a seperated github check? Or even more, we could split the UTs run as two seperated check, one has -PrunDevTests(for small/medium tests) and one has -PrunLargeTests(for large tests).

[ndimiduk]

Maybe we could split the UTs run as a seperated github check? Or even more, we could split the UTs run as two seperated check, one has -PrunDevTests(for small/medium tests) and one has -PrunLargeTests(for large tests).

Yep, that's exactly my thinking as well. Landing these other cleanup issues and I'll be back.

[ndimiduk]

Okay this is better. Module selection chose only hbase-examples for running the unit tests.

-1 overall
| Vote | Subsystem | Runtime | Comment
============================================================================
| 0 | reexec | 1m 13s | Docker mode activated. | -0 | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file | | | | --spotbugs-strict-precheck
| | | | --author-ignore-list
| | | | --blanks-eol-ignore-file
| | | | --blanks-tabs-ignore-file
| | | | --quick-hadoopcheck
+---------------------------------------------------------------------------
| | | | Prechecks +---------------------------------------------------------------------------
+---------------------------------------------------------------------------
| | | | master Compile Tests +---------------------------------------------------------------------------
| +1 | mvninstall | 3m 25s | master passed | +1 | compile | 0m 19s | master passed | +1 | javadoc | 0m 17s | master passed | +1 | shadedjars | 4m 47s | branch has no errors when building our | | | | shaded downstream artifacts.
+---------------------------------------------------------------------------
| | | | Patch Compile Tests +---------------------------------------------------------------------------
| +1 | mvninstall | 3m 7s | the patch passed | +1 | compile | 0m 18s | the patch passed | +1 | javac | 0m 18s | the patch passed | +1 | javadoc | 0m 15s | the patch passed | +1 | shadedjars | 4m 51s | patch has no errors when building our | | | | shaded downstream artifacts.
+---------------------------------------------------------------------------
| | | | Other Tests +---------------------------------------------------------------------------
| -1 | unit | 0m 24s | hbase-examples in the patch failed. | | | 19m 12s |
|| Subsystem || Report/Notes ||
============================================================================
| unit | /home/runner/work/hbase/hbase/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-examples.txt |
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: /home/runner/work/hbase/hbase/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7651 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 9b6b1363e88a 6.11.0-1018-azure #18~24.04.1-Ubuntu SMP Sat Jun 28 04:46:03 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / adfd0560c941109283928946bd671a5836604e46 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 72 (vs. ulimit of 30000) |
| modules | C: hbase-examples U: hbase-examples |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org/ |

[ndimiduk]

Looking over out last successful nightly on master, the Large tests on hbase-server still took 7h. I'm looking for other ways to partition this up.

[ndimiduk]

I spot-checked the tests run by large waves 1 and 3 (2 is still running) and they look to contain only the expected tests. I think we're good here.

Any other comments @Apache9 ?

[Apache-HBase]

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/20/console in case of problems.

[Apache-HBase]

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/20/console in case of problems.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+0 🆗	yamllint	0m 0s		yamllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ master Compile Tests _
+1 💚	mvninstall	3m 2s		master passed
+1 💚	compile	8m 29s		master passed
+1 💚	spotless	0m 43s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 57s		the patch passed
+1 💚	compile	8m 25s		the patch passed
+1 💚	javac	8m 25s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	pylint	0m 4s	/results-pylint.txt	The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
+1 💚	shellcheck	0m 1s		No new issues.
+1 💚	xmllint	0m 0s		No new issues.
+1 💚	hadoopcheck	11m 12s		Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 💚	spotless	0m 44s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 11s		The patch does not generate ASF License warnings.
		43m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/20/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7651
Optional Tests	dupname asflicense codespell detsecrets shellcheck shelldocs spotless pylint yamllint javac xmllint hadoopcheck compile
uname	Linux 50ae92ed5baf 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / add8096
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	192 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/20/console
versions	git=2.34.1 maven=3.9.8 pylint=2.15.5 shellcheck=0.8.0 xmllint=20913
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 13s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	2m 15s		master passed
+1 💚	compile	1m 39s		master passed
+1 💚	javadoc	1m 28s		master passed
+1 💚	shadedjars	4m 22s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 16s		the patch passed
+1 💚	compile	1m 39s		the patch passed
+1 💚	javac	1m 39s		the patch passed
+1 💚	javadoc	1m 28s		the patch passed
+1 💚	shadedjars	4m 22s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	267m 50s		root in the patch passed.
		293m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/20/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7651
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 86f7204404e4 6.14.0-1018-aws #18~24.04.1-Ubuntu SMP Mon Nov 24 19:46:27 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / add8096
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/20/testReport/
Max. process+thread count	7781 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/20/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[Apache9]

I think the current one is enough to be merged so we can disable the jenkins pre commit job.

There are still problems like mvninstall is still present in unit check, and the job summary for unit check still needs improvement, etc. Can file new issues for improving.

Thanks @ndimiduk for the great work!