HBASE-28836 Parallize the file archival to improve the split times by mnpoonia · Pull Request #6483 · apache/hbase

mnpoonia · 2024-11-20T09:59:08Z

mnpoonia · 2024-11-20T10:01:05Z

@d-c-manning @virajjasani @Umeshkumar9414 @Apache9 @apurtell FYI

Umeshkumar9414

LGTM

d-c-manning · 2024-11-22T22:00:05Z

+ boolean fileCleaned = fileFutureEntry.getValue().get();
+ if (!fileCleaned) {
+ LOG.warn("Couldn't archive %s into backup directory: %s"
+ .formatted(fileFutureEntry.getKey(), baseArchiveDir));


I don't know all the jdk requirements, but looks like .formatted was added only in JDK 15. If this is intended for backport, there may be earlier JDK versions to support.

Thanks for pointing to this. Yes i was using jdk 17 so missed this.

… object store (s3)

Apache-HBase · 2024-11-25T06:14:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 29s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 18s		master passed
+1 💚	compile	3m 10s		master passed
+1 💚	checkstyle	0m 37s		master passed
+1 💚	spotbugs	1m 35s		master passed
+1 💚	spotless	0m 45s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 0s		the patch passed
+1 💚	compile	3m 14s		the patch passed
+1 💚	javac	3m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 36s		the patch passed
+1 💚	spotbugs	1m 38s		the patch passed
+1 💚	hadoopcheck	11m 37s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 43s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 11s		The patch does not generate ASF License warnings.
		38m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6483
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux c2fb7253f3e4 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `925ff4a`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	84 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-11-25T06:20:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
-0 ⚠️	yetus	0m 2s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	3m 26s		master passed
+1 💚	compile	0m 53s		master passed
+1 💚	javadoc	0m 27s		master passed
+1 💚	shadedjars	5m 36s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 54s		the patch passed
+1 💚	compile	0m 58s		the patch passed
+1 💚	javac	0m 58s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed
+1 💚	shadedjars	5m 50s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	21m 39s	/patch-unit-hbase-server.txt	hbase-server in the patch failed.
		44m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6483
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 045cbd2cb87c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `925ff4a`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/testReport/
Max. process+thread count	2201 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

virajjasani

It looks like until this change, only MasterFileSystem#checkTempDir and DeleteTableProcedure#deleteFromFs were using parallel archival using thread pool of size defined by hbase.hfilearchiver.thread.pool.max.
With this change, we will use parallel archival for all use cases.

virajjasani · 2024-12-03T06:12:22Z

@Apache9 @apurtell any reviews from your side before we merge it?
Keeping this open for at least 24 hr.

virajjasani

+1

virajjasani · 2024-12-08T13:37:07Z

@mnpoonia could you also create branch-2 backport PR so that after clean build results, the changes can be ported to all active branches?

mnpoonia · 2024-12-09T16:24:12Z

@virajjasani Sure. It can be applied to all active branches.

virajjasani · 2024-12-09T16:30:55Z

Sounds good, let me verify on branch-2 and then will merge it today.

…6483) Signed-off-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: David Manning <david.manning@salesforce.com> Signed-off-by: Umesh Kumar <9414umeshkumar@gmail.com>

mnpoonia · 2024-12-10T04:15:49Z

Thank you @virajjasani

virajjasani · 2024-12-10T18:47:48Z

Thank you @mnpoonia for this nice improvement!

…pache#6483) Signed-off-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: David Manning <david.manning@salesforce.com> Signed-off-by: Umesh Kumar <9414umeshkumar@gmail.com>

…times (#6483)" This reverts commit 262c5bb.

…times (apache#6483)" This reverts commit 262c5bb.

…pache#6483) Signed-off-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: David Manning <david.manning@salesforce.com> Signed-off-by: Umesh Kumar <9414umeshkumar@gmail.com>

…times (apache#6483)" This reverts commit 262c5bb.

mnpoonia mentioned this pull request Nov 20, 2024
HBASE-28836 Parallelize the file archival to improve the split times #6243
Closed

This comment has been minimized.
Sign in to view

Umeshkumar9414 reviewed Nov 20, 2024
View reviewed changes

Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java

mnpoonia force-pushed the HBASE-28836 branch from d3703bb to f86aa9bCompare November 21, 2024 05:11

This comment has been minimized.
Sign in to view

Umeshkumar9414 approved these changes Nov 21, 2024
View reviewed changes

virajjasani self-requested a review November 21, 2024 17:07

d-c-manning reviewed Nov 22, 2024
View reviewed changes

HBASE-28836 Parallize the file archival to improve the split times in…
925ff4a
… object store (s3)

mnpoonia force-pushed the HBASE-28836 branch from f86aa9b to 925ff4aCompare November 25, 2024 05:31

d-c-manning approved these changes Nov 25, 2024
View reviewed changes

virajjasani reviewed Dec 3, 2024
View reviewed changes

virajjasani requested review from Apache9 and apurtell December 3, 2024 06:11

virajjasani approved these changes Dec 3, 2024
View reviewed changes

virajjasani merged commit 262c5bb into apache:masterDec 9, 2024

virajjasani added a commit that referenced this pull request Jan 6, 2025
Revert "HBASE-28836Parallize the file archival to improve the split …
97affac
…times (#6483)" This reverts commit 262c5bb.

virajjasani added a commit that referenced this pull request Jan 6, 2025
Revert "HBASE-28836Parallize the file archival to improve the split …
a8b5eb4
…times (#6483)" This reverts commit 262c5bb.

virajjasani added a commit that referenced this pull request Jan 6, 2025
Revert "HBASE-28836Parallize the file archival to improve the split …
4985106
…times (#6483)" This reverts commit 262c5bb.

virajjasani added a commit that referenced this pull request Jan 6, 2025
Revert "HBASE-28836Parallize the file archival to improve the split …
9a2c74a
…times (#6483)" This reverts commit 262c5bb.

virajjasani added a commit that referenced this pull request Jan 6, 2025
Revert "HBASE-28836Parallize the file archival to improve the split …
ef369dc
…times (#6483)" This reverts commit 262c5bb.

ragarkar pushed a commit to ragarkar/hbase that referenced this pull request Jan 13, 2025
Revert "HBASE-28836 Parallize the file archival to improve the split …
f69447b
…times (apache#6483)" This reverts commit 262c5bb.

mokai87 pushed a commit to mokai87/hbase that referenced this pull request Aug 7, 2025
Revert "HBASE-28836 Parallize the file archival to improve the split …
1a0a38d
…times (apache#6483)" This reverts commit 262c5bb.

Uh oh!

Conversation

mnpoonia commented Nov 20, 2024• edited by virajjasani Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mnpoonia commented Nov 20, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

This comment has been minimized.

This comment has been minimized.

Umeshkumar9414 left a comment

Choose a reason for hiding this comment

Uh oh!

d-c-manningNov 22, 2024

Choose a reason for hiding this comment

Uh oh!

mnpooniaNov 25, 2024

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Nov 25, 2024

Uh oh!

Apache-HBase commented Nov 25, 2024

Uh oh!

virajjasani left a comment

Choose a reason for hiding this comment

Uh oh!

virajjasani commented Dec 3, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

virajjasani left a comment

Choose a reason for hiding this comment

Uh oh!

virajjasani commented Dec 8, 2024

Uh oh!

mnpoonia commented Dec 9, 2024

Uh oh!

virajjasani commented Dec 9, 2024

Uh oh!

mnpoonia commented Dec 10, 2024

Uh oh!

virajjasani commented Dec 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mnpoonia commented Nov 20, 2024•
edited by virajjasani
Loading

virajjasani commented Dec 3, 2024•
edited
Loading