HBASE-28836 Parallelize the file archival to improve the split times by mnpoonia · Pull Request #6243 · apache/hbase

mnpoonia · 2024-09-13T07:40:03Z

No description provided.

… object store (s3)

mnpoonia · 2024-09-13T07:40:24Z

Umeshkumar9414

LGTM

Umeshkumar9414 · 2024-09-13T08:10:23Z

 List<File> failures = new ArrayList<>();
 String startTime = Long.toString(start);
-for (File file : toArchive) {
+toArchive.parallelStream().forEach(file -> {


Good to check the difference between the parallelStream() and stream().parallel(). Java doc say that for parallelStream "It is allowable for this method to return a sequential stream."

Umeshkumar9414 · 2024-09-13T08:11:54Z

 List<File> failures = new ArrayList<>();
 String startTime = Long.toString(start);
-for (File file : toArchive) {
+toArchive.parallelStream().forEach(file -> {


Also if toArchive is small, there are chances that making it parallel can take more time than doing the same sequentially.

This wouldn't be the case as it is an IO operation (and diff wouldn't be noticable). So parallelizing it would improve it over what we already have. In pricinple i agree about what you are saying

I thought parallelStream (and its underlying ForkJoinPool) is not really for I/O blocking operations.

Yes, that's another point to consider

Apache-HBase · 2024-09-13T08:26:45Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 41s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 6s		master passed
+1 💚	compile	3m 1s		master passed
+1 💚	checkstyle	0m 38s		master passed
+1 💚	spotbugs	1m 34s		master passed
+1 💚	spotless	0m 45s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 53s		the patch passed
+1 💚	compile	3m 1s		the patch passed
+1 💚	javac	3m 1s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 37s		the patch passed
+1 💚	spotbugs	1m 42s		the patch passed
+1 💚	hadoopcheck	10m 37s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 43s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 12s		The patch does not generate ASF License warnings.
		36m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6243
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 0fe806a19d7d 5.4.0-186-generic #206-Ubuntu SMP Fri Apr 26 12:31:10 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `be6e570`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	84 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/1/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-09-13T11:43:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 26s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	3m 4s		master passed
+1 💚	compile	0m 54s		master passed
+1 💚	javadoc	0m 27s		master passed
+1 💚	shadedjars	5m 32s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 53s		the patch passed
+1 💚	compile	0m 54s		the patch passed
+1 💚	javac	0m 54s		the patch passed
+1 💚	javadoc	0m 25s		the patch passed
+1 💚	shadedjars	5m 31s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	208m 40s	/patch-unit-hbase-server.txt	hbase-server in the patch failed.
		232m 44s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6243
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux ff7d3094ff74 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `be6e570`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/1/testReport/
Max. process+thread count	5083 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/1/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

mnpoonia · 2024-09-13T16:01:07Z

Unit test failures seems to be unrelated.


ERROR] Failures: [ERROR] TestSecureIPC>AbstractTestSecureIPC.testRpcServerDisallowFallbackToSimpleAuth:279 Expected: (an instance of java.io.EOFException or an instance of org.apache.hadoop.hbase.exceptions.ConnectionClosedException)
but: was <java.net.SocketException: Connection reset>
[ERROR] Errors: [ERROR] TestSecureIPC.setUp:74->AbstractTestSecureIPC.initKDCAndConf:99 Â» Krb Failed to load or create keytab /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-6243/yetus-jdk17-hadoop3-check/src/hbase-server/target/test-data/2cbc9b35-01c7-7660-1fc6-e1d4722aafe2/keytab
[INFO] [ERROR] Tests run: 7004, Failures: 1, Errors: 1, Skipped: 35
[INFO]

virajjasani

While it is good to have parallel stream, it still relies on CPU cores to distribute the task for the underlying fork-join pool right? Why not create our own thread-pool to execute the tasks to achieve deterministic behavior independent of CPU cores?

d-c-manning · 2024-09-13T21:52:30Z

 List<File> failures = new ArrayList<>();
 String startTime = Long.toString(start);
-for (File file : toArchive) {
+toArchive.parallelStream().forEach(file -> {


I thought parallelStream (and its underlying ForkJoinPool) is not really for I/O blocking operations.

d-c-manning · 2024-09-13T21:53:09Z

 List<File> failures = new ArrayList<>();
 String startTime = Long.toString(start);
-for (File file : toArchive) {
+toArchive.parallelStream().forEach(file -> {


is the access to ArrayList failures thread safe?

d-c-manning

is HFileArchiver in the path for Split? It should be background.

virajjasani · 2024-09-14T23:52:27Z

Since both region close as well as CompactedHFilesDischarger chore performs archival, we can keep Jira/PR title generic i.e. "Parallelize the file archival for performance improvement" or "Parallelize the file archival" (by removing both "split times" and "object stores" from it because the change is generic and not specific to object stores as per the PR change, and Jira description is good to explain why we need this specifically for S3).

…times in object store (s3)" This reverts commit be6e570.

Apache9 · 2024-09-16T14:15:34Z

 List<File> failures = new ArrayList<>();
 String startTime = Long.toString(start);
-for (File file : toArchive) {
+toArchive.parallelStream().forEach(file -> {


There are IO operations in the code block so usually it is not safe to use parallelStream...
AFAIK parallelStream will make use of a common thread pool(ForkJoinPool.commonPool IIRC) which could be used by lots of other operations in java, so it is easy to introduce strange dead locks.
If we want to do it in parallel, we'd better introduce our own thread pool.

Apache-HBase · 2024-09-17T06:00:10Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	4m 2s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 8s		master passed
+1 💚	compile	3m 1s		master passed
+1 💚	checkstyle	0m 35s		master passed
+1 💚	spotbugs	1m 33s		master passed
+1 💚	spotless	0m 45s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 53s		the patch passed
+1 💚	compile	2m 56s		the patch passed
+1 💚	javac	2m 56s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 37s		the patch passed
+1 💚	spotbugs	1m 40s		the patch passed
+1 💚	hadoopcheck	10m 44s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
-1 ❌	spotless	0m 38s		patch has 44 errors when running spotless:check, run spotless:apply to fix.
			_ Other Tests _
+1 💚	asflicense	0m 12s		The patch does not generate ASF License warnings.
		39m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6243
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux a485f83be116 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b33878a`
Default Java	Eclipse Adoptium-17.0.11+9
spotless	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/2/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count	84 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/2/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-09-17T09:17:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	2m 31s		Docker mode activated.
-0 ⚠️	yetus	0m 2s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	2m 51s		master passed
+1 💚	compile	0m 54s		master passed
+1 💚	javadoc	0m 26s		master passed
+1 💚	shadedjars	5m 32s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 52s		the patch passed
+1 💚	compile	0m 55s		the patch passed
+1 💚	javac	0m 55s		the patch passed
+1 💚	javadoc	0m 26s		the patch passed
+1 💚	shadedjars	5m 29s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	210m 45s		hbase-server in the patch passed.
		236m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6243
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 78c03a61ff1a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b33878a`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/2/testReport/
Max. process+thread count	5088 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/2/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

NihalJain · 2024-09-17T09:31:54Z

nit: Parallize -> Parallelize

mnpoonia · 2024-09-17T10:14:08Z

I have added the logic for executor service doing the cleanup. I am still looking at executor service creation and config part. Basically trying to figure out the right config. Currently i am thinking of using hbase.regionserver.region.split.threads.max config for number of executor threads.
@d-c-manning @virajjasani @Apache9 FYI

Apache-HBase · 2024-09-17T11:11:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	2m 55s		master passed
+1 💚	compile	2m 58s		master passed
+1 💚	checkstyle	0m 37s		master passed
+1 💚	spotbugs	1m 32s		master passed
+1 💚	spotless	0m 44s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 55s		the patch passed
+1 💚	compile	3m 6s		the patch passed
+1 💚	javac	3m 6s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 37s		the patch passed
+1 💚	spotbugs	1m 39s		the patch passed
+1 💚	hadoopcheck	10m 23s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 44s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 10s		The patch does not generate ASF License warnings.
		35m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6243
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux ab8328bd1e96 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `1e21b43`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	84 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/3/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-09-17T14:29:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 27s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	3m 5s		master passed
+1 💚	compile	0m 54s		master passed
+1 💚	javadoc	0m 27s		master passed
+1 💚	shadedjars	5m 29s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 53s		the patch passed
+1 💚	compile	0m 56s		the patch passed
+1 💚	javac	0m 56s		the patch passed
+1 💚	javadoc	0m 26s		the patch passed
+1 💚	shadedjars	5m 30s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	209m 42s		hbase-server in the patch passed.
		233m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6243
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux e08784b4e144 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `1e21b43`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/3/testReport/
Max. process+thread count	5077 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6243/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

mnpoonia · 2024-11-20T10:00:20Z

Created a new PR because of messed up local git.
#6483

HBASE-28836 Parallize the file archival to improve the split times in…
be6e570
… object store (s3)

Umeshkumar9414 approved these changes Sep 13, 2024
View reviewed changes

Umeshkumar9414 reviewed Sep 13, 2024
View reviewed changes

apurtell approved these changes Sep 13, 2024
View reviewed changes

virajjasani reviewed Sep 13, 2024
View reviewed changes

d-c-manning reviewed Sep 13, 2024
View reviewed changes

mnpoonia changed the title ~~HBASE-28836 Parallize the file archival to improve the split times in…~~HBASE-28836 Parallize the file archival to improve the split times Sep 16, 2024

Revert "HBASE-28836 Parallize the file archival to improve the split …
2f405e9
…times in object store (s3)" This reverts commit be6e570.

Apache9 reviewed Sep 16, 2024
View reviewed changes

Use executor service for cleanup of files
b33878a

mnpoonia changed the title ~~HBASE-28836 Parallize the file archival to improve the split times~~ HBASE-28836 Parallelize the file archival to improve the split times Sep 17, 2024

Use synchronized collection
1e21b43

mnpoonia closed this Nov 20, 2024

Uh oh!

Conversation

mnpoonia commented Sep 13, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mnpoonia commented Sep 13, 2024

Uh oh!

Umeshkumar9414 left a comment

Choose a reason for hiding this comment

Uh oh!

Umeshkumar9414Sep 13, 2024

Choose a reason for hiding this comment

Uh oh!

Umeshkumar9414Sep 13, 2024

Choose a reason for hiding this comment

Uh oh!

mnpooniaSep 13, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d-c-manningSep 13, 2024

Choose a reason for hiding this comment

Uh oh!

virajjasaniSep 13, 2024

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Sep 13, 2024

Uh oh!

Apache-HBase commented Sep 13, 2024

Uh oh!

mnpoonia commented Sep 13, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

virajjasani left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d-c-manningSep 13, 2024

Choose a reason for hiding this comment

Uh oh!

d-c-manningSep 13, 2024

Choose a reason for hiding this comment

Uh oh!

d-c-manning left a comment

Choose a reason for hiding this comment

Uh oh!

virajjasani commented Sep 14, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Apache9Sep 16, 2024

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Sep 17, 2024

Uh oh!

Apache-HBase commented Sep 17, 2024

Uh oh!

NihalJain commented Sep 17, 2024

Uh oh!

mnpoonia commented Sep 17, 2024

Uh oh!

Apache-HBase commented Sep 17, 2024

Uh oh!

Apache-HBase commented Sep 17, 2024

Uh oh!

mnpoonia commented Nov 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

mnpoonia commented Sep 13, 2024•
edited
Loading

mnpooniaSep 13, 2024•
edited
Loading

mnpoonia commented Sep 13, 2024•
edited
Loading

virajjasani left a comment •
edited
Loading

virajjasani commented Sep 14, 2024•
edited
Loading