HBASE-26271: Cleanup the broken store files under data directory by BukrosSzabolcs · Pull Request #3786 · apache/hbase

BukrosSzabolcs · 2021-10-22T13:54:24Z

Add new chore to delete lefotver files in case file based storefile
handling is used
Expose the target files of currently running compactions for easier
validation

BukrosSzabolcs · 2021-10-22T13:56:50Z

@Apache9, @wchevreuil This is the commit only containing changes related to the CleanerChore and exposing what is absolutely necessary for it's functionality.

Apache-HBase · 2021-10-22T14:44:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	4m 8s	HBASE-26067 passed
+1 💚	compile	3m 10s	HBASE-26067 passed
+1 💚	checkstyle	1m 6s	HBASE-26067 passed
+1 💚	spotbugs	2m 8s	HBASE-26067 passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 47s	the patch passed
+1 💚	compile	3m 11s	the patch passed
+1 💚	javac	3m 11s	the patch passed
-0 ⚠️	checkstyle	1m 5s	hbase-server: The patch generated 22 new + 59 unchanged - 0 fixed = 81 total (was 59)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	19m 21s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 17s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 16s	The patch does not generate ASF License warnings.
		49m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 6ffb598edba4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `29fb33b`
Default Java	AdoptOpenJDK-1.8.0_282-b08
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	95 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-10-22T18:03:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 2s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	5m 23s	HBASE-26067 passed
+1 💚	compile	1m 19s	HBASE-26067 passed
+1 💚	shadedjars	9m 14s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 44s	HBASE-26067 passed
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 6s	the patch passed
+1 💚	compile	1m 20s	the patch passed
+1 💚	javac	1m 20s	the patch passed
+1 💚	shadedjars	9m 6s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 42s	the patch passed
		_ Other Tests _
+1 💚	unit	212m 9s	hbase-server in the patch passed.
		248m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 0338de158941 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `29fb33b`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/1/testReport/
Max. process+thread count	3383 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-10-22T18:37:41Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 25s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	4m 55s	HBASE-26067 passed
+1 💚	compile	1m 11s	HBASE-26067 passed
+1 💚	shadedjars	9m 38s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 47s	HBASE-26067 passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 37s	the patch passed
+1 💚	compile	1m 10s	the patch passed
+1 💚	javac	1m 10s	the patch passed
+1 💚	shadedjars	10m 8s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 47s	the patch passed
		_ Other Tests _
-1 ❌	unit	244m 52s	hbase-server in the patch failed.
		281m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux ef0d0f909d6f 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `29fb33b`
Default Java	AdoptOpenJDK-1.8.0_282-b08
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/1/testReport/
Max. process+thread count	3159 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9

In general I like the approach, it is almost the same with what I expected when I review your first PR.

The problems here are all implementation details, we need to make sure that there are no races so we do not miss some corner cases.

Apache9 · 2021-10-24T13:36:21Z

+ * This Chore, every time it runs, will clear the unsused HFiles in the data
+ * folder.
+ */
+@InterfaceAudience.Private public class FileBasedStoreFileCleaner extends ScheduledChore {


Let's not name it 'FileBased'. I think it should be more general, maybe 'BrokenStoreFileCleaner'.
For now, the only condition is whether we will write to the data directory directly. No matter what is the actual store file tracker implementation, if it writes to the data directory directly, then we need this cleaner, otherwise we do not.

Ok, I'm going with BrokenStoreFileCleaner then.

Just another name suggestion: "LeftoversStoreFileCleaner"... Up to you, @BukrosSzabolcs !

Apache9 · 2021-10-24T13:37:11Z

+ for (HRegion region : regionServer.getRegions()) {
+ for (HStore store : region.getStores()) {
+ //only clean do cleanup in store using file based storefile tracking
+ if (store.getStoreEngine().requireWritingToTmpDirFirst()) {


OK, good, this is exactly what I expected. So the only problem is the naming. Let's find a better name.

Apache9 · 2021-10-24T13:39:24Z

+ deleteFile(file, store, deletedFiles, failedDeletes);
+ }
+
+ private boolean isCompactingFile(FileStatus file, HStore store) {


I suppose we should move the below isXXXFile to HRegion or HStore, with the protection of some locks. Otherwise we may have race, or at least, we depend on a very flaky order of the testing of each condition.
And do we really need to test isCompactingFile here? The compactedFile will not be tracked but compacting files should always be tracked?

I suppose we should move the below isXXXFile to HRegion or HStore, with the protection of some locks. Otherwise we may have race, or at least, we depend on a very flaky order of the testing of each condition.
I guess the isOldEnough check would prevent that problem, as we would only care about files older than DEFAULT_FILEBASED_STOREFILE_CLEANER_TTL?
And do we really need to test isCompactingFile here? The compactedFile will not be tracked but compacting files should always be tracked?
Yeah, this seems redundant with the isActiveStorefile check.

And do we really need to test isCompactingFile here? The compactedFile will not be tracked but compacting files should always be tracked?
The confusion might come from suboptimal naming.
isActiveStorefile -> lists the currently active storefiles
isCompactedFile -> are hfiles that got compacted, no longer active storefiles, but were not deleted yet (deletion is handled by a separate subsystem, so we should not touch them)
isCompactingFile -> file(s) a currently running compaction is writing into. This will become the new storefile when the compaction is done. It is checked to make sure we do not break stuck/longrunning compactions even is they are stuck/idle for more than the configured TTL.
I think we do need all of these checks.
I suppose we should move the below isXXXFile to HRegion or HStore, with the protection of some locks. Otherwise we may have race, or at least, we depend on a very flaky order of the testing of each condition.
As Wellington mentioned, the TTL should make sure we filter out any currently handled hfile. Any file we check afterwards was not touched in 12 hours. I think the only usecase where this might be an issue is a stuck compaction where we check isCompactingFile, but that method (getCompactionTargets) is synchronised so should be safe. Do you see any other possible issue?

The confusion might come from suboptimal naming.
isActiveStorefile -> lists the currently active storefiles
isCompactedFile -> are hfiles that got compacted, no longer active storefiles, but were not deleted yet (deletion is handled by a separate subsystem, so we should not touch them)
isCompactingFile -> file(s) a currently running compaction is writing into. This will become the new storefile when the compaction is done. It is checked to make sure we do not break stuck/longrunning compactions even is they are stuck/idle for more than the configured TTL.
I think we do need all of these checks.
Thanks for clarifying. Thinking again, maybe it's worth having this extra check, for the exceptional cases compaction can last longer than the time threshold.

wchevreuil · 2021-10-25T11:05:53Z

+ deleteFile(file, store, deletedFiles, failedDeletes);
+ }
+
+ private boolean isCompactingFile(FileStatus file, HStore store) {


I suppose we should move the below isXXXFile to HRegion or HStore, with the protection of some locks. Otherwise we may have race, or at least, we depend on a very flaky order of the testing of each condition.
I guess the isOldEnough check would prevent that problem, as we would only care about files older than DEFAULT_FILEBASED_STOREFILE_CLEANER_TTL?
And do we really need to test isCompactingFile here? The compactedFile will not be tracked but compacting files should always be tracked?
Yeah, this seems redundant with the isActiveStorefile check.

Apache-HBase · 2021-10-28T16:24:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 33s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	4m 15s	HBASE-26067 passed
+1 💚	compile	3m 17s	HBASE-26067 passed
+1 💚	checkstyle	1m 11s	HBASE-26067 passed
+1 💚	spotbugs	2m 22s	HBASE-26067 passed
-0 ⚠️	patch	2m 31s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 13s	the patch passed
+1 💚	compile	3m 32s	the patch passed
+1 💚	javac	3m 32s	the patch passed
-0 ⚠️	checkstyle	1m 20s	hbase-server: The patch generated 21 new + 62 unchanged - 0 fixed = 83 total (was 62)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	22m 15s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 20s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		54m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 5ec5a3338182 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `3240b4b`
Default Java	AdoptOpenJDK-1.8.0_282-b08
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	96 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-10-28T19:42:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 2s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	4m 36s	HBASE-26067 passed
+1 💚	compile	1m 4s	HBASE-26067 passed
+1 💚	shadedjars	9m 9s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 40s	HBASE-26067 passed
-0 ⚠️	patch	10m 0s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 18s	the patch passed
+1 💚	compile	1m 7s	the patch passed
+1 💚	javac	1m 7s	the patch passed
+1 💚	shadedjars	9m 13s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	the patch passed
		_ Other Tests _
+1 💚	unit	218m 21s	hbase-server in the patch passed.
		251m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 2d61df0298f6 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `3240b4b`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/2/testReport/
Max. process+thread count	3889 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-10-28T20:10:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 11s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	5m 54s	HBASE-26067 passed
+1 💚	compile	1m 33s	HBASE-26067 passed
+1 💚	shadedjars	10m 1s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 53s	HBASE-26067 passed
-0 ⚠️	patch	11m 5s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 38s	the patch passed
+1 💚	compile	1m 34s	the patch passed
+1 💚	javac	1m 34s	the patch passed
+1 💚	shadedjars	9m 55s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 50s	the patch passed
		_ Other Tests _
-1 ❌	unit	239m 57s	hbase-server in the patch failed.
		279m 25s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 0cb6e6a95b9b 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `3240b4b`
Default Java	AdoptOpenJDK-11.0.10+9
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/2/testReport/
Max. process+thread count	3362 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

BukrosSzabolcs · 2021-10-29T09:07:13Z

The failing tests seem to be flaky. I was able to successfully run them locally.

joshelser

Nit-picky things from me, and one request for an additional test method.

Otherwise, it looks fine to me.

Apache-HBase · 2021-11-04T15:22:40Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	4m 34s	HBASE-26067 passed
+1 💚	compile	3m 39s	HBASE-26067 passed
+1 💚	checkstyle	1m 11s	HBASE-26067 passed
+1 💚	spotbugs	2m 17s	HBASE-26067 passed
-0 ⚠️	patch	2m 27s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 20s	the patch passed
+1 💚	compile	3m 30s	the patch passed
+1 💚	javac	3m 30s	the patch passed
-0 ⚠️	checkstyle	1m 10s	hbase-server: The patch generated 23 new + 62 unchanged - 0 fixed = 85 total (was 62)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	22m 0s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 36s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 16s	The patch does not generate ASF License warnings.
		55m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux b3c8beb2b0f4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `3240b4b`
Default Java	AdoptOpenJDK-1.8.0_282-b08
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	95 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/3/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-11-04T17:23:01Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	4m 41s	HBASE-26067 passed
+1 💚	compile	1m 15s	HBASE-26067 passed
+1 💚	shadedjars	8m 21s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 44s	HBASE-26067 passed
-0 ⚠️	patch	9m 18s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 33s	the patch passed
+1 💚	compile	1m 14s	the patch passed
+1 💚	javac	1m 14s	the patch passed
+1 💚	shadedjars	8m 24s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 39s	the patch passed
		_ Other Tests _
+1 💚	unit	143m 1s	hbase-server in the patch passed.
		175m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b840afe10d42 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `3240b4b`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/3/testReport/
Max. process+thread count	3767 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/3/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

joshelser · 2021-11-04T17:46:50Z

Just to make my intent clear, I think with a test method to exercise the TTL effectiveness and the minor code-formatting cleanup, this is fine to commit.

Not sure if @Apache9 has more opinions on this change before he'd like to see it merged to the feature branch.

Apache-HBase · 2021-11-04T17:53:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	4m 8s	HBASE-26067 passed
+1 💚	compile	1m 1s	HBASE-26067 passed
+1 💚	shadedjars	10m 45s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 49s	HBASE-26067 passed
-0 ⚠️	patch	11m 46s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 28s	the patch passed
+1 💚	compile	1m 19s	the patch passed
+1 💚	javac	1m 19s	the patch passed
+1 💚	shadedjars	11m 35s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 45s	hbase-server generated 5 new + 21 unchanged - 0 fixed = 26 total (was 21)
		_ Other Tests _
-1 ❌	unit	167m 52s	hbase-server in the patch failed.
		206m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c31a3e1d5a27 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `3240b4b`
Default Java	AdoptOpenJDK-1.8.0_282-b08
javadoc	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/3/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/3/testReport/
Max. process+thread count	3543 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/3/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

joshelser

LGTM if QA comes back positively.

Will let this go a day or two for Duo to come back with feedback if he has some.

Apache9 · 2021-11-05T00:54:51Z

+ return this.enabled.get();
+ }
+
+ @InterfaceAudience.Private


Do we need this extra IA annotation? The class itself is IA.Private.

You are right, deleting it.

Apache9 · 2021-11-05T01:00:31Z

+ return;
+ }
+
+ if(isCompactingFile(file, store)){


The 'compacting file' has its meaning in HBase. There is a filesCompacting field in HStore class, which is the files being compacted currently, not the files written out. So here let's change the name, otherwise it will confuse people.

I'm renaming it to "isCompactionResultFile"

Apache9 · 2021-11-06T15:19:09Z

#3700 has been merged and I rebased HBASE-26067 with the newest master.

Please rebase the PR too. Thanks for your patient. We are very close to merge this PR.

) Signed-off-by: Duo Zhang <zhangduo@apache.org>

… tracker (apache#3681) Signed-off-by: Josh Elser <elserj@apache.org>

Add new chore to delete lefotver files in case file based storefile handling is used Expose the target files of currently running compactions for easier validation

fixes based on feedback

added javadoc fixed typos additonal fileTTL UT

improve naming and add some explanation in the comments

clean up after rebase

BukrosSzabolcs · 2021-11-08T19:46:55Z

@Apache9 Rebased my changes

Apache-HBase · 2021-11-08T20:45:54Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 30s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	3m 56s	HBASE-26067 passed
+1 💚	compile	3m 14s	HBASE-26067 passed
+1 💚	checkstyle	1m 5s	HBASE-26067 passed
+1 💚	spotbugs	2m 3s	HBASE-26067 passed
-0 ⚠️	patch	2m 13s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 50s	the patch passed
+1 💚	compile	3m 9s	the patch passed
+1 💚	javac	3m 9s	the patch passed
-0 ⚠️	checkstyle	1m 4s	hbase-server: The patch generated 23 new + 61 unchanged - 1 fixed = 84 total (was 62)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	19m 27s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 15s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		49m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux ef2af575e44e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `a0ec946`
Default Java	AdoptOpenJDK-1.8.0_282-b08
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	96 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/5/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-11-08T22:53:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 37s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	4m 49s	HBASE-26067 passed
+1 💚	compile	1m 13s	HBASE-26067 passed
+1 💚	shadedjars	8m 26s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 44s	HBASE-26067 passed
-0 ⚠️	patch	9m 23s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 28s	the patch passed
+1 💚	compile	1m 14s	the patch passed
+1 💚	javac	1m 14s	the patch passed
+1 💚	shadedjars	8m 19s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 42s	the patch passed
		_ Other Tests _
+1 💚	unit	142m 23s	hbase-server in the patch passed.
		176m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 829bfc833fa7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `a0ec946`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/5/testReport/
Max. process+thread count	3806 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/5/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-11-09T00:09:08Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 8s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	4m 34s	HBASE-26067 passed
+1 💚	compile	1m 5s	HBASE-26067 passed
+1 💚	shadedjars	9m 9s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 39s	HBASE-26067 passed
-0 ⚠️	patch	9m 59s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 16s	the patch passed
+1 💚	compile	1m 3s	the patch passed
+1 💚	javac	1m 3s	the patch passed
+1 💚	shadedjars	9m 2s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 38s	hbase-server generated 5 new + 21 unchanged - 0 fixed = 26 total (was 21)
		_ Other Tests _
+1 💚	unit	218m 56s	hbase-server in the patch passed.
		252m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c01de929f3c1 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `a0ec946`
Default Java	AdoptOpenJDK-1.8.0_282-b08
javadoc	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/5/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/5/testReport/
Max. process+thread count	3194 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/5/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9

Overall LGTM. Just need to add more javadoc or comments to let later developers better understand the code.

And please fix the checkstyle issue before merging.

Thanks @BukrosSzabolcs

checkstyle fixes additonal comments

Apache-HBase · 2021-11-09T10:15:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	4m 30s	HBASE-26067 passed
+1 💚	compile	3m 46s	HBASE-26067 passed
+1 💚	checkstyle	1m 12s	HBASE-26067 passed
+1 💚	spotbugs	2m 18s	HBASE-26067 passed
-0 ⚠️	patch	2m 28s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 8s	the patch passed
+1 💚	compile	3m 18s	the patch passed
+1 💚	javac	3m 18s	the patch passed
+1 💚	checkstyle	1m 9s	hbase-server: The patch generated 0 new + 61 unchanged - 1 fixed = 61 total (was 62)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	19m 33s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 14s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		50m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 0de76c87344c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `a0ec946`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	95 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/6/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-11-09T12:29:14Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 17s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	5m 6s	HBASE-26067 passed
+1 💚	compile	1m 27s	HBASE-26067 passed
+1 💚	shadedjars	8m 26s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 47s	HBASE-26067 passed
-0 ⚠️	patch	9m 25s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 42s	the patch passed
+1 💚	compile	1m 19s	the patch passed
+1 💚	javac	1m 19s	the patch passed
+1 💚	shadedjars	8m 14s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 41s	the patch passed
		_ Other Tests _
+1 💚	unit	150m 47s	hbase-server in the patch passed.
		184m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 7ecdc4fad9ee 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `a0ec946`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/6/testReport/
Max. process+thread count	3982 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/6/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-11-09T13:44:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 6s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-26067 Compile Tests _
+1 💚	mvninstall	4m 19s	HBASE-26067 passed
+1 💚	compile	1m 6s	HBASE-26067 passed
+1 💚	shadedjars	9m 7s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	HBASE-26067 passed
-0 ⚠️	patch	9m 56s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 18s	the patch passed
+1 💚	compile	1m 3s	the patch passed
+1 💚	javac	1m 3s	the patch passed
+1 💚	shadedjars	9m 9s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	the patch passed
		_ Other Tests _
-1 ❌	unit	226m 19s	hbase-server in the patch failed.
		259m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3786
JIRA Issue	HBASE-26271
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 9e385a985cb9 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-26067 / `a0ec946`
Default Java	AdoptOpenJDK-1.8.0_282-b08
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/6/testReport/
Max. process+thread count	3347 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3786/6/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Josh Elser <elserj@apache.org> Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>

…he#3786) Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Josh Elser <elserj@apache.org> Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org> Conflicts: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java

…others) to branch-2.5 Previous cherry picks: commit 6aaef89 HBASE-26064 Introduce a StoreFileTracker to abstract the store file tracking logic commit 43b40e9 HBASE-25988 Store the store file list by a file apache#3578) commit 6e05376 HBASE-26079 Use StoreFileTracker when splitting and merging apache#3617) commit 090b2fe HBASE-26224 HBASE-26224 Introduce a MigrationStoreFileTracker to support migratin… apache#3656) commit 0ee1689 HBASE-26246 Persist the StoreFileTracker configurations to TableDescriptor when creating table apache#3666) commit 2052e80 HBASE-26248 Should find a suitable way to let users specify the store… apache#3665) commit 5ff0f98 HBASE-26264 Add more checks to prevent misconfiguration on store file… apache#3681) commit fc4f6d1 HBASE-26280 HBASE-26280 Use store file tracker when snapshoting apache#3685) commit 06db852 HBASE-26326 CreateTableProcedure fails when FileBasedStoreFileTracker… apache#3721) commit e4e7cf8 HBASE-26386 Refactor StoreFileTracker implementations to expose the s… apache#3774) commit 08d1171 HBASE-26328 Clone snapshot doesn't load reference files into FILE SFT impl apache#3749) commit 8bec26e HBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker configur… apache#3700) commit a288365 HBASE-26271: Cleanup the broken store files under data directory apache#3786) commit d00b5fa HBASE-26454 CreateTableProcedure still relies on temp dir and renames… apache#3845) commit 771e552 HBASE-26286: Add support for specifying store file tracker when restoring or cloning snapshot commit f16b7b1 HBASE-26265 Update ref guide to mention the new store file tracker im… apache#3942) commit 755b3b4 HBASE-26585 Add SFT configuration to META table descriptor when creating META apache#3998) commit 39c42c7 HBASE-26639 The implementation of TestMergesSplitsAddToTracker is pro… apache#4010) commit 6e1f5b7 HBASE-26586 Should not rely on the global config when setting SFT implementation for a table while upgrading apache#4006) commit f1dd865 HBASE-26654 ModifyTableDescriptorProcedure shoud load TableDescriptor… apache#4034) commit 8fbc9a2 HBASE-26674 Should modify filesCompacting under storeWriteLock apache#4040) commit 5aa0fd2 HBASE-26675 Data race on Compactor.writer apache#4035) commit 3021c58 HBASE-26700 The way we bypass broken track file is not enough in Stor… apache#4055) commit a8b68c9 HBASE-26690 Modify FSTableDescriptors to not rely on renaming when wr… apache#4054) commit dffeb8e HBASE-26587 Introduce a new Admin API to change SFT implementation (#… apache#4080) commit b265fe5 HBASE-26673 Implement a shell command for change SFT implementation apache#4113) commit 4cdb380 HBASE-26640 Reimplement master local region initialization to better … apache#4111) commit 77bb153 HBASE-26707: Reduce number of renames during bulkload (apache#4066) apache#4122) commit a4b192e HBASE-26611 Changing SFT implementation on disabled table is dangerous apache#4082) commit d3629bb HBASE-26837 Set SFT config when creating TableDescriptor in TestClone… apache#4226) commit 541d748 HBASE-26881 Backport HBASE-25368 to branch-2 (apache#4267) Fixups for precommit error prone, checkstyle, and javadoc warnings after applying cherry picks. Signed-off-by: Josh Elser <elserj@apache.org> Reviewed-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>

Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Josh Elser <elserj@apache.org> Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org> Conflicts: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java

…others) to branch-2.5 Previous cherry picks: commit 6aaef89 HBASE-26064 Introduce a StoreFileTracker to abstract the store file tracking logic commit 43b40e9HBASE-25988 Store the store file list by a file #3578) commit 6e05376HBASE-26079 Use StoreFileTracker when splitting and merging #3617) commit 090b2feHBASE-26224 HBASE-26224 Introduce a MigrationStoreFileTracker to support migratin… #3656) commit 0ee1689HBASE-26246 Persist the StoreFileTracker configurations to TableDescriptor when creating table #3666) commit 2052e80HBASE-26248 Should find a suitable way to let users specify the store… #3665) commit 5ff0f98HBASE-26264 Add more checks to prevent misconfiguration on store file… #3681) commit fc4f6d1HBASE-26280 HBASE-26280 Use store file tracker when snapshoting #3685) commit 06db852HBASE-26326 CreateTableProcedure fails when FileBasedStoreFileTracker… #3721) commit e4e7cf8HBASE-26386 Refactor StoreFileTracker implementations to expose the s… #3774) commit 08d1171HBASE-26328 Clone snapshot doesn't load reference files into FILE SFT impl #3749) commit 8bec26eHBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker configur… #3700) commit a288365HBASE-26271: Cleanup the broken store files under data directory #3786) commit d00b5faHBASE-26454 CreateTableProcedure still relies on temp dir and renames… #3845) commit 771e552HBASE-26286: Add support for specifying store file tracker when restoring or cloning snapshot commit f16b7b1HBASE-26265 Update ref guide to mention the new store file tracker im… #3942) commit 755b3b4HBASE-26585 Add SFT configuration to META table descriptor when creating META #3998) commit 39c42c7HBASE-26639 The implementation of TestMergesSplitsAddToTracker is pro… #4010) commit 6e1f5b7HBASE-26586 Should not rely on the global config when setting SFT implementation for a table while upgrading #4006) commit f1dd865HBASE-26654 ModifyTableDescriptorProcedure shoud load TableDescriptor… #4034) commit 8fbc9a2HBASE-26674 Should modify filesCompacting under storeWriteLock #4040) commit 5aa0fd2HBASE-26675 Data race on Compactor.writer #4035) commit 3021c58HBASE-26700 The way we bypass broken track file is not enough in Stor… #4055) commit a8b68c9HBASE-26690 Modify FSTableDescriptors to not rely on renaming when wr… #4054) commit dffeb8eHBASE-26587 Introduce a new Admin API to change SFT implementation (#… #4080) commit b265fe5HBASE-26673 Implement a shell command for change SFT implementation #4113) commit 4cdb380HBASE-26640 Reimplement master local region initialization to better … #4111) commit 77bb153HBASE-26707: Reduce number of renames during bulkload (#4066) #4122) commit a4b192e HBASE-26611 Changing SFT implementation on disabled table is dangerous #4082) commit d3629bbHBASE-26837 Set SFT config when creating TableDescriptor in TestClone… #4226) commit 541d748HBASE-26881 Backport HBASE-25368 to branch-2 (#4267) Fixups for precommit error prone, checkstyle, and javadoc warnings after applying cherry picks. Signed-off-by: Josh Elser <elserj@apache.org> Reviewed-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>

…he#3786) Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Josh Elser <elserj@apache.org> Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org> Change-Id: If9825f5199385e8ccc65f4f7637461f3651a6836

Apache9 reviewed Oct 24, 2021
View reviewed changes

wchevreuil reviewed Oct 25, 2021
View reviewed changes

joshelser requested changes Nov 3, 2021
View reviewed changes

joshelser approved these changes Nov 4, 2021
View reviewed changes

Apache9 reviewed Nov 5, 2021
View reviewed changes

Apache9 force-pushed the HBASE-26067 branch from 49a9211 to a0ec946Compare November 6, 2021 15:03

wchevreuiland others added 7 commits November 8, 2021 17:38

HBASE-26079 Use StoreFileTracker when splitting and merging (apache#3617
15681eb
) Signed-off-by: Duo Zhang <zhangduo@apache.org>

HBASE-26264 Add more checks to prevent misconfiguration on store file…
4fa03c3
… tracker (apache#3681) Signed-off-by: Josh Elser <elserj@apache.org>

HBASE-26271: Cleanup the broken store files under data directory
b047f09
Add new chore to delete lefotver files in case file based storefile handling is used Expose the target files of currently running compactions for easier validation

HBASE-26271: Cleanup the broken store files under data directory
2f5cd86
fixes based on feedback

HBASE-26271: Cleanup the broken store files under data directory
9608c7f
added javadoc fixed typos additonal fileTTL UT

HBASE-26271: Cleanup the broken store files under data directory
956f74e
improve naming and add some explanation in the comments

HBASE-26271: Cleanup the broken store files under data directory
ae0fa65
clean up after rebase

BukrosSzabolcs force-pushed the HBASE-26271-2 branch from a86b9ca to ae0fa65Compare November 8, 2021 19:45

wchevreuil approved these changes Nov 8, 2021
View reviewed changes

Apache9 approved these changes Nov 9, 2021
View reviewed changes

Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/BrokenStoreFileCleaner.java
Comment thread...main/java/org/apache/hadoop/hbase/regionserver/compactions/AbstractMultiOutputCompactor.java

HBASE-26271: Cleanup the broken store files under data directory
ed0e58d
checkstyle fixes additonal comments

Apache9 merged commit 96cc039 into apache:HBASE-26067Nov 9, 2021

apurtell mentioned this pull request Mar 18, 2022
Backport StoreFileTracker (HBASE-26067, HBASE-26584, and others) to branch-2.5 #4241
Merged

Uh oh!

Conversation

BukrosSzabolcs commented Oct 22, 2021

Uh oh!

BukrosSzabolcs commented Oct 22, 2021

Uh oh!

Apache-HBase commented Oct 22, 2021

Uh oh!

Apache-HBase commented Oct 22, 2021

Uh oh!

Apache-HBase commented Oct 22, 2021

Uh oh!

Apache9 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Apache-HBase commented Oct 28, 2021

Uh oh!

Apache-HBase commented Oct 28, 2021

Uh oh!

Apache-HBase commented Oct 28, 2021

Uh oh!

BukrosSzabolcs commented Oct 29, 2021

Uh oh!

joshelser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Apache-HBase commented Nov 4, 2021

Uh oh!

Apache-HBase commented Nov 4, 2021

Uh oh!

joshelser commented Nov 4, 2021

Uh oh!

Apache-HBase commented Nov 4, 2021

Uh oh!

joshelser left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!