HBASE-27547 Close store file readers after region warmup by EungsopYoo · Pull Request #4942 · apache/hbase

EungsopYoo · 2022-12-29T09:57:38Z

No description provided.

Apache-HBase · 2022-12-29T10:31:09Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 39s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 41s	master passed
+1 💚	compile	2m 28s	master passed
+1 💚	checkstyle	0m 34s	master passed
+1 💚	spotless	0m 42s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 23s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 25s	the patch passed
+1 💚	compile	2m 24s	the patch passed
-0 ⚠️	javac	2m 24s	hbase-server generated 1 new + 194 unchanged - 1 fixed = 195 total (was 195)
-0 ⚠️	checkstyle	0m 33s	hbase-server: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	8m 55s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
-1 ❌	spotless	0m 34s	patch has 34 errors when running spotless:check, run spotless:apply to fix.
-1 ❌	spotbugs	1m 34s	hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		30m 50s

Reason	Tests
FindBugs	module:hbase-server
	Private method org.apache.hadoop.hbase.regionserver.HStoreFile.open() is never called At HStoreFile.java:called At HStoreFile.java:[lines 381-382]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4942
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux bc0f3cc23178 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7ef63b6`
Default Java	Eclipse Adoptium-11.0.17+8
javac	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt
checkstyle	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
spotless	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/1/artifact/yetus-general-check/output/patch-spotless.txt
spotbugs	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html
Max. process+thread count	78 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2022-12-29T12:50:54Z

I think the problem here is we do not close the region after warmup?

https://issues.apache.org/jira/browse/HBASE-13316
ba6345f#diff-a754d10861bd8d44a6fac8dd2257b1154a03415851613d8b3cd3e99964713d7f

In the original commit, we will close the region after warmup...

Let me see when we removed the r.close() line... Maybe it is just a mistake...

Apache9 · 2022-12-29T13:01:56Z

OK, it is https://issues.apache.org/jira/browse/HBASE-15441.

I think for warmup, the region should be opened as read only so it should not write any markers to WAL.

And HBASE-15441 should have been included in 1.4.13, so this should not be the root cause of too many FDs?

WDYT? @EungsopYoo

Thanks.

Apache-HBase · 2022-12-29T13:56:39Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 55s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 48s	master passed
+1 💚	compile	0m 47s	master passed
+1 💚	shadedjars	3m 48s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 28s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 32s	the patch passed
+1 💚	compile	0m 49s	the patch passed
+1 💚	javac	0m 49s	the patch passed
+1 💚	shadedjars	3m 48s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	the patch passed
		_ Other Tests _
+1 💚	unit	215m 51s	hbase-server in the patch passed.
		236m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4942
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 638c84314852 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7ef63b6`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/1/testReport/
Max. process+thread count	2718 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-12-29T14:02:07Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	5m 33s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 25s	master passed
+1 💚	compile	0m 40s	master passed
+1 💚	shadedjars	3m 47s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 10s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	3m 46s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	217m 36s	hbase-server in the patch passed.
		241m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4942
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b453b2c83546 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7ef63b6`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/1/testReport/
Max. process+thread count	2494 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-12-29T14:32:23Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 39s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 24s	master passed
+1 💚	compile	2m 20s	master passed
+1 💚	checkstyle	0m 32s	master passed
+1 💚	spotless	0m 36s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 18s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 18s	the patch passed
+1 💚	compile	2m 18s	the patch passed
+1 💚	javac	2m 18s	the patch passed
+1 💚	checkstyle	0m 31s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	8m 32s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	0m 36s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 26s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 8s	The patch does not generate ASF License warnings.
		29m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4942
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 5172f7b06061 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7ef63b6`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	78 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-12-29T18:01:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 8s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 12s	master passed
+1 💚	compile	0m 40s	master passed
+1 💚	shadedjars	3m 44s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 16s	the patch passed
+1 💚	compile	0m 38s	the patch passed
+1 💚	javac	0m 38s	the patch passed
+1 💚	shadedjars	3m 45s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 25s	hbase-server generated 1 new + 23 unchanged - 0 fixed = 24 total (was 23)
		_ Other Tests _
+1 💚	unit	218m 21s	hbase-server in the patch passed.
		237m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4942
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 375accf6c3a0 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7ef63b6`
Default Java	Temurin-1.8.0_352-b08
javadoc	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/2/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/2/testReport/
Max. process+thread count	2529 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

EungsopYoo · 2022-12-30T09:23:19Z

OK, it is https://issues.apache.org/jira/browse/HBASE-15441.
I think for warmup, the region should be opened as read only so it should not write any markers to WAL.
And HBASE-15441 should have been included in 1.4.13, so this should not be the root cause of too many FDs?
WDYT? @EungsopYoo
Thanks.

Right. In 1.4.13, the warmed up region is not closed too. It's not the cause.
https://github.com/apache/hbase/blob/rel/1.4.13/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L7163-L7164

EungsopYoo · 2023-01-10T04:52:05Z

@Apache9
This PR does not close the warmed up region. It just closes the store file reader in the region.

We applied this patch to our cluster. And then the number of open FD decreased and never increased during RS rolling restart, which causes region move and warmup. It had been increased during RS rolling restart without this patch.

EungsopYoo · 2023-01-10T11:23:42Z

OK, it is https://issues.apache.org/jira/browse/HBASE-15441.
I think for warmup, the region should be opened as read only so it should not write any markers to WAL.
And HBASE-15441 should have been included in 1.4.13, so this should not be the root cause of too many FDs?
WDYT? @EungsopYoo
Thanks.
Right. In 1.4.13, the warmed up region is not closed too. It's not the cause. https://github.com/apache/hbase/blob/rel/1.4.13/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L7163-L7164

I misunderstood your question.

There are two points. The first is the leakage of open FDs during region moving(due to warmup), and the second is that HBase 2.4 opens much more FDs than HBase 1.4.

In this PR, I fixed the first. The second is still in question.

Apache9 · 2023-01-11T03:55:15Z

OK, it is https://issues.apache.org/jira/browse/HBASE-15441.
I think for warmup, the region should be opened as read only so it should not write any markers to WAL.
And HBASE-15441 should have been included in 1.4.13, so this should not be the root cause of too many FDs?
WDYT? @EungsopYoo
Thanks.
Right. In 1.4.13, the warmed up region is not closed too. It's not the cause. https://github.com/apache/hbase/blob/rel/1.4.13/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L7163-L7164
I misunderstood your question.
There are two points. The first is the leakage of open FDs during region moving(due to warmup), and the second is that HBase 2.4 opens much more FDs than HBase 1.4.
In this PR, I fixed the first. The second is still in question.

So I mean we should try to restore the old behavior, to close the warm up region? Now we support open a region in read only mode so when closing the region, we will not write any wal out, so it will not mess up wal splitting.
WDYT?

Thanks.

EungsopYoo · 2023-01-11T11:38:14Z

OK, it is https://issues.apache.org/jira/browse/HBASE-15441.
I think for warmup, the region should be opened as read only so it should not write any markers to WAL.
And HBASE-15441 should have been included in 1.4.13, so this should not be the root cause of too many FDs?
WDYT? @EungsopYoo
Thanks.
Right. In 1.4.13, the warmed up region is not closed too. It's not the cause. https://github.com/apache/hbase/blob/rel/1.4.13/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L7163-L7164
I misunderstood your question.
There are two points. The first is the leakage of open FDs during region moving(due to warmup), and the second is that HBase 2.4 opens much more FDs than HBase 1.4.
In this PR, I fixed the first. The second is still in question.
So I mean we should try to restore the old behavior, to close the warm up region? Now we support open a region in read only mode so when closing the region, we will not write any wal out, so it will not mess up wal splitting. WDYT?
Thanks.

I think it's good to close warmed up region. Is it enough just to close the region?

--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java@@ -7773,6 +7773,7 @@ public class HRegion implements HeapSize, PropagatingConfigurationObserver, Regi
}
HRegion r = HRegion.newHRegion(tableDir, wal, fs, conf, info, htd, null);
r.initializeWarmup(reporter);
+ r.close();
}
/**

Apache9 · 2023-01-11T13:15:53Z

Ah

 // Always write close marker to wal even for read only table. This is not a big problem as we
// do not write any data into the region; it is just a meta edit in the WAL file.
if (
!abort && wal != null && getRegionServerServices() != null
&& RegionReplicaUtil.isDefaultReplica(getRegionInfo())
) {
writeRegionCloseMarker(wal);
}

So set region to read only is not enough...

Let me try if it is possible to set rsServices to null or wal to null here...

Apache9 · 2023-01-11T13:31:30Z

Oh, we pass null as rsServices in HBASE-15441, so we will not mess up the sequence id, so closing the region will not introduce the problem described in HBASE-15441 back.

In HBASE-15441, @elliottneilclark said

Yea, it sure seems like the close isn't needed at all.
It tries to:
clean the cache ( something that warmup shouldn't be doing anyway ).
Flush all edit ( there should be no edits ever ).
Disable all compactions ( there are no compactions running since this is read only).

Setting the region to readOnly can prevent the flushing of edits, disabling all compactions is no harm.

IIRC, there is a config to control whether we should evict block cache entries when closing a region.

Let me check.

Apache9 · 2023-01-11T13:55:44Z

EvictOnClose is a global config...

 for (HStoreFile f : result) {
completionService.submit(new Callable<Void>() {
@Override
public Void call() throws IOException {
boolean evictOnClose =
getCacheConfig() != null ? getCacheConfig().shouldEvictOnClose() : true;
f.closeStoreFile(evictOnClose);
return null;
}
});
}

I think we could introduce a config here, like hbase.hstore.cache.evictblocksonclose. If it is set, we can override the global config, if it is not set, we will use the global config. And when wramming up a region, we will set this flag to false.

EungsopYoo · 2023-01-12T07:57:20Z

How about this?

Apache-HBase · 2023-01-12T08:35:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 41s	master passed
+1 💚	compile	2m 21s	master passed
+1 💚	checkstyle	0m 35s	master passed
+1 💚	spotless	0m 40s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 20s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 18s	the patch passed
+1 💚	compile	2m 23s	the patch passed
+1 💚	javac	2m 23s	the patch passed
-0 ⚠️	checkstyle	0m 32s	hbase-server: The patch generated 1 new + 28 unchanged - 0 fixed = 29 total (was 28)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	9m 0s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
-1 ❌	spotless	0m 33s	patch has 24 errors when running spotless:check, run spotless:apply to fix.
+1 💚	spotbugs	1m 26s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 9s	The patch does not generate ASF License warnings.
		30m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4942
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 8aff2fe3ca75 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c1b9d8b`
Default Java	Eclipse Adoptium-11.0.17+8
checkstyle	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
spotless	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/3/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/3/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-01-16T10:40:45Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 39s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 46s	master passed
+1 💚	compile	2m 21s	master passed
+1 💚	checkstyle	0m 32s	master passed
+1 💚	spotless	0m 39s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 23s	the patch passed
+1 💚	compile	2m 23s	the patch passed
+1 💚	javac	2m 23s	the patch passed
+1 💚	checkstyle	0m 33s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	8m 55s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	0m 37s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 25s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 8s	The patch does not generate ASF License warnings.
		30m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4942
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 561040ddb183 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2a7c69d`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/4/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-01-16T14:02:23Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 2s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 33s	master passed
+1 💚	compile	0m 48s	master passed
+1 💚	shadedjars	3m 44s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 31s	the patch passed
+1 💚	compile	0m 47s	the patch passed
+1 💚	javac	0m 47s	the patch passed
+1 💚	shadedjars	3m 45s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	211m 47s	hbase-server in the patch passed.
		231m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4942
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b44500476ea2 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2a7c69d`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/4/testReport/
Max. process+thread count	2502 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4942/4/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9

OK, good, saving the warmup flag is also a good way to fix it, as if we set warmup to true,it does not make sense to still clear the cache entries.

Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 45fd3f6)

taklwu · 2023-01-29T04:20:27Z

patch to branch-2.5 is failing the build , it would need a import line for TableDescriptor in hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestWarmupRegion.java.

I may add an addendum for it in branch-2.5 and branch-2 (branch-2.4 has it) , or open PR if I'm too late for appending an addendum

import org.apache.hadoop.hbase.client.TableDescriptor;

Apache9 · 2023-01-29T05:33:43Z

Just do it, when applying HBASE-27598 to branch-2 I also found the problem and include the fix when cherry-picking HBASE-27598, so you just need to fix branch-2.5. Sorry for that.
Branch-2.4 is OK because the cherry-picking is not clean so I manually fixed the compilation error...

Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 45fd3f6) (cherry picked from commit 79ecc22) Change-Id: I6c77d5f1179f0d568be46c2a55343ed86522f48a

Close store file readers after region warmup
96b7fe3

fix violations
9c22e81

Revert two commits
377c5e1

EungsopYoo force-pushed the HBASE-27547 branch from 78cf4c9 to cade1a1Compare January 12, 2023 07:58

Close region after warmup
e808205

EungsopYoo force-pushed the HBASE-27547 branch from cade1a1 to e808205Compare January 12, 2023 08:03

fix a violation
77e44f5

Apache9 approved these changes Jan 28, 2023
View reviewed changes

Apache9 merged commit 45fd3f6 into apache:masterJan 28, 2023

Apache9 pushed a commit that referenced this pull request Jan 28, 2023
HBASE-27547Close store file readers after region warmup (#4942)
93ad9c4
Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 45fd3f6)

Apache9 pushed a commit that referenced this pull request Jan 28, 2023
HBASE-27547Close store file readers after region warmup (#4942)
f65dfe5
Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 45fd3f6)

Apache9 pushed a commit that referenced this pull request Jan 28, 2023
HBASE-27547Close store file readers after region warmup (#4942)
79ecc22
Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 45fd3f6)

Uh oh!

Conversation

EungsopYoo commented Dec 29, 2022

Uh oh!

Apache-HBase commented Dec 29, 2022

Uh oh!

Apache9 commented Dec 29, 2022

Uh oh!

Apache9 commented Dec 29, 2022

Uh oh!

Apache-HBase commented Dec 29, 2022

Uh oh!

Apache-HBase commented Dec 29, 2022

Uh oh!

Apache-HBase commented Dec 29, 2022

Uh oh!

Apache-HBase commented Dec 29, 2022

Uh oh!

EungsopYoo commented Dec 30, 2022

Uh oh!

EungsopYoo commented Jan 10, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EungsopYoo commented Jan 10, 2023

Uh oh!

Apache9 commented Jan 11, 2023

Uh oh!

EungsopYoo commented Jan 11, 2023

Uh oh!

Apache9 commented Jan 11, 2023

Uh oh!

Apache9 commented Jan 11, 2023

Uh oh!

Apache9 commented Jan 11, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EungsopYoo commented Jan 12, 2023

Uh oh!

Apache-HBase commented Jan 12, 2023

Uh oh!

Apache-HBase commented Jan 16, 2023

Uh oh!

Apache-HBase commented Jan 16, 2023

Uh oh!

Apache9 left a comment

Choose a reason for hiding this comment

Uh oh!

taklwu commented Jan 29, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Apache9 commented Jan 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

EungsopYoo commented Jan 10, 2023•
edited
Loading

Apache9 commented Jan 11, 2023•
edited
Loading

taklwu commented Jan 29, 2023•
edited
Loading