HBASE-28114 Add more comments to explain why replication log queue could never be empty for normal replication queue by Apache9 · Pull Request #5443 · apache/hbase

Apache9 · 2023-09-29T11:02:14Z

No description provided.

Apache9 · 2023-09-29T11:07:39Z

I've found a way to deal with the problem without checking whether the replication source is recovered or not. And Why I notice this is because I can not reproduce the problem simply when writing UT...

As if we want to simulate the problem on branch-2+, we need to first let the shipper thread finish calling walFileLengthProvider.getLogFileSizeIfBeingWritten, and halt there, waiting for the log roller thread to start rolling, finish writing the trailer, and before finish postLogRoll(where we enqueue the new WAL file into logQueue), the shipper thread resumes, finish the read, dequeue the current WAL file, get a RETRY_IMMEDIATELY, and then issue the next read, finally we can hit the empty logQueue...

Let me think how to simuate this in a UT...

Apache-HBase · 2023-09-29T11:54:48Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 17s	master passed
+1 💚	compile	3m 15s	master passed
+1 💚	checkstyle	0m 42s	master passed
+1 💚	spotless	0m 49s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 45s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 30s	the patch passed
+1 💚	compile	3m 0s	the patch passed
+1 💚	javac	3m 0s	the patch passed
+1 💚	checkstyle	0m 43s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	13m 9s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	0m 58s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 21s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 17s	The patch does not generate ASF License warnings.
		44m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 1e58b2209219 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `78e6a51`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-09-29T15:13:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 22s	master passed
+1 💚	compile	1m 7s	master passed
+1 💚	shadedjars	6m 14s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 34s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 27s	the patch passed
+1 💚	compile	0m 53s	the patch passed
+1 💚	javac	0m 53s	the patch passed
+1 💚	shadedjars	5m 47s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
+1 💚	unit	215m 6s	hbase-server in the patch passed.
		242m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux fb3e4c769056 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `78e6a51`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/1/testReport/
Max. process+thread count	4663 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-09-29T15:36:00Z

I tried to reproduce the problem by matching the above executing sequence, but then I found that the newly added code was not executed...

The problem here is that, we are only safe to move to the next file when beingWritten == true when we get a EOF_WITH_TRAILER, and we will schedule a close writer task, which will write the trailer, under the rollWriterLock. And the trailerPresent flag is set while opening the wal reader.

This means that if we get beingWritten == true, then it is impossible the reader has a trailerPresent == true at the same time, because we can make sure that the wal reader is opened before we write the trailer, so we can only get a EOF_AND_RESET, and next time when we hit walFileLengthProvider.getLogFileSizeIfBeingWritten again, it can only return beingWritten == false, because we need to get the rollWriterLock, and once we get the rollWriterLock, it means the log roll has already been finished, thus the postLogRoll has been called so the new WAL file has been enqueued to the logQueue...

Let me just add more comments in code to describe the logic here.

Apache-HBase · 2023-09-29T16:19:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 40s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 23s	master passed
+1 💚	compile	0m 54s	master passed
+1 💚	shadedjars	6m 18s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 32s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 41s	the patch passed
+1 💚	compile	0m 49s	the patch passed
+1 💚	javac	0m 49s	the patch passed
+1 💚	shadedjars	5m 15s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	282m 53s	hbase-server in the patch failed.
		308m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux fbb2be8e0223 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `78e6a51`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/1/testReport/
Max. process+thread count	4840 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-09-30T09:29:23Z

@sunhelly PTAL.

Thanks.

Apache-HBase · 2023-09-30T10:05:53Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 25s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 46s	master passed
+1 💚	compile	2m 20s	master passed
+1 💚	checkstyle	0m 33s	master passed
+1 💚	spotless	0m 40s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 22s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 24s	the patch passed
+1 💚	compile	2m 20s	the patch passed
+1 💚	javac	2m 20s	the patch passed
+1 💚	checkstyle	0m 32s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	9m 11s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	0m 40s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 28s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 9s	The patch does not generate ASF License warnings.
		30m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 294dc2977912 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `ff0f0e4`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	77 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-09-30T14:08:53Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 33s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 35s	master passed
+1 💚	compile	0m 40s	master passed
+1 💚	shadedjars	4m 49s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 17s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	4m 48s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	251m 52s	hbase-server in the patch failed.
		273m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux af2b87913c68 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `ff0f0e4`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/2/testReport/
Max. process+thread count	4866 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-09-30T14:28:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 25s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 58s	master passed
+1 💚	compile	0m 58s	master passed
+1 💚	shadedjars	5m 59s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 31s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 19s	the patch passed
+1 💚	compile	0m 56s	the patch passed
+1 💚	javac	0m 56s	the patch passed
+1 💚	shadedjars	6m 1s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 29s	the patch passed
		_ Other Tests _
-1 ❌	unit	266m 48s	hbase-server in the patch failed.
		293m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 941fc6e58e11 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `ff0f0e4`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/2/testReport/
Max. process+thread count	4698 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-10-01T07:24:32Z

Got this assertion error while runing UTs

***** ABORTING region server 941fc6e58e11,38119,1696075363909: Unexpected exception in RS_REFRESH_PEER-regionserver/941fc6e58e11:0-0.replicationSource,2-941fc6e58e11,38119,1696075363909.replicationSource.wal-reader.941fc6e58e11%2C38119%2C1696075363909,2-941fc6e58e11,38119,1696075363909 *****
Cause:
java.lang.AssertionError
at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:350)
at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:129)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:147)
at org.apache.hadoop.hbase.replication.regionserver.SerialReplicationSourceWALReader.run(SerialReplicationSourceWALReader.java:35)

Let me think how could this happen.

Apache9 · 2023-10-01T07:32:35Z

OK, I think the problem is inflightWALClosures.

 @Override
public OptionalLong getLogFileSizeIfBeingWritten(Path path) {
rollWriterLock.lock();
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
// Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) : OptionalLong.empty();
} else {
W temp = inflightWALClosures.get(path.getName());
if (temp != null) {
// In the process of being closed, trailer bytes may or may not be flushed.
// Ensuring that we read all the bytes in a file is critical for correctness of tailing
// use cases like replication, see HBASE-25924/HBASE-25932.
return OptionalLong.of(temp.getSyncedLength());
}
// Log rolled successfully.
return OptionalLong.empty();
}
} finally {
rollWriterLock.unlock();
}
}

Since we close the WAL writers in a background thread, to speed up the log rolling, it is possible that, we have write the trailer and closed the WAL writer, but we haven't removed it from the inflightWALClosures yet, so we will get a beingWritten == true but we have a trailer, but there should be no harm.

Let me think what is the best way to handle this here.

Apache-HBase · 2023-10-01T08:27:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 14s	master passed
+1 💚	compile	2m 55s	master passed
+1 💚	checkstyle	0m 39s	master passed
+1 💚	spotless	0m 46s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 37s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 24s	the patch passed
+1 💚	compile	2m 54s	the patch passed
+1 💚	javac	2m 54s	the patch passed
+1 💚	checkstyle	0m 42s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 24s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	0m 58s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 9s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		42m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 7c90cf1515fc 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `076b5fb`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/3/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-01T14:22:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 16s	master passed
+1 💚	compile	1m 0s	master passed
+1 💚	shadedjars	5m 58s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 23s	the patch passed
+1 💚	compile	0m 51s	the patch passed
+1 💚	javac	0m 51s	the patch passed
+1 💚	shadedjars	5m 34s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	354m 46s	hbase-server in the patch failed.
		397m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 88ee58eb3359 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `076b5fb`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/3/testReport/
Max. process+thread count	29434 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/3/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-02T10:52:33Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 45s	master passed
+1 💚	compile	3m 7s	master passed
+1 💚	checkstyle	0m 39s	master passed
+1 💚	spotless	0m 49s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 50s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 40s	the patch passed
+1 💚	compile	2m 59s	the patch passed
+1 💚	javac	2m 59s	the patch passed
+1 💚	checkstyle	0m 41s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 57s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	0m 54s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 21s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		44m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 896b568d1542 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `076b5fb`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	78 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/4/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-02T16:52:16Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 41s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 48s	master passed
+1 💚	compile	1m 3s	master passed
+1 💚	shadedjars	6m 12s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 34s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 31s	the patch passed
+1 💚	compile	0m 56s	the patch passed
+1 💚	javac	0m 56s	the patch passed
+1 💚	shadedjars	5m 46s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	361m 15s	hbase-server in the patch failed.
		404m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 5609ca89cf1a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `076b5fb`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/4/testReport/
Max. process+thread count	25400 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/4/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-02T16:57:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 43s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 26s	master passed
+1 💚	compile	0m 40s	master passed
+1 💚	shadedjars	4m 52s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 20s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	4m 49s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	369m 13s	hbase-server in the patch failed.
		409m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 361a8346c3cb 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `076b5fb`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/4/testReport/
Max. process+thread count	6633 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/4/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-10-03T12:10:54Z

OK, seems there are still possible ways to have beingWritten == true and we successfully read the trailer but the logQueue still has only 1 WAL file...

Anyway, let me change the code to retry without polling out the last WAL file, to not allow empty queue.

It could fix the problem and make us safe but I still cannot find out how to reproduce this problem in real world...

Can file another issue to track this problem in long term...

Thanks.

Apache-HBase · 2023-10-03T12:51:33Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 37s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 43s	master passed
+1 💚	compile	2m 58s	master passed
+1 💚	checkstyle	0m 35s	master passed
+1 💚	spotless	0m 48s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 43s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 35s	the patch passed
+1 💚	compile	3m 5s	the patch passed
+1 💚	javac	3m 5s	the patch passed
+1 💚	checkstyle	0m 42s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	13m 4s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	0m 56s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 24s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		46m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 81edc965f9b9 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `3bf3b96`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/5/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-03T18:45:47Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 37s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 39s	master passed
+1 💚	compile	0m 57s	master passed
+1 💚	shadedjars	6m 7s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 33s	the patch passed
+1 💚	compile	0m 55s	the patch passed
+1 💚	javac	0m 55s	the patch passed
+1 💚	shadedjars	5m 52s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	the patch passed
		_ Other Tests _
-1 ❌	unit	355m 43s	hbase-server in the patch failed.
		400m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 74658fc4484b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `3bf3b96`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/5/testReport/
Max. process+thread count	29963 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/5/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-07T07:12:35Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 49s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	5m 17s	master passed
+1 💚	compile	3m 8s	master passed
+1 💚	checkstyle	0m 39s	master passed
+1 💚	spotless	0m 49s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 50s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 37s	the patch passed
+1 💚	compile	3m 6s	the patch passed
+1 💚	javac	3m 6s	the patch passed
+1 💚	checkstyle	0m 43s	the patch passed
+1 💚	whitespace	0m 1s	The patch has no whitespace issues.
+1 💚	hadoopcheck	13m 23s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	0m 58s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 29s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		45m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 69f65c0509df 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `6455c49`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/6/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-07T12:49:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 51s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	5m 12s	master passed
+1 💚	compile	1m 3s	master passed
+1 💚	shadedjars	6m 11s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 34s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 40s	the patch passed
+1 💚	compile	0m 55s	the patch passed
+1 💚	javac	0m 55s	the patch passed
+1 💚	shadedjars	5m 56s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	the patch passed
		_ Other Tests _
-1 ❌	unit	353m 13s	hbase-server in the patch failed.
		382m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 0b01e8adef20 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `6455c49`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/6/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/6/testReport/
Max. process+thread count	4660 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/6/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-07T14:22:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 46s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 24s	master passed
+1 💚	compile	0m 40s	master passed
+1 💚	shadedjars	4m 53s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 20s	the patch passed
+1 💚	compile	0m 41s	the patch passed
+1 💚	javac	0m 41s	the patch passed
+1 💚	shadedjars	4m 47s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
-1 ❌	unit	454m 55s	hbase-server in the patch failed.
		476m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 0042352f5d22 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `6455c49`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/6/testReport/
Max. process+thread count	4851 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/6/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-10-08T03:40:42Z

Oh I wrote a wrong condition...

Let's see if the new assertion can pass all the replication related UTs.

But anyway, now I do not think we should add such strict assertion(where we even do not enable it in production), instead, we'd better deal with the situation in code and add a WARN log to say that typically this should not happen...

Apache-HBase · 2023-10-08T04:35:46Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 39s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 55s	master passed
+1 💚	compile	3m 8s	master passed
+1 💚	checkstyle	0m 37s	master passed
+1 💚	spotless	0m 49s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 52s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 35s	the patch passed
+1 💚	compile	3m 8s	the patch passed
+1 💚	javac	3m 8s	the patch passed
+1 💚	checkstyle	0m 42s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	13m 15s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	1m 1s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 24s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 16s	The patch does not generate ASF License warnings.
		45m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/7/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux a33adf45aa65 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `865a595`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	77 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/7/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-10-08T10:52:38Z

2023-10-08T06:42:20,699 INFO [RS:0;c85796445aa7:36275 {}] wal.AbstractFSWAL(980): New WAL /user/jenkins/test-data/daf7aa3e-9117-b174-d093-9e8787b5dfb7/WALs/c85796445aa7,36275,1696747321009/c85796445aa7%2C36275%2C1696747321009-1696747340579-1.1696747340583.syncrep
2023-10-08T06:42:20,708 DEBUG [RS:0;c85796445aa7:36275 {}] regionserver.ReplicationSourceManager(789): Start tracking logs for wal group c85796445aa7%2C36275%2C1696747321009-1696747340579-1 for peer 1
2023-10-08T06:42:20,709 DEBUG [RS:0;c85796445aa7:36275 {}] regionserver.ReplicationSource(369): peerId=1, starting shipping worker for walGroupId=c85796445aa7%2C36275%2C1696747321009-1696747340579-1
2023-10-08T06:42:20,709 INFO [RS:0;c85796445aa7:36275 {}] regionserver.ReplicationSourceWALReader(109): peerClusterZnode=1-c85796445aa7,36275,1696747321009, ReplicationSourceWALReaderThread : 1 inited, replicationBatchSizeCapacity=102400, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=1
2023-10-08T06:42:20,710 DEBUG [RS:0;c85796445aa7:36275.replicationSource.wal-reader.c85796445aa7%2C36275%2C1696747321009-1696747340579-1,1-c85796445aa7,36275,1696747321009 {}] regionserver.WALEntryStream(249): Creating new reader hdfs://localhost:32875/user/jenkins/test-data/daf7aa3e-9117-b174-d093-9e8787b5dfb7/WALs/c85796445aa7,36275,1696747321009/c85796445aa7%2C36275%2C1696747321009-1696747340579-1.1696747340583.syncrep, startPosition=0, beingWritten=false

You can see that, the beingWritten flag for a newly created WAL file is false, which obviously incorrect...
I think the problem is specific to sync replication

 private WAL getRemoteWAL(RegionInfo region, String peerId, String remoteWALDir)
throws IOException {
Optional<WAL> opt = peerId2WAL.get(peerId);
if (opt != null) {
return opt.orElse(null);
}
Lock lock = createLock.acquireLock(peerId);
try {
opt = peerId2WAL.get(peerId);
if (opt != null) {
return opt.orElse(null);
}
WAL wal = createRemoteWAL(region, ReplicationUtils.getRemoteWALFileSystem(conf, remoteWALDir),
ReplicationUtils.getPeerRemoteWALDir(remoteWALDir, peerId), getRemoteWALPrefix(peerId),
ReplicationUtils.SYNC_WAL_SUFFIX);
initWAL(wal);
peerId2WAL.put(peerId, Optional.of(wal));
return wal;
} finally {
lock.unlock();
}
}

Here, we will init the WAL first, before putting it into the peerId2WAL map, and in WAL.init, we will call rollWriter, so we will insert the WAL file into the replication queue, and if we test whether the file is beingWritten before putting it into the peerId2WAL map, we will get a false, which is incorrect...

For normal replication, in AbstractFSWAL, if the wal field is null, we will hold the createLock before returning nothing so it will be safe.

Let me think how to fix this, should be another issue...

Apache-HBase · 2023-10-13T15:25:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 35s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 43s	master passed
+1 💚	compile	3m 1s	master passed
+1 💚	checkstyle	0m 39s	master passed
+1 💚	spotless	0m 46s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 49s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 28s	the patch passed
+1 💚	compile	2m 56s	the patch passed
+1 💚	javac	2m 56s	the patch passed
+1 💚	checkstyle	0m 40s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	13m 0s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	0m 57s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 25s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		46m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/8/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux bdcb9d5cbe03 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `391dfda`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	78 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/8/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-13T20:55:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 38s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 38s	master passed
+1 💚	compile	1m 2s	master passed
+1 💚	shadedjars	6m 8s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 32s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 28s	the patch passed
+1 💚	compile	0m 55s	the patch passed
+1 💚	javac	0m 55s	the patch passed
+1 💚	shadedjars	5m 46s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	346m 12s	hbase-server in the patch failed.
		376m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/8/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 33bcc95c3cce 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `391dfda`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/8/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/8/testReport/
Max. process+thread count	4692 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/8/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-13T21:25:09Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 46s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 12s	master passed
+1 💚	compile	0m 57s	master passed
+1 💚	shadedjars	7m 2s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 33s	the patch passed
+1 💚	compile	1m 2s	the patch passed
+1 💚	javac	1m 2s	the patch passed
+1 💚	shadedjars	7m 9s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 31s	the patch passed
		_ Other Tests _
-1 ❌	unit	375m 43s	hbase-server in the patch failed.
		405m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/8/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 77d9d916acf4 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `391dfda`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/8/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/8/testReport/
Max. process+thread count	5018 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/8/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-14T08:23:43Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 41s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 46s	master passed
+1 💚	compile	3m 0s	master passed
+1 💚	checkstyle	0m 35s	master passed
+1 💚	spotless	0m 47s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 44s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 27s	the patch passed
+1 💚	compile	2m 58s	the patch passed
+1 💚	javac	2m 58s	the patch passed
+1 💚	checkstyle	0m 39s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 58s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	0m 55s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 29s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 16s	The patch does not generate ASF License warnings.
		43m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/9/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 533d741aac90 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `391dfda`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	77 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/9/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-14T11:41:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 39s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 45s	master passed
+1 💚	compile	0m 58s	master passed
+1 💚	shadedjars	6m 11s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 24s	the patch passed
+1 💚	compile	0m 52s	the patch passed
+1 💚	javac	0m 52s	the patch passed
+1 💚	shadedjars	5m 45s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
+1 💚	unit	214m 43s	hbase-server in the patch passed.
		241m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux de01db21ae9a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `391dfda`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/9/testReport/
Max. process+thread count	4602 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/9/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-14T11:53:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 18s	master passed
+1 💚	compile	0m 40s	master passed
+1 💚	shadedjars	4m 49s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 23s	the patch passed
+1 💚	compile	0m 41s	the patch passed
+1 💚	javac	0m 41s	the patch passed
+1 💚	shadedjars	4m 47s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	231m 48s	hbase-server in the patch failed.
		253m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux ae92f3c442bd 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `391dfda`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/9/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/9/testReport/
Max. process+thread count	4834 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/9/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-10-15T10:16:20Z

The ony failed UT is TestZooKeeper, which is not related.

So the assertion works fine. Anyway, let me change the assertion to a warning log to make the code more robust.

Apache-HBase · 2023-10-15T14:40:55Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 39s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 44s	master passed
+1 💚	compile	2m 59s	master passed
+1 💚	checkstyle	0m 38s	master passed
+1 💚	spotless	0m 46s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 43s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 28s	the patch passed
+1 💚	compile	3m 2s	the patch passed
+1 💚	javac	3m 2s	the patch passed
+1 💚	checkstyle	0m 41s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	13m 11s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	0m 55s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 27s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		44m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/10/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux babdf1e8e3de 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `391dfda`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	83 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/10/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-15T18:03:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 40s	master passed
+1 💚	compile	0m 56s	master passed
+1 💚	shadedjars	6m 9s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 34s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 27s	the patch passed
+1 💚	compile	0m 54s	the patch passed
+1 💚	javac	0m 54s	the patch passed
+1 💚	shadedjars	5m 50s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
-1 ❌	unit	219m 3s	hbase-server in the patch failed.
		246m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/10/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 8c5d72df62de 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `391dfda`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/10/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/10/testReport/
Max. process+thread count	4666 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/10/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-15T18:58:05Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 45s	master passed
+1 💚	compile	0m 47s	master passed
+1 💚	shadedjars	5m 12s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 44s	the patch passed
+1 💚	compile	0m 52s	the patch passed
+1 💚	javac	0m 52s	the patch passed
+1 💚	shadedjars	5m 13s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
-1 ❌	unit	278m 35s	hbase-server in the patch failed.
		302m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/10/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 2c0085825d5f 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `391dfda`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/10/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/10/testReport/
Max. process+thread count	4648 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/10/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-10-16T14:12:06Z

For TestZooKeeper, I filed HBASE-28154 for it.

And for TestSyncReplicationStandbyKillRS, this should be a bug as we hit a NPE...

Let me dig more...

2023-10-15T16:04:54,666 ERROR [RS_CLAIM_REPLICATION_QUEUE-regionserver/8c5d72df62de:0-0.replicationSource,1-8c5d72df62de,35419,1697385888915/8c5d72df62de,46527,1697385859222.replicationSource.shipper8c5d72df62de%2C46527%2C1697385859222,1-8c5d72df62de,35419,1697385888915/8c5d72df62de,46527,1697385859222 {}] regionserver.ReplicationSource(452): Unexpected exception in RS_CLAIM_REPLICATION_QUEUE-regionserver/8c5d72df62de:0-0.replicationSource,1-8c5d72df62de,35419,1697385888915/8c5d72df62de,46527,1697385859222.replicationSource.shipper8c5d72df62de%2C46527%2C1697385859222,1-8c5d72df62de,35419,1697385888915/8c5d72df62de,46527,1697385859222 currentPath=null
java.lang.NullPointerException: null
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.cleanOldLogs(ReplicationSourceManager.java:662) ~[classes/:?]
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:649) ~[classes/:?]
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceInterface.logPositionAndCleanOldLogs(ReplicationSourceInterface.java:211) ~[classes/:?]
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.updateLogPosition(ReplicationSourceShipper.java:266) ~[classes/:?]
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:158) ~[classes/:?]
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:119) ~[classes/:?]
2023-10-15T16:04:54,666 ERROR [RS_CLAIM_REPLICATION_QUEUE-regionserver/8c5d72df62de:0-0.replicationSource,1-8c5d72df62de,35419,1697385888915/8c5d72df62de,46527,1697385859222.replicationSource.shipper8c5d72df62de%2C46527%2C1697385859222,1-8c5d72df62de,35419,1697385888915/8c5d72df62de,46527,1697385859222 {}] regionserver.HRegionServer(2389): ***** ABORTING region server 8c5d72df62de,35419,1697385888915: Unexpected exception in RS_CLAIM_REPLICATION_QUEUE-regionserver/8c5d72df62de:0-0.replicationSource,1-8c5d72df62de,35419,1697385888915/8c5d72df62de,46527,1697385859222.replicationSource.shipper8c5d72df62de%2C46527%2C1697385859222,1-8c5d72df62de,35419,1697385888915/8c5d72df62de,46527,1697385859222 *****
java.lang.NullPointerException: null
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.cleanOldLogs(ReplicationSourceManager.java:662) ~[classes/:?]
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:649) ~[classes/:?]
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceInterface.logPositionAndCleanOldLogs(ReplicationSourceInterface.java:211) ~[classes/:?]
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.updateLogPosition(ReplicationSourceShipper.java:266) ~[classes/:?]
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:158) ~[classes/:?]
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:119) ~[classes/:?]

…uld never be empty for normal replication queue Also add a retry logic to make the code more robust

Apache-HBase · 2023-10-20T04:54:07Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 47s	master passed
+1 💚	compile	3m 1s	master passed
+1 💚	checkstyle	0m 35s	master passed
+1 💚	spotless	0m 48s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 42s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 30s	the patch passed
+1 💚	compile	3m 2s	the patch passed
+1 💚	javac	3m 2s	the patch passed
+1 💚	checkstyle	0m 39s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 47s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	0m 58s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 28s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		43m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/11/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 2abdeceb9882 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `dde504c`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	78 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/11/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-20T08:14:48Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 45s	master passed
+1 💚	compile	1m 0s	master passed
+1 💚	shadedjars	6m 7s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 34s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 26s	the patch passed
+1 💚	compile	0m 53s	the patch passed
+1 💚	javac	0m 53s	the patch passed
+1 💚	shadedjars	5m 43s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
-1 ❌	unit	216m 41s	hbase-server in the patch failed.
		243m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/11/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux bf8514c2c9d3 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `dde504c`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/11/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/11/testReport/
Max. process+thread count	4684 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/11/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-20T08:28:15Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 20s	master passed
+1 💚	compile	0m 42s	master passed
+1 💚	shadedjars	4m 50s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 23s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	4m 48s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	235m 39s	hbase-server in the patch passed.
		257m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/11/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 05c2f18da303 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `dde504c`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/11/testReport/
Max. process+thread count	4811 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/11/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-10-20T09:15:52Z

OK, the only failed UT is TestZooKeeper, which is not related.

Let me kick another run to see if the result is stable.

Apache-HBase · 2023-10-20T09:58:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 39s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 43s	master passed
+1 💚	compile	2m 51s	master passed
+1 💚	checkstyle	0m 37s	master passed
+1 💚	spotless	0m 45s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 37s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 44s	the patch passed
+1 💚	compile	2m 56s	the patch passed
+1 💚	javac	2m 56s	the patch passed
+1 💚	checkstyle	0m 36s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 49s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	1m 6s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 26s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		42m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/12/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux cfdaea9e89d2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `dde504c`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	83 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/12/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-20T13:25:05Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 25s	master passed
+1 💚	compile	0m 42s	master passed
+1 💚	shadedjars	5m 56s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 5s	the patch passed
+1 💚	compile	0m 48s	the patch passed
+1 💚	javac	0m 48s	the patch passed
+1 💚	shadedjars	5m 48s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
+1 💚	unit	222m 50s	hbase-server in the patch passed.
		248m 25s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/12/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 89837b462f9c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `dde504c`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/12/testReport/
Max. process+thread count	4474 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/12/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-10-20T13:50:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 56s	master passed
+1 💚	compile	0m 57s	master passed
+1 💚	shadedjars	6m 20s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 28s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 5s	the patch passed
+1 💚	compile	0m 46s	the patch passed
+1 💚	javac	0m 46s	the patch passed
+1 💚	shadedjars	6m 2s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 29s	the patch passed
		_ Other Tests _
+1 💚	unit	246m 28s	hbase-server in the patch passed.
		273m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/12/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5443
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 403b58e7d2c4 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `dde504c`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/12/testReport/
Max. process+thread count	4686 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5443/12/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-10-20T14:10:06Z

OK, good. All green. Let me merge.

…uld never be empty for normal replication queue (#5443) Also add a retry logic to make the code more robust Signed-off-by: Xiaolin Ha <haxiaolin@apache.org> (cherry picked from commit 4429de4)

…uld never be empty for normal replication queue (apache#5443) Also add a retry logic to make the code more robust Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>

Apache9 self-assigned this Sep 29, 2023

Apache9 force-pushed the HBASE-28114 branch from c702eb5 to 0bbca07Compare September 30, 2023 09:27

Apache9 requested a review from sunhelly September 30, 2023 09:29

Apache9 changed the title ~~HBASE-28114 Replication log reader should not simply quit when queue …~~HBASE-28114 Add more comments to explain why replication log queue could never be empty for normal replication queueSep 30, 2023

sunhelly approved these changes Oct 4, 2023
View reviewed changes

Apache9 force-pushed the HBASE-28114 branch from 196e9f1 to dfeeb5eCompare October 8, 2023 03:39

Apache9 mentioned this pull request Oct 13, 2023
HBASE-28140 AbstractWALProvider may miss the WAL which is under creat… #5455
Merged

Apache9 force-pushed the HBASE-28114 branch from dfeeb5e to 00fef6fCompare October 13, 2023 14:34

Apache9 force-pushed the HBASE-28114 branch from 00fef6f to d5154c9Compare October 14, 2023 07:25

Apache9 force-pushed the HBASE-28114 branch from d5154c9 to 0c09178Compare October 15, 2023 13:50

HBASE-28114 Add more comments to explain why replication log queue co…
9710314
…uld never be empty for normal replication queue Also add a retry logic to make the code more robust

Apache9 force-pushed the HBASE-28114 branch from 0c09178 to 9710314Compare October 20, 2023 04:08

Apache9 merged commit 4429de4 into apache:masterOct 20, 2023

Uh oh!

Conversation

Apache9 commented Sep 29, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Apache9 commented Sep 29, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Apache-HBase commented Sep 29, 2023

Uh oh!

Apache-HBase commented Sep 29, 2023

Uh oh!

Apache9 commented Sep 29, 2023

Uh oh!

Apache-HBase commented Sep 29, 2023

Uh oh!

Apache9 commented Sep 30, 2023

Uh oh!

Apache-HBase commented Sep 30, 2023

Uh oh!

Apache-HBase commented Sep 30, 2023

Uh oh!

Apache-HBase commented Sep 30, 2023

Uh oh!

Apache9 commented Oct 1, 2023

Uh oh!

Apache9 commented Oct 1, 2023

Uh oh!

Apache-HBase commented Oct 1, 2023

Uh oh!

Apache-HBase commented Oct 1, 2023

Uh oh!

Apache-HBase commented Oct 2, 2023

Uh oh!

Apache-HBase commented Oct 2, 2023

Uh oh!

Apache-HBase commented Oct 2, 2023

Uh oh!

Apache9 commented Oct 3, 2023

Uh oh!

Apache-HBase commented Oct 3, 2023

Uh oh!

Apache-HBase commented Oct 3, 2023

Uh oh!

Apache-HBase commented Oct 7, 2023

Uh oh!

Apache-HBase commented Oct 7, 2023

Uh oh!

Apache-HBase commented Oct 7, 2023

Uh oh!

Apache9 commented Oct 8, 2023

Uh oh!

Apache-HBase commented Oct 8, 2023

Uh oh!

Apache9 commented Oct 8, 2023

Uh oh!

Apache-HBase commented Oct 13, 2023

Uh oh!

Apache-HBase commented Oct 13, 2023

Uh oh!

Apache-HBase commented Oct 13, 2023

Uh oh!

Apache-HBase commented Oct 14, 2023

Uh oh!

Apache-HBase commented Oct 14, 2023

Uh oh!

Apache-HBase commented Oct 14, 2023

Uh oh!

Apache9 commented Oct 15, 2023

Uh oh!

Apache-HBase commented Oct 15, 2023

Uh oh!

Apache-HBase commented Oct 15, 2023

Uh oh!

Apache-HBase commented Oct 15, 2023

Uh oh!

Apache9 commented Oct 16, 2023

Uh oh!

Apache-HBase commented Oct 20, 2023

Uh oh!

Apache9 commented Sep 29, 2023•
edited
Loading

Apache9 commented Sep 29, 2023•
edited
Loading