HBASE-27034 NegativeArraySizeException was encountered during compaction

[xianjingfeng]

No description provided.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 56s	master passed
+1 💚	compile	2m 10s	master passed
+1 💚	checkstyle	0m 31s	master passed
+1 💚	spotless	0m 41s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 14s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 3s	the patch passed
+1 💚	compile	2m 9s	the patch passed
+1 💚	javac	2m 9s	the patch passed
+1 💚	checkstyle	0m 27s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
-1 ❌	hadoopcheck	8m 20s	The patch causes 10 errors with Hadoop v3.3.1.
+1 💚	spotless	0m 37s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 19s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		25m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4430/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4430
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 8a4cfb2d7c6a 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 369adcd
Default Java	AdoptOpenJDK-1.8.0_282-b08
hadoopcheck	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4430/1/artifact/yetus-general-check/output/patch-javac-3.3.1.txt
Max. process+thread count	65 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4430/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 15s	master passed
+1 💚	compile	0m 41s	master passed
+1 💚	shadedjars	3m 51s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 30s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 24s	the patch passed
+1 💚	compile	0m 41s	the patch passed
+1 💚	javac	0m 41s	the patch passed
+1 💚	shadedjars	3m 44s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	175m 47s	hbase-server in the patch passed.
		194m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4430/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4430
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux bf41c565f8a0 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 369adcd
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4430/1/testReport/
Max. process+thread count	2966 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4430/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 51s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 28s	master passed
+1 💚	compile	0m 41s	master passed
+1 💚	shadedjars	4m 24s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 31s	the patch passed
+1 💚	compile	0m 41s	the patch passed
+1 💚	javac	0m 41s	the patch passed
+1 💚	shadedjars	4m 20s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
+1 💚	unit	191m 12s	hbase-server in the patch passed.
		211m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4430/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4430
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 4d77107fd846 5.4.0-1068-aws #72~18.04.1-Ubuntu SMP Thu Mar 3 08:49:49 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 369adcd
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4430/1/testReport/
Max. process+thread count	2946 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4430/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[binlijin]

Can you describe when this can cause the problem?

[xianjingfeng]

Can you describe when this can cause the problem?
I don't know, I add some code to print lastCellOfPreviousBlock and i found it's value was modified, but i don't know where it was modified, We have a table for storing logs and this problem happens many times every day but other table not.
Other people seem to have encountered this problem. https://developer.aliyun.com/ask/387299

[Apache9]

The change here will impact performance, so we need more detailed analysis on what is the root cause, and then decide how to better fix it.

[xianjingfeng]

The change here will impact performance, so we need more detailed analysis on what is the root cause, and then decide how to better fix it.
HBASE-16372 has explain why references to previous cell in read path will lead to incorrect results. So method beforeShipped of writer was called when bytesWrittenProgressForShippedCall > shippedCallSizeLimit , But this trigger condition looks incorrect

[Apache9]

The change here will impact performance, so we need more detailed analysis on what is the root cause, and then decide how to better fix it.
HBASE-16372 has explain why references to previous cell in read path will lead to incorrect results. So method beforeShipped of writer was called when bytesWrittenProgressForShippedCall > shippedCallSizeLimit , But this trigger condition looks incorrect

Mind explaining why it is incorrect?

[xianjingfeng]

Mind explaining why it is incorrect?

For example, if the number of files to compact is 10 and blocksize is 64k, beforeShipped was called once when every 640k was writeed. How does it make sure the the block that containing lastCellOfPreviousBlock will not be evicted in the meantime? I can't understand. More details in org.apache.hadoop.hbase.regionserver.compactions.Compactor#performCompaction,