HBASE-28903: Incremental backup test missing explicit test for bulkloads by hgromer · Pull Request #6345 · apache/hbase

hgromer · 2024-10-04T20:07:59Z

No description provided.

hgromer · 2024-10-04T20:09:19Z

hgromer · 2024-10-07T21:08:47Z


 // implement all test cases in 1 test since incremental
 // backup/restore has dependencies
+ @SuppressWarnings("checkstyle:MethodLength")


Let me know if there's a better way to go about this. We could split the test up, but I think it's nice to have one single test for the entire backup/restore process. It helps simulate real-life scenarios where we're constantly taking backups of the same table.

Checkstyle failures are not fatal to the build, so please don't mask them. If you think that the current length limit is too low, that's a discission for the dev list.

hgromer · 2024-10-07T21:09:48Z

cc @DieterDP-ng going to tag you here as well :)

Apache-HBase · 2024-10-07T21:48:40Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 6s		master passed
+1 💚	compile	0m 32s		master passed
+1 💚	checkstyle	0m 12s		master passed
+1 💚	spotbugs	0m 32s		master passed
+1 💚	spotless	0m 46s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 53s		the patch passed
+1 💚	compile	0m 31s		the patch passed
+1 💚	javac	0m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 11s		the patch passed
+1 💚	spotbugs	0m 39s		the patch passed
+1 💚	hadoopcheck	10m 34s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 44s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 12s		The patch does not generate ASF License warnings.
		28m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6345/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6345
JIRA Issue	HBASE-28903
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux f63f962e5218 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `be7ef29`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	87 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6345/5/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-10-07T21:50:54Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
-0 ⚠️	yetus	0m 2s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	3m 2s		master passed
+1 💚	compile	0m 21s		master passed
+1 💚	javadoc	0m 15s		master passed
+1 💚	shadedjars	5m 21s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 0s		the patch passed
+1 💚	compile	0m 21s		the patch passed
+1 💚	javac	0m 21s		the patch passed
+1 💚	javadoc	0m 16s		the patch passed
+1 💚	shadedjars	5m 19s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	10m 40s		hbase-backup in the patch passed.
		30m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6345/5/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6345
JIRA Issue	HBASE-28903
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 54de420ea93f 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `be7ef29`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6345/5/testReport/
Max. process+thread count	3733 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6345/5/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

DieterDP-ng · 2024-10-08T08:46:03Z

+ Assert.assertEquals(HBaseTestingUtil.countRows(hTable, famsToLoad), rowsExpected);
+ }
+
+ private static Path createHFiles(TableName tn, String regionName, byte[]... fams)


I'd avoid generating random data. Test failures may get ignored as flaky tests.

DieterDP-ng · 2024-10-08T08:52:57Z

+ newTable1Desc = TableDescriptorBuilder.newBuilder(newTable1Desc)
+ .setColumnFamily(ColumnFamilyDescriptorBuilder.of(fam3Name)).build();
+ updateDesc(newTable1Desc, tables, conn);
+ testBulkloads(conn, famName, mobName);


I think it would be better to include the bulkfile generation and tests in the the above logic rather than add them at the end here. Unless you specifically want to test the behavior when a new CF was added, in that case I'd suggest to create a dedicated test method so it is clear what is being tested.
Just some background: normally each functionality would be tested in a separate test method, but because the backup-creation process takes a long time, some backup tests test multiple things at the same time. Hence they get messy real quick. It's a delicate balance.

Maybe I'll split up the tests in that case. I just want to test that bulkloaded files actually are restored to the table correctly. So I can have a bulkload-specific test that
Bulkloads data + writes data
Takes a full backup
Verifies all the data exists
Bulkloads more data
Takes incremental backup
Verifies data exists

DieterDP-ng · 2024-10-08T08:57:07Z

+ return regionDir;
+ }
+
+ private static void verifyHFileHasNRows(Path path, int rows) throws IOException {


Best to check (some) actual row values as well, not just the counts. This will also be easier without random data.

DieterDP-ng · 2024-10-08T09:01:09Z

If you're up to it, feel free to refactor this test to make it more readable.
If you do so, I suggest 2 commits: one to refactor the test, one to add the bulkload tests to it.

DieterDP-ng · 2024-10-08T09:04:28Z

+ Assert.assertEquals(HBaseTestingUtil.countRows(hTable, famsToLoad), rowsExpected);
+ }
+
+ private static Path createHFiles(TableName tn, String regionName, byte[]... fams)


I'd actually expect that the TestFullBackup also needs to test bulkloads. So either a similar method to generate HFiles already exists, or this method can move to a higher level so it can be used in multiple tests.

hgromer force-pushed the HBASE-28903 branch from f4fcac8 to 426f0b0Compare October 4, 2024 20:15

This comment has been minimized.
Sign in to view

hgromer force-pushed the HBASE-28903 branch from 426f0b0 to 64e85a9Compare October 4, 2024 20:55

This comment has been minimized.
Sign in to view

hgromer force-pushed the HBASE-28903 branch from 64e85a9 to 36bdcd6Compare October 7, 2024 12:45

This comment has been minimized.
Sign in to view

hgromer force-pushed the HBASE-28903 branch from 36bdcd6 to 2291daeCompare October 7, 2024 14:33

This comment has been minimized.
Sign in to view

add explicit bulkload test
be7ef29

hgromer force-pushed the HBASE-28903 branch from 2291dae to be7ef29Compare October 7, 2024 21:07

hgromer commented Oct 7, 2024
View reviewed changes

DieterDP-ng suggested changes Oct 8, 2024
View reviewed changes

Uh oh!

Conversation

hgromer commented Oct 4, 2024

Uh oh!

hgromer commented Oct 4, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hgromer commented Oct 7, 2024

Uh oh!

Apache-HBase commented Oct 7, 2024

Uh oh!

Apache-HBase commented Oct 7, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants