HBASE-22942 move Snapshot verification to procedure

[sanjeetnishad95]

Problem : Right now we do snapshot verification prior to queueing the restore / clone request from the client. That means the initial call from the client has to block until we're done. On large manifests (~single digit millions) this easily takes longer than the default timeout of 20 minutes.

Solution: Instead we should handle verification as one of the steps in the relevant procedure (hbase 2+) or table handler (hbase 1) so that the master can do it in the background and report status.

[sanjeetnishad95]

Hi @busbey , I have moved the SnapshotReferenceUtil.verifySnapshot() from SnapshotManager to prepareRestore() step of RestoreSnapshotProcedure. Please review the PR and let me know if this is what was intended?
Thanks.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
💙	reexec	0m 33s	Docker mode activated.
		_ Prechecks _
💚	dupname	0m 0s	No case conflicting files found.
💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
💚	@author	0m 0s	The patch does not contain any @author tags.
💛	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
💚	mvninstall	5m 22s	master passed
💚	compile	0m 56s	master passed
💚	checkstyle	1m 19s	master passed
💚	shadedjars	4m 35s	branch has no errors when building our shaded downstream artifacts.
💚	javadoc	0m 36s	master passed
💙	spotbugs	4m 10s	Used deprecated FindBugs config; considering switching to SpotBugs.
💚	findbugs	4m 9s	master passed
		_ Patch Compile Tests _
💚	mvninstall	4m 56s	the patch passed
💚	compile	0m 58s	the patch passed
💚	javac	0m 58s	the patch passed
💔	checkstyle	1m 16s	hbase-server: The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20)
💚	whitespace	0m 0s	The patch has no whitespace issues.
💚	shadedjars	4m 49s	patch has no errors when building our shaded downstream artifacts.
💚	hadoopcheck	16m 55s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚	javadoc	0m 35s	the patch passed
💚	findbugs	4m 7s	the patch passed
		_ Other Tests _
💔	unit	157m 15s	hbase-server in the patch failed.
💚	asflicense	0m 35s	The patch does not generate ASF License warnings.
		215m 2s

Reason	Tests
Failed junit tests	hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple
	hadoop.hbase.client.TestRestoreSnapshotFromClientSimple

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/out/Dockerfile
GITHUB PR	#662
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux 6429fa7337aa 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-662/out/precommit/personality/provided.sh
git revision	master / ea24ea7
Default Java	1.8.0_181
checkstyle	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/out/diff-checkstyle-hbase-server.txt
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/out/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/testReport/
Max. process+thread count	4567 (vs. ulimit of 10000)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/console
versions	git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by	Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

[saintstack]

What happens to the procedure if the verify fails? Will the Procedure close out cleanly? Or will we it be perpetually rescheduled? Thanks.

[sanjeetnishad95]

Hi @saintstack ,If the procedure fails as a result of verifySnapshot, it will throw corruptedSnapshotException or IOException which will be taken care as a part of the procedure framework. The procedure will close out cleanly.

[busbey]

Please check on the unit test failures since they're supposed to cover the code that changed.

[busbey]

This is currently in a block for only checking non-system tables. Needs to happen for all tables.

Also should probably be the first thing we do in prepareRestore, since it used to proceed the step.

Please update the status monitor about doing the verification as well.

[saintstack]

@sanjeetnishad95 See @busbey comments above? Any chance of addressing them? Thanks.

[sanjeetnishad95]

yes @saintstack , I will address them ASAP.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 31s	master passed
+1 💚	checkstyle	1m 21s	master passed
+1 💚	spotbugs	2m 32s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 14s	the patch passed
-0 ⚠️	checkstyle	1m 16s	hbase-server: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	13m 34s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚	spotbugs	2m 35s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		41m 57s

Subsystem	Report/Notes
Docker	Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#662
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux e7d2dcbeb9c4 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / e1d7dc8
checkstyle	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	84 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 17s	master passed
+1 💚	compile	1m 4s	master passed
+1 💚	shadedjars	5m 48s	branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 42s	hbase-server in master failed.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 5s	the patch passed
+1 💚	compile	1m 3s	the patch passed
+1 💚	javac	1m 3s	the patch passed
+1 💚	shadedjars	5m 44s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 41s	hbase-server in the patch failed.
		_ Other Tests _
-1 ❌	unit	139m 19s	hbase-server in the patch failed.
		165m 20s

Subsystem	Report/Notes
Docker	Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#662
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 8e8876e9906d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / e1d7dc8
Default Java	2020-01-14
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/testReport/
Max. process+thread count	4098 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 8s	master passed
+1 💚	compile	1m 6s	master passed
+1 💚	shadedjars	6m 18s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 49s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 40s	the patch passed
+1 💚	compile	1m 12s	the patch passed
+1 💚	javac	1m 12s	the patch passed
+1 💚	shadedjars	7m 33s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 43s	the patch passed
		_ Other Tests _
-1 ❌	unit	265m 9s	hbase-server in the patch failed.
		293m 44s

Subsystem	Report/Notes
Docker	Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#662
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux ceb98a23fea8 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / e1d7dc8
Default Java	1.8.0_232
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/testReport/
Max. process+thread count	2862 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 30s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 45s	master passed
+1 💚	checkstyle	1m 4s	master passed
+1 💚	spotbugs	2m 1s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 21s	the patch passed
-0 ⚠️	checkstyle	1m 5s	hbase-server: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	10m 59s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚	spotbugs	2m 7s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		32m 10s

Subsystem	Report/Notes
Docker	Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#662
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux e139c5add47b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 2b894dc
checkstyle	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	94 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 45s	master passed
+1 💚	compile	0m 54s	master passed
+1 💚	shadedjars	5m 36s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 23s	the patch passed
+1 💚	compile	0m 55s	the patch passed
+1 💚	javac	0m 55s	the patch passed
+1 💚	shadedjars	5m 33s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	the patch passed
		_ Other Tests _
-1 ❌	unit	169m 30s	hbase-server in the patch failed.
		193m 21s

Subsystem	Report/Notes
Docker	Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#662
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux e4d02cd1fed1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 2b894dc
Default Java	1.8.0_232
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/testReport/
Max. process+thread count	5480 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 56s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	6m 15s	master passed
+1 💚	compile	1m 23s	master passed
+1 💚	shadedjars	7m 40s	branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 52s	hbase-server in master failed.
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 21s	the patch passed
+1 💚	compile	1m 29s	the patch passed
+1 💚	javac	1m 29s	the patch passed
+1 💚	shadedjars	7m 44s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 58s	hbase-server in the patch failed.
		_ Other Tests _
-1 ❌	unit	209m 22s	hbase-server in the patch failed.
		244m 46s

Subsystem	Report/Notes
Docker	Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#662
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 210f721a6603 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 2b894dc
Default Java	2020-01-14
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/testReport/
Max. process+thread count	3143 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

[saintstack]

Closing. No progress. Make a new PR if I have this wrong. Thanks.