Merge PHOENIX-6182: IndexTool to verify and repair every index row (#… by stoty · Pull Request #1126 · apache/phoenix

stoty · 2021-02-01T09:10:45Z

…1022)

PHOENIX-6198 Add option to IndexTool to specify the source table for scan (PHOENIX-6198 Add option to IndexTool to specify the source table for scan #937)
PHOENIX-6198 Add option to IndexTool to specify the source table for scan
Addressed feedback for PHOENIX-6198

Extended the -from-index option to support -vBOTH, -vAFTER and -vNONE.
Added the disclaimer for -vAFTER. Also, using the source table enum from
IndexScrutinyTool.

PHOENIX-6199 Generate different query plan depending upon if the source (PHOENIX-6199 Generate different query plan depending upon if the source is index table or data table #958)

is index table or data table

PHOENIX-6200 Add counters for extra index rows, log results to PIT and PIT_RESULT table (PHOENIX-6200 Add counters for extra index rows, log results to PIT and PIT_RESULT table #995)
PHOENIX-6200 Add counters for extra index rows, log results to PIT and PIT_RESULT table
Address feedback
PHOENIX-6200 (addendum) Fix test case because invalid rows now are
reported as beyond max lookback when max lookback is set to 0
Also add ASF license to one file

stoty · 2021-02-01T09:51:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 3 new or modified test files.
		_ master Compile Tests _
+1 💚	mvninstall	15m 7s	master passed
+0	hbaserecompile	21m 21s	HBase recompiled.
+1 💚	compile	1m 2s	master passed
+1 💚	checkstyle	1m 41s	master passed
+1 💚	javadoc	0m 52s	master passed
+0 🆗	spotbugs	3m 2s	phoenix-core in master has 955 extant spotbugs warnings.
		_ Patch Compile Tests _
-1 ❌	mvninstall	0m 30s	root in the patch failed.
+0	hbaserecompile	9m 59s	HBase recompiled.
-1 ❌	compile	0m 25s	phoenix-core in the patch failed.
-1 ❌	javac	0m 25s	phoenix-core in the patch failed.
-1 ❌	checkstyle	1m 40s	phoenix-core: The patch generated 221 new + 2782 unchanged - 48 fixed = 3003 total (was 2830)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 51s	the patch passed
-1 ❌	spotbugs	0m 19s	phoenix-core in the patch failed.
		_ Other Tests _
-1 ❌	unit	0m 18s	phoenix-core in the patch failed.
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		40m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1126
Optional Tests	dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
uname	Linux af75594d0099 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / `68239ff`
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
mvninstall	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/1/artifact/yetus-general-check/output/patch-mvninstall-root.txt
compile	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/1/artifact/yetus-general-check/output/patch-compile-phoenix-core.txt
javac	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/1/artifact/yetus-general-check/output/patch-compile-phoenix-core.txt
checkstyle	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
spotbugs	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/1/artifact/yetus-general-check/output/patch-spotbugs-phoenix-core.txt
unit	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/1/testReport/
Max. process+thread count	96 (vs. ulimit of 30000)
modules	C: phoenix-core U: phoenix-core
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/1/console
versions	git=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

stoty · 2021-02-02T00:19:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 0s	Docker mode activated.
-1 ❌	patch	0m 3s	#1126 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem	Report/Notes
GITHUB PR	#1126
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/4/console
versions	git=2.17.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

stoty · 2021-02-02T10:00:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 0s	Docker mode activated.
-1 ❌	patch	0m 5s	#1126 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem	Report/Notes
GITHUB PR	#1126
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/5/console
versions	git=2.17.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

stoty · 2021-02-02T10:13:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 0s	Docker mode activated.
-1 ❌	patch	0m 3s	#1126 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem	Report/Notes
GITHUB PR	#1126
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/6/console
versions	git=2.17.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…pache#1022) * PHOENIX-6198 Add option to IndexTool to specify the source table for scan (apache#937) * PHOENIX-6198 Add option to IndexTool to specify the source table for scan * Addressed feedback for PHOENIX-6198 Extended the `-from-index` option to support -vBOTH, -vAFTER and -vNONE. Added the disclaimer for -vAFTER. Also, using the source table enum from IndexScrutinyTool. * PHOENIX-6199 Generate different query plan depending upon if the source (apache#958) is index table or data table * PHOENIX-6200 Add counters for extra index rows, log results to PIT and PIT_RESULT table (apache#995) * PHOENIX-6200 Add counters for extra index rows, log results to PIT and PIT_RESULT table * Address feedback * PHOENIX-6200 (addendum) Fix test case because invalid rows now are reported as beyond max lookback when max lookback is set to 0 Also add ASF license to one file

stoty · 2021-02-02T12:57:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	4m 22s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 5 new or modified test files.
		_ master Compile Tests _
+1 💚	mvninstall	15m 19s	master passed
+0	hbaserecompile	22m 43s	HBase recompiled.
+1 💚	compile	1m 0s	master passed
+1 💚	checkstyle	1m 42s	master passed
+1 💚	javadoc	0m 54s	master passed
+0 🆗	spotbugs	3m 5s	phoenix-core in master has 955 extant spotbugs warnings.
		_ Patch Compile Tests _
+1 💚	mvninstall	8m 18s	the patch passed
+0	hbaserecompile	17m 48s	HBase recompiled.
+1 💚	compile	1m 1s	the patch passed
+1 💚	javac	1m 1s	the patch passed
-1 ❌	checkstyle	1m 50s	phoenix-core: The patch generated 244 new + 2773 unchanged - 56 fixed = 3017 total (was 2829)
-1 ❌	whitespace	0m 0s	The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	javadoc	0m 50s	the patch passed
-1 ❌	spotbugs	3m 12s	phoenix-core generated 4 new + 955 unchanged - 0 fixed = 959 total (was 955)
		_ Other Tests _
-1 ❌	unit	103m 4s	phoenix-core in the patch failed.
+1 💚	asflicense	0m 44s	The patch does not generate ASF License warnings.
		162m 24s

Reason	Tests
FindBugs	module:phoenix-core
	org.apache.phoenix.mapreduce.index.IndexVerificationResultRepository.AFTER_REPAIR_EXTRA_UNVERIFIED_INDEX_ROW_COUNT_BYTES is a mutable array At IndexVerificationResultRepository.java: At IndexVerificationResultRepository.java:[line 138]
	org.apache.phoenix.mapreduce.index.IndexVerificationResultRepository.AFTER_REPAIR_EXTRA_VERIFIED_INDEX_ROW_COUNT_BYTES is a mutable array At IndexVerificationResultRepository.java: At IndexVerificationResultRepository.java:[line 135]
	org.apache.phoenix.mapreduce.index.IndexVerificationResultRepository.BEFORE_REPAIR_EXTRA_UNVERIFIED_INDEX_ROW_COUNT_BYTES is a mutable array At IndexVerificationResultRepository.java: At IndexVerificationResultRepository.java:[line 131]
	org.apache.phoenix.mapreduce.index.IndexVerificationResultRepository.BEFORE_REPAIR_EXTRA_VERIFIED_INDEX_ROW_COUNT_BYTES is a mutable array At IndexVerificationResultRepository.java: At IndexVerificationResultRepository.java:[line 128]
Failed junit tests	phoenix.end2end.PermissionNSEnabledWithCustomAccessControllerIT

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/7/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1126
Optional Tests	dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
uname	Linux 1ba911bd743f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / `d4021de`
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstyle	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/7/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
whitespace	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/7/artifact/yetus-general-check/output/whitespace-eol.txt
spotbugs	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/7/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html
unit	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/7/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/7/testReport/
Max. process+thread count	14920 (vs. ulimit of 30000)
modules	C: phoenix-core U: phoenix-core
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/7/console
versions	git=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

stoty · 2021-02-02T14:46:20Z

Got a clear test run for hbase.profile=2.1 and hbase.profile=2.4 offline.

stoty · 2021-02-02T15:57:39Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 5 new or modified test files.
		_ master Compile Tests _
+1 💚	mvninstall	15m 15s	master passed
+0	hbaserecompile	22m 38s	HBase recompiled.
+1 💚	compile	1m 1s	master passed
+1 💚	checkstyle	1m 42s	master passed
+1 💚	javadoc	0m 52s	master passed
+0 🆗	spotbugs	3m 3s	phoenix-core in master has 955 extant spotbugs warnings.
		_ Patch Compile Tests _
+1 💚	mvninstall	8m 17s	the patch passed
+0	hbaserecompile	18m 2s	HBase recompiled.
+1 💚	compile	1m 2s	the patch passed
+1 💚	javac	1m 2s	the patch passed
-1 ❌	checkstyle	1m 42s	phoenix-core: The patch generated 244 new + 2774 unchanged - 56 fixed = 3018 total (was 2830)
-1 ❌	whitespace	0m 0s	The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	javadoc	0m 50s	the patch passed
-1 ❌	spotbugs	3m 19s	phoenix-core generated 4 new + 955 unchanged - 0 fixed = 959 total (was 955)
		_ Other Tests _
-1 ❌	unit	106m 23s	phoenix-core in the patch failed.
+1 💚	asflicense	0m 45s	The patch does not generate ASF License warnings.
		162m 2s

Reason	Tests
FindBugs	module:phoenix-core
	org.apache.phoenix.mapreduce.index.IndexVerificationResultRepository.AFTER_REPAIR_EXTRA_UNVERIFIED_INDEX_ROW_COUNT_BYTES is a mutable array At IndexVerificationResultRepository.java: At IndexVerificationResultRepository.java:[line 138]
	org.apache.phoenix.mapreduce.index.IndexVerificationResultRepository.AFTER_REPAIR_EXTRA_VERIFIED_INDEX_ROW_COUNT_BYTES is a mutable array At IndexVerificationResultRepository.java: At IndexVerificationResultRepository.java:[line 135]
	org.apache.phoenix.mapreduce.index.IndexVerificationResultRepository.BEFORE_REPAIR_EXTRA_UNVERIFIED_INDEX_ROW_COUNT_BYTES is a mutable array At IndexVerificationResultRepository.java: At IndexVerificationResultRepository.java:[line 131]
	org.apache.phoenix.mapreduce.index.IndexVerificationResultRepository.BEFORE_REPAIR_EXTRA_VERIFIED_INDEX_ROW_COUNT_BYTES is a mutable array At IndexVerificationResultRepository.java: At IndexVerificationResultRepository.java:[line 128]
Failed junit tests	phoenix.end2end.PermissionsCacheIT
	phoenix.end2end.PermissionNSEnabledIT

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/8/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1126
Optional Tests	dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
uname	Linux ba059a512fe9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / `d4021de`
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstyle	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/8/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
whitespace	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/8/artifact/yetus-general-check/output/whitespace-eol.txt
spotbugs	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/8/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html
unit	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/8/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/8/testReport/
Max. process+thread count	12322 (vs. ulimit of 30000)
modules	C: phoenix-core U: phoenix-core
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1126/8/console
versions	git=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

stoty force-pushed the PHOENIX-6182-master branch from 9cb5c33 to b1d6d86Compare February 1, 2021 10:29

stoty mentioned this pull request Feb 1, 2021
Merge PHOENIX-6182: IndexTool to verify and repair every index row (#… #1124
Closed

tkhuranaand others added 5 commits February 2, 2021 11:14

use 4.x logic in IndexRepairRegionScanner
2de1145

more sync with 4.x, EnvironmentEdgeManager fixes
e003f29

fix uncleared dummy row in GlobalIndexRegionScanner
14f88ac

remove unneccessary changes from patch
1ec94e6

stoty force-pushed the PHOENIX-6182-master branch from f0758f7 to 1ec94e6Compare February 2, 2021 10:14

skip tests and don't use skipScanFilter when isRawFilterSupportd=false
ce04dd4

stoty closed this Feb 2, 2021

stoty deleted the PHOENIX-6182-master branch July 29, 2021 12:41

Navigation Menu

Uh oh!

Merge PHOENIX-6182: IndexTool to verify and repair every index row (#… - #1126

Merge PHOENIX-6182: IndexTool to verify and repair every index row (#…#1126
stoty wants to merge 6 commits into
apache:masterfrom
stoty:PHOENIX-6182-master

stoty commented Feb 1, 2021

Uh oh!

stoty commented Feb 1, 2021

Uh oh!

stoty commented Feb 2, 2021

Uh oh!

stoty commented Feb 2, 2021

Uh oh!

stoty commented Feb 2, 2021

Uh oh!

stoty commented Feb 2, 2021

Uh oh!

stoty commented Feb 2, 2021

Uh oh!

stoty commented Feb 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

stoty commented Feb 1, 2021

Uh oh!

stoty commented Feb 1, 2021

Uh oh!

stoty commented Feb 2, 2021

Uh oh!

stoty commented Feb 2, 2021

Uh oh!

stoty commented Feb 2, 2021

Uh oh!

stoty commented Feb 2, 2021

Uh oh!

stoty commented Feb 2, 2021

Uh oh!

stoty commented Feb 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants